Title: Double-Bayesian Learning
Venue: NeurIPS 2024
Accepted: no
Keywords: Artificial Intelligence

Machine Learning

Learning Theory

Bayesian Inference

Neural Networks

Information Theory

Optimization

Stochastic Gradient Descent

Golden Ratio
Abstract: Contemporary machine learning methods will try to approach the Bayes error, as it is the lowest possible error any model can achieve. This paper postulates that any decision is composed of not one but two Bayesian decisions and that decision-making is, therefore, a double-Bayesian process. The paper shows how this duality implies intrinsic uncertainty in decisions and how it incorporates explainability. The proposed approach understands that Bayesian learning is tantamount to finding a base for a logarithmic function measuring uncertainty, with solutions being fixed points. Furthermore, following this approach, the golden ratio describes possible solutions satisfying Bayes' theorem. The double-Bayesian framework suggests using a learning rate and momentum weight with values similar to those used in the literature to train neural networks with stochastic gradient descent.

Abstract

Contemporary machine learning methods will try to approach the Bayes error, as it is the lowest possible error any model can achieve. This paper postulates that any decision is composed of not one but two Bayesian decisions and that decisionmaking is, therefore, a double-Bayesian process. The paper shows how this duality implies intrinsic uncertainty in decisions and how it incorporates explainability. The proposed approach understands that Bayesian learning is tantamount to finding a base for a logarithmic function measuring uncertainty, with solutions being fixed points. Furthermore, following this approach, the golden ratio describes possible solutions satisfying Bayes' theorem. The double-Bayesian framework suggests using a learning rate and momentum weight with values similar to those used in the literature to train neural networks with stochastic gradient descent.

1 Introduction

Despite the progress in machine learning, several problems stand out for which convincing solutions have yet to be found. With massive training sets, enormously sized networks, and immense computing power, training machine learning models has become a brute force approach, arguably more concerned with memorization than generalization. However, quoting from a post by Y. LeCun (Nov. 23, 2023), we know that

Animals and humans get very smart very quickly with vastly smaller amounts of training data than current AI systems. Current large language models (LLMs) are trained on text data that would take 20,000 years for a human to read. And still, they haven't learned that if A is the same as B, then B is the same as A. Humans get a lot smarter than that with comparatively little training data. Even corvids, parrots, dogs, and octopuses get smarter than that very, very quickly, with only 2 billion neurons and a few trillion "parameters."

This raises the question of whether modern training techniques and principles are actually biologically implemented in the human brain and, if not, what alternative methods could save resources. More efficient methods would be better at generalizing with smaller amounts of training data, which almost certainly would also improve the explainability and interpretability of neural networks.

This paper investigates what it takes for a classifier to be optimal. The starting point is Bayes' theorem, 28 which is the foundation of the Bayes classifier. The Bayes classifier is considered optimal because 29 it minimizes the Bayes risk, meaning it has the smallest probability of misclassification among all 30 classifiers. However, applying the Bayes classifier directly is often impossible because of the difficulty 31 in computing the posterior probabilities. For this reason, most classifiers are trying to approximate 32 the Bayes classifier, like the naïve Bayes classifier, for instance. The information-theoretical analysis 33 presented in this paper splits the decision of a Bayes classifier into two decisions, each following 34 Bayes' theorem, where one decision can serve as an explanation or verification of the other. Each of 35 the two decision processes faces intrinsic uncertainty, as its decision depends on the output of the 36

other process. The paper will investigate the theoretical ramifications of this approach. As a practical 37

result, it will discuss the consequences for two hyperparameters of stochastic gradient descent used 38

in the training process of a neural network: learning rate and momentum weight. 39

The structure of the paper is as follows: After this introduction, Section 2 motivates one of the main 40

ideas, namely that learning to make a decision involves solving two sub-problems and, thus, two 41

decisions. Section 3 discusses Bayes' theorem, which is central to statistical decision-making and 42

is the starting point of the theoretical approach outlined in the following. Section 4 then introduces 43

the double-Bayesian model as the key concept of the paper. The next section, Section 5, shows 44 how to represent possible solutions of the double-Bayesian decision model. Section 6 discusses the 45 golden ratio, including its functional equations and how it defines a solution to the double-Bayesian 46 model. Then, Section 7 discusses the theoretical implications for training double-Bayesian networks 47 with stochastic gradient descent. Finally, Section 8 summarizes the key concepts, followed by a 48 conclusion. 49

2 Dual decisions 50

Suppose a sender transmits the image on the left-hand side of Figure 1 to a receiver. This image

depicts Rubin's vase by the Danish psychologist Edgar Rubin (Rubin, 1915), which shows a vase or two faces looking at each other, depending on the receiver's perception. The receiver then faces an unsolvable conundrum: 1) If the receiver thinks the image represents a vase, the receiver cannot be certain that the vase is indeed the intended message the sender wanted to convey. Maybe the sender wanted to send the faces. 2) If the receiver is expecting a picture of a vase (or faces) and thus knows the intended message, there is no certainty that an image of a vase has been transmitted. After all, the image could show faces. Therefore, two decisions are involved in making the final interpretation of the image: 1) a decision about the perception of the image (vase or faces), and 2) a decision about whether the perceived image coincides with the intended message, meaning the image transmitted. Both decisions together are fraught with intrinsic uncertainty because deciding the ultimate interpretation of Rubin's vase, a vase or faces, is impossible. Therefore, neither the sender nor the receiver can make both decisions without uncertainty. Instead, the knowledge is distributed. The sender knows the intended message (a vase or faces) but not the receiver's perceived image. On the other hand, the receiver knows the perceived image (a vase or faces) but not the intended message. Therefore, the sender and the receiver must collaborate to get the true interpretation across their communication channel.

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

Let the sender and receiver perceive Rubin's vase differently, with contrary opinions about the foreground and background color (black or white), where the foreground represents the perceived image, either a vase or faces. Furthermore, let the sender and the receiver both be able to send an image of Rubin's vase to each other so that both become senders and receivers alike and can share their knowledge about the perceived image and intended message. The image that the sender perceives is then the inverted image that the sender perceives. The goal is to collaborate so that the perceived image (foreground) equals the intended message on both ends.

A sender can either send the image of Rubin's vase on the left-hand side of Figure 1 or send the 75 image with colors inverted, as shown on the right-hand side of Figure 1, depending on the perceived 76 image or intended message, respectively. On the other end, the receiver has two options: 1) accept 77 the received image if it is identical to the image expected, or 2) tell the sender to invert the image if it 78 is different. After this feedback, the image on the receiver end will be the same as the image on the 79

sender side. By making the images on both sides the same, the receiver has completed half of the 80 decision process without making a mistake and has thus behaved optimally. The receiver has ensured 81 that both sides see the same image. It is now up to the sender to make the final, second decision about 82 what image needs to be inverted to arrive at the final interpretation, either the image of the sender 83 or the image of the receiver. Thus, the first process tries to make the images identical, whereas the 84 second process tries to make the images different on both ends to reflect the different perceptions of 85 the sender and receiver. 86

Although described as a sequential process, the two dual decision processes leading to the final 87 interpretation are running in parallel. The sender is also a receiver, and the receiver is also a sender. 88 One of them conveys the correct foreground information (black or white), while the other conveys the 89 message. Note that neither the sender nor the receiver will ever see the true interpretation of the image. 90 The receiver in the example above will never know whether the received image needs to be inverted 91 after making the images identical because this would mean the receiver knows the true interpretation 92 of the image, which is not possible according to the uncertainty principle described above. A similar 93 statement can be made for the sender. The sender and the receiver can be considered dual and 94 complementary forces because of their different interpretations of foreground and background. They 95 make two binary decisions, deciding on the correct foreground color (black or white) and on the 96 message (a vase or faces). They decide whether Rubin's vase should be interpreted as a white vase, a 97 black vase, white faces, or black faces. 98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

3 Bayes theorem

Bayes' theorem is a fundamental law in probability theory that describes the probability of an event given prior knowledge. The theorem is of central importance in machine learning, where it guides the training of machines for decision-making, such as in Bayesian inference or naïve Bayes classification. For two events A and B , with prior probabilities P ( A ) and P ( B ) , and P ( B ) = 0 , Bayes' theorem states the following:

̸

where P ( A | B ) and P ( B | A ) are the conditional or posterior probabilities. Thus, P ( A | B ) is the probability of event A occurring when B is true, and analogously, P ( B | A ) is the probability of B given that A is true.

For a machine learning application, A would be the class of an observed input pattern B . The probability P ( A ) is then the prior probability of class A , and P ( B ) is the prior probability of seeing pattern B . Consequently, P ( A | B ) is the posterior probability of class A when seeing pattern B , and P ( B | A ) is the posterior probability of B within A . According to Bayes' theorem, three probabilities are needed to compute the probability P ( A | B ) that class A is observed when seeing pattern B : P ( A ) , P ( B ) , and P ( B | A ) . However, several obstacles prevent Bayes' theorem from being applied in this way. No particular method can help determine the prior probabilities, which are often unknown. Furthermore, the posterior probability is often not readily available and is approximated by making assumptions about the distribution of B given A , for example, assuming a normal distribution.

To cope with these limitations, the next section describes decision-making as a dual process based on Bayes' theorem, with uncertainty intrinsically involved.

4 Double-Bayesian framework

The Bayes Theorem is typically stated as in Eq. 1. However, restating the theorem in the following equivalent form highlights the two decision processes for the two subproblems involved, as motivated in Section 2:

The left-hand side of Eq. 2 features a fraction of the posterior probabilities, whereas the right-hand 123 side shows the prior probabilities. Following the motivation in Section 2, the posterior probabilities, 124 P ( A | B ) and P ( B | A ) , can be understood as the probability that A or B is the intended message, 125 respectively. Then, the prior probabilities, P ( A ) and P ( B ) , would express the probabilities that A or 126 B is in the foreground. 127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

With only one equation for four parameters, Eq. 2 is underdetermined. However, it is fair to assume that 1 -( P ( A | B ) = P ( B | A ) and 1 -P ( A ) = P ( B ) , which leaves one equation with one parameter on each side. This is possible because either A or B can be the message or foreground, not both of them at the same time, following again the reasoning in Section 2. Therefore, the intrinsic uncertainty in Bayes' theorem can be described as follows: if the true foreground is known, then whether the message needs to be swapped is unknown; on the other hand, if the message is known, then whether the foreground needs to be swapped is unknown. The fractions on both sides of Eq. 2 are thus "cognitively entangled."

The two remaining unknown parameters can be computed using two separate processes, each adding a constraint to handle the uncertainty. To illustrate this, Eq. 3 restates Bayes' theorem in yet another way:

Assuming that P ( B ) = P ( B | A ) , Eq. 3 simplifies to P ( A | B ) = P ( A ) . This assumption of B being independent of A is fair because, according to the motivation in Section 2, the decisions about the message and the foreground are independent of each other. Under this assumption, only one unknown remains, either P ( A | B ) or P ( A ) , which follows directly from either P ( A ) or P ( A | B ) , depending on which is input and which is output.

A similar, symmetric statement can be made when using the reciprocals on both sides of Eq. 2, which leads to the following equation:

Here, assuming that A is independent of B simplifies Eq. 4 to P ( B | A ) = P ( B ) .

Solving Eq. 2, Eq. 3, or Eq. 4 will be referred to as solving the outer Bayes equation. On the other hand, making both multiplicands on the right-hand side of Eq. 3 or Eq. 4 identical will be referred to as solving the inner Bayes equation, or simply solving the inner equation of Eq. 3 or Eq. 4. For Eq. 3, the inner Bayes equation thus states as follows:

Accordingly, the inner Bayes equation for Eq. 4 is obtained by using the reciprocals of the fractions on both sides of Eq. 5:

Consequently, the inner Bayes equations can derived by inverting a fraction on one side of Bayes' 153 theorem, as stated in Eq. 2. The inner Bayes equations are thus "entangled" versions of Bayes' 154 theorem. 155

156

157

158

159

160

The two independent decision processes motivated above are solving the inner and outer Bayes equations. To further formalize these processes, the following section will add a logarithmic expression to Eq. 3 and Eq. 4. Adding a logarithm offers several advantages: 1) using information theory to measure uncertainty; 2) using a reciprocal becomes equivalent to changing the sign of a logarithm; and 3) solving the equation in Bayes' theorem is reduced to finding a suitable base for a logarithm.

161

162

163

164

165

5 Fixpoint solutions

Using a logarithmic expression in Eq. 3 and Eq. 4 is possible when solutions become fixed points of a logarithmic function. To illustrate this, let log b ( x ) be the logarithm for an input x and a base b . By definition, the logarithm is the inverse function of taking the power. Therefore, the following equation holds:

̸

For the base b of a logarithm, any positive real number can be used so long as b = 1 . A logarithm 166 computed for base b can be converted into a logarithm for base b ′ as follows: 167

Therefore, the simple term log is used for the logarithm in the following. 168

By applying the logarithm to probabilities, they become information. For the two dual processes 169 above, the information of one process will be its counterpart's information with a different sign. To 170 achieve this, the following identity is required: 171

The following lemma states that this requirement can be met for general input values. 172

Lemma: For every x ∈ R + \ { 1 } , there exists a base λ so that log λ ( x ) = x . 173

Proof: Let b ∈ R + \ { 1 } be an arbitrary basis for which log b ( x ) = y . Furthermore, let k be 174 a multiplier so that ky = x . Then, log λ ( x ) = x for λ = b 1 /k . This follows from Eq.8, with 175 log λ ( x ) = log b ( x ) / log b ( λ ) = log b ( x ) / log b ( b 1 /k ) = log b ( x ) · k = x . 176

̸

Note that the common logarithmic rules apply for a fixed λ . However, when requiring a λ that always 177 satisfies log λ ( x ) = x , computations become ambiguous, as seen here: -log λ ( x ) = -x = 1 /x = 178 log λ (1 /x ) . The base λ should be understood as a dynamic parameter that a learning system can 179 modify over time so that log λ ( x ) converges to the input x . 180

Using the log λ expression of the above Lemma, the Bayes' equation in Eq. 3 can be written as 181 follows: 182

Then, the following sequence of transformations can be derived from Eq. 10: 183

where the last expression holds for an angle ϕ ∈ [ 0 ; π 2 ] . The reasoning behind these transformations 184 is as follows: 185

The first step, Eq. 11, moves the posterior probability P ( A | B ) back to the left-hand side of the 186 equation. The result is Bayes' theorem in its original form, as shown in Equation 1. 187

The next step, Eq. 12, replaces P ( A ) with 1 -P ( B ) , removing one degree of freedom as motivated 188 above. 189

192

193

Then, Eq. 14 rewrites the right-hand side of Eq. 13, transforming 1 -P ( B ) = P ( B ) 2 into the equivalent P ( B ) = 1 -P ( B ) 2 , which must hold true to satisfy the inner Bayes equation.

194

195

196

197

198

199

200

201

202

203

204

205

206

207

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

equations can be visualized. Following the reasoning above, the right-hand side of Eq. 16 represents the inner Bayes equation. Accordingly, after bringing the factor 2 on the other side of Eq. 16, the inner Bayes equation is satisfied when sin( ϕ ) = cos( ϕ ) , which is the case for ϕ = π/ 4 , with sin( π/ 4) = cos( π/ 4) = 1 / √ 2 .

For the dual process, the log λ expression can be used in combination with the other term of the inner Bayes equation in Eq. 3, as shown here:

Note that the log λ expression has moved to the left compared to the right-hand side of Eq. 10. From 208 this equation, the following sequence of transformations can be derived similar to the transformations 209 above. 210

During this sequence, assumptions similar to the ones in Eq. 12 and Eq. 13 are made. In Eq. 19, P ( B | A ) was replaced by 1 -P ( A | B ) , and Eq. 20 assumes that P ( A ) = P ( A | B ) . Again, all transformations assume that both multiplicands on the right-hand side are equal to satisfy the inner Bayes equation.

The intrinsic uncertainty for the dual processes can again be seen in Eq. 16 and Eq. 23, where it manifests like this: if the base λ is known, then the angle ϕ is unknown; and vice versa, if ϕ is known, then λ is unknown. Each process contributes knowledge about λ and ϕ , which the other process does not know.

The process knowledge about λ and ϕ does not need to be "all-or-nothing." The uncertainty ranges continuously between two extremes, and both dual processes can be somewhat knowledgeable about both parameters. When sin( ϕ ) = cos( ϕ ) , with ϕ = π/ 4 , one process has no or full knowledge about one parameter. With ϕ approaching 0 or π/ 2 , where sin( ϕ ) and cos( ϕ ) become different, this knowledge increases or decreases, respectively.

6 Golden ratio

The solution to the inner Bayes equation is connected to the golden ratio (Livio, 2002), which becomes evident from the transformations of equations above and the assumptions made for both processes. Based on their right-hand equations, both dual processes must meet the same requirement to satisfy the inner Bayes equation, assuming that log λ ( x ) produces x . For Eq. 12, with P ( B ) = P ( B | A ) , and for the corresponding Eq. 19 of the dual process, with P ( A ) = P ( A | B ) , this requirement can be written as

where the variable p is a placeholder for one of the probabilities. Eq. 24 holds true if p is the golden ratio, which is defined by the equivalent quadratic equation,

which has two irrational solutions p 1 and p 2 :

and 234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

A key observation is that the complement of both solutions, 1 -p , equals their square:

Alternatively, another quadratic equation that may be more frequently encountered in textbooks can be used to arrive at the golden ratio. This equation is obtained by substituting -p for p in Eq. 25:

The alternative equation also possesses two irrational solutions, namely the negations of p 1 and p 2 :

For these solutions, the complement 1 -p is the negative reciprocal:

Computing the complement of the golden ratio allows changing viewpoints and switching between the solutions to the inner and outer Bayes equations. This will become important in the next section for training neural networks.

The golden ratio is sometimes represented by the letter φ in the literature. It is often defined as a single value, usually φ ≈ 1 . 618 , and negative values are not considered (Livio, 2002; Huntley, 1970). However, each of the four solutions to the aforementioned quadratic equations will be referred to as the golden ratio in the context of this paper.

7 Theoretical implications

Supervised training methods first present a teaching input to a neural network and then try to make the network's output the same as the input by adjusting the network weights. This equalizing of input and output can be related to equalizing multiplicands to satisfy the inner Bayes equation. For example, in Eq. 18, the term P ( B | A ) /P ( A | B ) can be considered as input and the term P ( A ) in the lambda expression as output. The task of the lambda expression is then to make both terms the same to satisfy the inner Bayes equation. Moreover, the lambda expression log λ ( P ( A ) ) becomes the gradient of a linear function for the outer Bayes equation. These relationships help to determine the optimal learning rate and momentum weight for training based on backpropagation and stochastic gradient descent (SGD).

A training method based on backpropagation estimates the gradient of a loss function with respect to each network weight, where the loss function measures the difference between input and network output. Backpropagation methods try to minimize the loss by following the gradient and updating the network weights accordingly (LeCun et al., 2012). They accomplish this for one network layer at a time, iteratively propagating the gradient back from the output layer to the input layer. To move along the gradient towards the minimum of the loss function, a delta is added to each weight, which often has the following form, including a momentum term:

In (32), L is the loss function, and ∆ w ij ( t ) denotes the delta added to each weight w ij between a node i and a node j in the network at training iteration (or time) t . The term ∂L/∂w ij ( t ) is the partial derivative of the loss function with respect to w ij , at time t , which is multiplied with the learning rate η . The sign of ∆ w ij ( t ) is negative, so the loss function approaches its minimum. In practice, a momentum term describing the weight change at time t -1 , ∆ w ij ( t -1) , is commonly added. This term is typically multiplied by a weighting factor α , as seen in (32).

The traditional understanding is that the momentum term improves stochastic gradient descent by 270 dampening oscillations. However, the dual process model offers another explanation for the per271 formance improvement brought about by the momentum term. As of yet, a conclusive theory for 272 the optimal values of the learning rate η and the momentum weight α has been lacking. Although 273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

second-order methods (Bengio, 2012; Sutskever et al., 2013; Spall, 2000) as well as adaptive methods (Jacobs, 1988; Kingma and Ba, 2014; Duchi et al., 2011; Tieleman and Hinton, 2012) have been tried with various degrees of success, an ultimate answer has still to be found. Both parameters are usually determined heuristically through empirical experiments or systematic search (Bergstra and Bengio, 2012). Training results can be very sensitive to the value of the learning rate. For example, a small learning rate may result in slow convergence, whereas a larger learning rate may result in the search passing over the minimum loss. Negotiating this delicate trade-off in the regularization of the training process can be time-consuming in practical applications. The literature seems to prefer initial learning rates around 0 . 01 or smaller for SGD, although reported values differ by several orders of magnitude. For the momentum weight, higher initial values around 0 . 9 are more common (Li et al., 2020; Krizhevsky et al., 2012; Simonyan and Zisserman, 2014; He et al., 2016).

As shown in the following, the proposed dual process model allows deriving theoretical values for both regularization parameters: learning rate η and momentum weight α . In the weight adjustment given by Eq. 32, each summand represents a gradient of one of the two dual processes. These are the partial derivative ∂L/∂w ij ( t ) and the momentum term ∆ w ij ( t -1) . The momentum weight α follows from the results above, where the lambda expression can be considered as the gradient of the current iteration at time t . The other multiplicand of the inner Bayes equation corresponds to the gradient of the other dual process at time t -1 , assuming that both dual processes are interleaved, if not in parallel.

The previous sections showed that the inner Bayes equation is met when both summands are equal to sin ( π/ 4) = cos( π/ 4) = 1 / √ 2 and when they are equal to the golden ratio. Therefore, the delta at t -1 , ∆ w ij ( t -1) , needs to be multiplied by a constant to obtain the golden ratio. This constant is the momentum weight α , which needs to satisfy α/ √ 2 = p 1 , and can thus be computed as follows.

where p 1 is the value of the golden ratio in Eq. 26. So, this logic provides the value of the first regularization term, namely the momentum weight α , with α ≈ 0 . 874 .

The learning rate η can be derived from the momentum weight α by converting the latter to the corresponding value for the dual process. The dual process does not aim to satisfy the inner Bayes equation with ϕ = π/ 4 . Instead, it aims to satisfy the outer Bayes equation, with ϕ = 0 or ϕ = π/ 2 , and thus sin( ϕ ) = 0 and cos( ϕ ) = 1 , or sin( ϕ ) = 1 and cos( ϕ ) = 0 . By moving in the opposite direction of the gradient of its dual counterpart, the first process can minimize its loss in satisfying the inner Bayes equation. Accordingly, taking the complement of the momentum weight α twice results in the learning rate η for the gradient change at time t . Taking the complement of α twice can be understood as looking at the same process from a dual point of view. Mathematically, this can be achieved by squaring the simple complement, 1 -α . Squaring the complement follows the functional equation of the golden ratio described by Eq.28. Squaring also means bringing the multiplier 2 back in, which was extracted from the lambda expression in Eq. 15 and Eq. 22 to represent all solutions graphically. Applying these steps to the momentum weight α then results in the following equation for the learning rate η :

So, this computation provides the value for the second regularization term, learning rate η , with η ≈ 0 . 016 .

8 Discussion

Starting from Bayes' theorem, this paper develops a theoretical framework that describes any decision of a machine classifier as the result of two processes. The first decision process determines the input message; specifically, it decides whether the input is encoded according to its true value or needs to be inverted. On the other hand, the second decision process decides whether the output should be equal to the input or needs to be inverted. Although both decision processes run simultaneously, they are independent processes, with each possessing knowledge not accessible to the other process. What is uncertain for one process is certain for the other, and vice versa. The first process does not know whether the input should be equal to the output, and conversely, the second process does not know whether the input needs to be inverted. This means a binary decision always involves two bits, one indicating the encoding of the input and the other defining the relationship between input and output.

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

However, practically, only one of the two processes can be performed at a time, leaving one bit of uncertainty for one of the processes.

Theoretically, the framework proposed here formulates this duality with two processes having different perceptions of zero and one (black and white). The output of one process is the input to the other process. While one process tries to make its output equal to its input, the other aims for the opposite and tries to make its output as different as possible. The mathematical definitions of these processes are defined by the outer and inner Bayes equation, the latter of which is an entangled version of the original Bayes' theorem. By introducing the logarithm, each process is given a control parameter, namely the base of the logarithm, to achieve its goal. This parameter, which is essentially a multiplier, allows each process to control the magnitude of the input/output.

The solution space of the proposed double-Bayesian decision framework can be visualized with the trigonometric functions sin and cos . Furthermore, the golden ratio defines solutions to the inner Bayes equation. Connecting these two observations leads to specific values for momentum weight and learning rate for stochastic gradient descent, which tries to minimize the difference between training input and output during training.

The supplemental material to this paper contains experiments for the MNIST dataset (LeCun et al., accessed May 21, 2024), where the proposed double-Bayesian learning framework is practically evaluated. The theoretical parameters found in this paper did, in fact, provide the best performance for a network trained with stochastic gradient descent in a large grid search for learning rate and momentum weight.

9 Conclusion

Three primary characteristics define the work presented in this paper: First, a double-Bayesian approach that understands learning as a process involving two Bayesian decisions instead of a single decision, like in contemporary approaches. Second, solving a Bayesian decision problem is equivalent to finding a fixed point for a logarithmic function measuring uncertainty. Third, the golden ratio defines solutions to a Bayesian decision problem. These three characteristics make the proposed approach novel and unique.

The double-Bayesian framework leads to new theoretical results for training neural networks, particularly specific hyperparameter values for backpropagation and gradient descent. These results are in contrast with other gradient descent heuristics in the literature that either use dynamic hyperparameters or second-order methods for adjusting parameters during training. It will be interesting to see how this conceptual difference will be resolved in the future. The proposed framework offers new ways to understand how neural networks make decisions and may thus contribute to the interpretability and explainability of neural networks, an actively investigated research area.

The proposed framework may also help build bridges to other disciplines like neuroscience or physics. For example, representing all possible solutions to a double-Bayesian decision by means of trigonometric functions, as done in this paper, introduces waves. Incorporating brain waves into machine learning, a feature that traditional machine learning approaches are arguably lacking, would likely entail a better understanding of learning in general. This better understanding could mean training methods for smaller networks that could achieve the same performance with less training data, as motivated at the beginning of this paper.

Another example of a discipline that could be related to this work is quantum mechanics. One of the fundamental concepts in quantum mechanics is Heisenberg's uncertainty principle, which states that certain pairs of physical properties, such as the position and momentum of an electron, cannot be measured with absolute certainty. The more accurately one property is measured, the less is known about the other property. The proposed double-Bayesian framework incorporates such an intrinsic uncertainty and makes a connection to Bayesian decision theory, which could lead to new insights.

Although empirical evidence in the literature supports the theoretical hyperparameter values derived in this paper, and the experiments in the supplemental material show that these values outperform other value pairs, more practical experiments are needed to corroborate these values. To address this limitation, future work will validate the practicality of the derived hyperparameter values in additional experiments across different domains and compare their performance with the performance of other values and other optimization strategies.

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

References

Bengio. Practical recommendations for gradient-based training of deep architectures. In Neural networks: Tricks of the trade , pages 437-478. Springer, 2012.

Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine learning research , 13(2), 2012.

Buitinck, G. Louppe, M. Blondel, F. Pedregosa, A. Mueller, O. Grisel, V. Niculae, P. Prettenhofer, A. Gramfort, J. Grobler, R. Layton, J. VanderPlas, A. Joly, B. Holt, and G. Varoquaux. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning , pages 108-122, 2013.

Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research , 12(7), 2011.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on computer vision and pattern recognition , pages 770-778, 2016.

Huntley. The Divine Proportion . Dover Publications, 1970.

Jacobs. Increased rates of convergence through learning rate adaptation. Neural networks , 1(4): 295-307, 1988.

Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , pages 1097-1105, 2012.

LeCun, L. Bottou, G. Orr, and K. Müller. Efficient backprop. In Neural networks: Tricks of the trade , pages 9-48. Springer, 2012.

LeCun, C. Cortes, and C. Burges. The MNIST Database , accessed May 21, 2024. URL http: //yann.lecun.com/exdb/mnist/ .

Li, P. Chaudhari, H. Yang, M. Lam, A. Ravichandran, R. Bhotika, and S. Soatto. Rethinking the hyperparameters for fine-tuning. arXiv preprint arXiv:2002.11770 , 2020.

Livio. The Golden Ratio . Random House, Inc., 2002.

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Rubin. Rubin Vase . Wikimedia Commons (last accessed March 4, 2022, CC BY-SA 3.0), 1915. URL https://commons.wikimedia.org/wiki/File:Facevase.png .

Scikit-learn developers (BSD License). Scikit-learn machine learning library , accessed May 21, 2024. URL https://scikit-learn.org/stable/modules/generated/sklearn.model_ selection.StratifiedShuffleSplit.html .

Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Spall. Adaptive stochastic approximation by the simultaneous perturbation method. IEEE transactions on automatic control , 45(10):1839-1853, 2000.

Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning , pages 1139-1147, 2013.

Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning , 4(2):26-31, 2012.

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: This is a theoretical paper that tries to explain hyperparameter values that have been successfully used in the literature. The paper investigates what it takes for a classifier to be optimal, as stated in the introduction. Although the literature and the practical experiments provided in the supplemental material support the theoretical results, providing more practical experiments would be desirable.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed at the very end of the paper in the conclusion. A comparison with other hyperparameter optimization strategies would be desirable to corroborate the theoretical results even more. Specifically, a systematic comparison with second-order methods and other methods that dynamically adapt hyperparameters during training should shed more light on the performance of this approach.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are discussed in detail, and one proof has been included. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Experimental results are listed in the supplemental material, with information to reproduce the results, including the code itself.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Please see the supplemental material for the code and information about reproducing the experimental results. The publicly available MNIST database has been used for the experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please see the information in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper provides theoretical results. For the experimental results in the supplemental material, only the relative performance to other hyperparameter combinations was investigated, significant or not, to see whether the proposed values

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

define the optimum or are at least close to it. To compare the proposed method and values with other optimization methods, future experiments may require significance tests.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please see the supplemental material for more information.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: There is no violation of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper proposes a generic method to find hyperparameter values for optimizing the performance of neural networks. Its societal impacts, therefore, correlate with the risks of machine learning in general, which does not need to be pointed out in particular according to the guidelines below.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no risk of misusing the proposed method beyond misusing machine learning in general.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

Justification: The main paper cites relevant references for the scientific content and the supplemental material provides more details about the data and software sources.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper provides new assets in the form of knowledge about hyperparameter values to train neural networks with gradient descent and software to find the best combination of momentum weight and learning rate with a grid search. Each asset is documented in the paper and supplemental material, respectively.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

A Appendix / supplemental material

Two grid searches for the publicly available MNIST dataset were performed to corroborate the 756 learning rate and momentum weight derived in the main paper (LeCun et al., accessed May 21, 2024). 757 The MNIST dataset contains gray-scale images of handwritten digits and is one of the prominent 758 datasets used to evaluate machine learning methods. It is split into a training and a test set, where the 759 latter serves as a standard of comparison. Figure 2 shows an example of the MNIST data.

A.1 Experiments

760

761

762

763

764

765

766

767

768

The grid searches were performed on the full-size MNIST dataset and a smaller version of MNIST containing only 50% of the training data. In the latter case, a stratified sampling method named StratifiedShuffleSplit was used to create a stratified random subset of the training samples (Scikitlearn developers , BSD License; Pedregosa et al., 2011; Buitinck et al., 2013). This ensured that the class distribution in the training subset was the same as in the original full-size training set. The degradation in dataset size allowed observing how each optimizer performed under varying amounts of training data, assuming that providing less training data posed a harder problem.

A deep learning model was trained based on a convolutional neural network (CNN). The model 769 consisted of two convolutional layers, each followed by a ReLU activation function and a max 770 pooling operation. The first convolutional layer had a single-channel input (grayscale image) and 771 applied 16 filters, followed by a second convolutional layer that expanded the channel size to 32. 772 Both convolutional layers used a 3x3 kernel size, a stride of one, and a padding of one. After each 773 convolution, a ReLU activation function introduced non-linearity, and a max pooling operation with 774

a 2x2 kernel and stride reduced the spatial dimensions by half. A dropout layer with a rate of 0.25 775 was applied after flattening the output to prevent overfitting. The network concluded with two fully 776 connected layers with a final output of 10 classes, where the maximum output value determined the 777 class of an input image. The number of parameters was around two hundred thousand for an MNIST 778 input image of size 28x28. A weight initialization was performed using the Kaiming uniform method. 779 No data augmentation techniques were applied; however, the input was normalized to the range [-1,1]. 780 The training used a batch size of 64 and was conducted over 30 epochs, employing cross entropy 781 as the loss function. The sizes of the training, validation, and test datasets were 54,000, 6,000, and 782 10,000, respectively. Finally, the model's performance was assessed through 10-fold cross-validation. 783

784

A.2 Results

The results of both grid searches are shown in Figure 3 for the full-size training set and in Figure 3 785 for the smaller training set with 50% of the size. The following values were used as momentum

Accuracy (Mean: 99.32,Std: 0.01)

Accuracy (Mean: 99.32,Std: 0.01)

-100

0.925

97.22

99.10

99.30

99.32

10.54

10.42

96.53

99.01

99.31

99.34

10.55

10.49

80

0.874

95.86

98.93

99.31

99.34

88.58

10.68

0.85

95.22

98.86

99.28

99.33

98.81

10.68

60

Momentum

0.825

94.53

98.78

99.29

99.27

99.04

10.68

0.8

93.89

98.74

99.29

99.31

99.18

10.52

40

0

89.60

98.16

99.23

99.27

99.31

99.19

0

84.53

97.66

99.15

99.23

99.34

99.30

20

2

77.06

97.08

99.07

99.18

99.33

99.31

0

69.27

96.54

99.01

99.15

99.30

99.29

0.0001

0.001

0.01

0.016

0.1

0.2

Learning rate

786 weights for each grid search: 0, 0.2, 0.4, 0.6, 0.8, 0.825, 0.85, 0.874, 0.9, and 0.925. On the other 787 hand, the following values were used as learning rates: 0.0001, 0.001, 0.01, 0.016, 0.1, 0.2. These 788 values included the momentum weight derived in the paper ( α ≈ 0 . 874 ) and the derived learning 789 rate ( η ≈ 0 . 016 ). Other values were chosen based on their use in the literature or to increase the 790 resolution around the derived theoretical values. All possible combinations of values span a 6x10 791 grid. The color of each square in the grids of Figure 3 and Figure 4 represent the performance of the 792 corresponding pair of momentum weight and learning rate, with lighter colors representing higher 793 performance. Green rectangles indicate the top ten performing pairs, whereas blue rectangles show 794 the best-performing pair. Note that more than one pair can share the best performance, as in Figure 3. 795

Figure 3 shows that no pair of momentum weight and learning rate provides better performance on 796 the full-size MNIST set than the pair derived in the paper, (0 . 016 , 0 . 874) , although this pair has to 797 share its first place with other pairs. The classification accuracies for the reduced training set size 798 are slightly lower in the table of Figure 4, as one would expect for a problem with less training data. 799

800

801

802

803

804

805

806

807

808

809

810

Nevertheless, the theoretical values derived in the paper for momentum weight and learning rate show

Accuracy (Mean: 99.14,Std: 0.02)

Accuracy (Mean: 99.14,Std: 0.02)

-100

0.925

94.91

98.76

99.15

99.15

10.56

10.19

0

93.60

98.63

99.11

99.14

10.91

10.25

0.874

80

92.26

98.50

99.08

99.17

98.05

10.40

0.85

91.20

98.37

99.07

99.09

98.71

10.34

60

Momentum

0.825

90.06

98.25

99.08

99.09

98.87

10.35

0.8

88.94

98.12

99.06

99.11

98.99

10.35

40

0.6

73.97

96.95

98.96

99.04

99.13

99.06

62.32

95.81

98.85

98.97

99.13

99.11

20

2

53.78

94.64

98.73

98.90

99.12

99.16

0.0

46.73

93.60

98.63

98.83

99.10

99.15

0

0.0001

0.001

0.01

0.016

0.1

0.2

Learning rate

again the best performance.

A.3 Computational environment and runtime

The software was developed using Python 3.10, and the Convolutional Neural Network (CNN) model was implemented in Pytorch 2.2.2. For each combination of learning rate and momentum weight (60 combinations in total), the training time was approximately three hours for 100% of the training set size and about 1.5 hours for 50% of the training set. Consequently, the cumulative GPU time for all experiments was approximately (3 + 1.5) × 60 hours, which is 270 hours. The average memory usage was roughly 1 GB for each combination. For more information about the software requirements and workflow, see the Readme file uploaded as supplemental material together with the code.

A.4 Computing cluster

Figure 5 shows an overview of the GPU computing cluster that was available for the experiments, 811 including the type of GPUs among which the processing was distributed. 812

Peer reviews:
comments: SUMMARY:
This paper appears to suggest that any decision is composed of two Bayesian decisions and it trys to evaluate the implications of this idea. 

I am very confused by this paper and really don't know what to make out of it. For example, the conclusion seems to be only a brainstorming session of random ideas and the rest of the paper does not appear to be much better.

At the very least, it is not well written, at worst the proposed approach does not make any sense.

STRENGTHS:
Given that I don't properly understand what exactly the authors want to achieve, I am unable to formulate the strengths of this paper.

WEAKNESSES:
The presentation is very messy. The paper jumps from topic to topic without me understanding their relations to each other.

QUESTIONS:
see above

LIMITATIONS:
see above
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 1
DATE: 2024-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses the implications of Bayes' theorem, making assumptions inspired by a thought experiment of communicating a message. Prior (and model) elicitation by solving a fixed point equation is discussed.\n\nSTRENGTHS:\n* The paper takes a fresh look at decision marking under uncertainty, which is at the center of machine learning.\n* The generality of the setting makes the discussion applicable to virtually all of ML.\n\nWEAKNESSES:\nWhile I am sensible to the topic of prior and model elicitation from coherence arguments, I believe the paper needs a thorough revision focussing on clarity. While I have some intuition now, it is still not crystal clear to me what the exact goal or claims of the paper are. See bullets below for constructive comments.\n\n## Major\n1. Section 4: what is the probability $P$? What is the underlying space and sigma algebra? What are they supposed to represent?  \n2. Section 4 introduces several very strong assumptions, like $1-P(A\\vert B) = P(B\\vert A)$ (is it for all $A,B$ in some sigma-algebra or for a specific pair of events?), that are motivated by an analogy about communicating a message. It is not clear why I should be prepared to make these strong assumptions. The fact that I don't know what $P$ is supposed to model or serve as does not help. Is it a joint probability over the variables describing a decision problem, as in decision theory? In that case, will it be used in conjunction to a loss function to make decisions? Will it be judged by some measure of decision accuracy? Or are we in a de Finetti framework, coming up with a personal probability $P$ which we will use to make predictions about unobserved variables? My intuition is that we are dealing with the latter kind, but this should be explained. And the strong assumptions need to be motivated by more than an analogy about communication.\n3. The information analogy which motivates imposing the fixed point equation (9) is unclear, as well to what probability and what events it should apply.\n4. p5 L179: the sentence about the parameter being a dynamic parameter for a learning system is unclear. We haven't discussed any learning algorithm yet.\n5. I am not sure I see where Eqn (11) comes from. $\\lambda$ has been chosen to derive (10) from Bayes' theorem, but it doesn't have to be the right base to write (11), right? Same remark for (18).\n\n## Minor\n1. p7 L248: Although neural networks have been a popular class of models and algorithms, supervised learning is not synonymous with neural network training.\n2. p7 L252: the meaning of \"the $\\lambda$ expression\" is unclear.\n\nQUESTIONS:\n* Can you formally rephrase the goal and claims of the paper?\n* Can you explain what $P$ is representing? Is it a personal probability in the spirit of de Finetti, or a model of the data generating process? Or maybe something else?\n* Can you formalize and list the assumptions you make on $P$, and justify them in the context of predicting a categorical variable?\n\nLIMITATIONS:\nThis is fundamental work that does not have any immediate negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe purpose of this paper is to investigate the optimality of a classifier. It is known that the Bayes classifier is optimal, and it is likewise known that an explicit computation of the Bayes classifier is often very challenging if not impossible. This paper offers an analysis of the Bayes classifier as a sequential solution of two problems. An analysis and interpretation of a vase / faces example is presented and some theory is developed to further understand it. The paper concludes with an application.\n\nSTRENGTHS:\nThe authors are exploring an idea which is novel, and the whole thinking about Bayes classifiers as comprising two sub-problems seems novel and worth pursuing.\n\nWEAKNESSES:\nI did not really understand the discussion with the vase, the sender and receiver. Perhaps the authors should somehow connect the Bayesian ideas to the description of the problem earlier? I think the paper would really benefit from rewriting Section 4 with the vase as a running example, because it is hard to connect the various decisions with the probabilities. Maybe it's worth to add more illustrations / diagrams for this? The authors are presenting novel ideas and it's hard to understand them as they are currently presented.\n\nFor the theoretical implications, I think it would be better to illustrate the approach on a simpler model like a linear one. \n\nThe paper started by mentioning the Bayes classifier but does not come back to it as an example. \n\nThe paper states that the Bayes classifier is broken up into two decisions, but those are just briefly mentioned in the vase / faces example. The authors should carry this thread of reasoning through the whole paper.\n\nQUESTIONS:\nIn line 134, you say that \"...if the message is known, then whether the foreground needs to be swapped is unknown.\" But isn't knowing the message \"vase\" or \"faces\" enough? How will swapping the background help?\n\nHow are the fixpoint solutions connected to the whole vase / faces example?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-19"}