Title: Indeterminate Probability Neural Network
Venue: NeurIPS 2023
Accepted: no
Keywords: Indeterminate Probability

Discrete Random Variable

Unsupervised Clustering

Classification

IPNN
Abstract: We propose a new general model called IPNN - Indeterminate Probability Neural Network, which combines neural network and probability theory together. In the classical probability theory, the calculation of probability is based on the occurrence of events, which is hardly used in current neural networks. In this paper, we propose a new general probability theory, which is an extension of classical probability theory, and makes classical probability theory a special case to our theory. With this new theory, some intractable probability problems have now become tractable (analytical solution). Besides, for our proposed neural network framework, the output of neural network is defined as probability events, and based on the statistical analysis of these events, the inference model for classification task is deduced. IPNN shows new property: It can perform unsupervised clustering while doing classification. Besides, IPNN is capable of making very large classification with very small neural network, e.g. model with 100 output nodes can classify 10 billion categories. Theoretical advantages are reflected in experimental results.

Abstract

We propose a new general model called IPNN -I ndeterminate P robability N eural N etwork, which combines neural network and probability theory together. In the classical probability theory, the calculation of probability is based on the occurrence of events, which is hardly used in current neural networks. In this paper, we propose a new general probability theory, which is an extension of classical probability theory, and makes classical probability theory a special case to our theory. With this new theory, some intractable probability problems have now become tractable (analytical solution). Besides, for our proposed neural network framework, the output of neural network is defined as probability events, and based on the statistical analysis of these events, the inference model for classification task is deduced. IPNN shows new property: It can perform unsupervised clustering while doing classification. Besides, IPNN is capable of making very large classification with very small neural network, e.g. model with 100 output nodes can classify 10 billion categories. Theoretical advantages are reflected in experimental results.

1 Introduction

Humans can distinguish at least 30,000 basic object categories [1], classification of all these would 16

have two challenges: It requires huge well-labeled images; Model with softmax for large scaled 17

datasets is computationally expensive. Zero-Shot Learning ZSL [2, 3] method provides an idea 18

for solving the first problem, which is an attribute-based classification method. ZSL performs object 19

detection based on a human-specified high-level description of the target object instead of training 20

images, like shape, color or even geographic information. But labelling of attributes still needs great 21

efforts and expert experience. Hierarchical softmax can solve the computationally expensive problem, 22

but the performance degrades as the number of classes increase [4]. 23

Probability theory has not only achieved great successes in the classical area, such as Naïve Bayesian 24

method [5], but also in deep neural networks (V AE [6], ZSL, etc.) over the last years. However, both 25

have their shortages: Classical probability can not extract features from samples; For neural networks, 26

27

the extracted features are usually abstract and cannot be directly used for numerical probability

calculation. What if we combine them? 28

There are already some combinations of neural network and bayesian approach, such as probability

29 distribution recognition [7, 8], Bayesian approach are used to improve the accuracy of neural 30 modeling [9], etc. However, current combinations do not take advantages of ZSL method. 31

32

33

34

35

36

We propose an approach to solve the mentioned problems, and our contributions are as follows:

We propose a new general probability theory indeterminate probability theory, which is an extension of classical probability theory, and makes classical probability theory a special case to our theory. The proposed general tractable Equation (12) is analytical solutions even for some intractable probability calculation problems.

37

38

39

40

41

42

43

44

With this new theory, CIPNN [10] has found the analytical solution for the posterior calculation of continuous latent variables, which was regarded as intractable [6, 11]. Besides, CIPNN applied our theory and proposed a general auto encoder (CIPAE), the decoder part is not a neural network and uses a fully probabilistic inference model for the first time.

We propose a novel unified combination of (indeterminate) probability theory and deep neural network. The neural network is used to extract attributes which are defined as discrete random variables, and the inference model for classification task is derived. Besides, these attributes do not need to be labeled in advance.

The rest of this paper is organized as follows: In Section 2, related works are discussed. In Section 3, 45 we first introduce a coin toss game as example of human cognition to explain the core idea of 46 IPNN. In Section 4, the indeterminate probability theory and IPNN is proposed. In Section 5, the 47 training strategy is discussed. In Section 6, we evaluate IPNN and make an impact analysis on its 48 hyper-parameters. Finally, we conclude the paper in Section 7. 49

50

51

52

53

54

55

56

57

58

59

60

61

62

2 Related Work

Tractable Probabilistic Models. There are a large family of tractable models including probabilistic circuits [12, 13], arithmetic circuits [14, 15], sum-product networks [16], cutset networks [17], and-or search spaces [18], and probabilistic sentential decision diagrams [19]. The analytical solution of a probability calculation is defined as occurrence, P ( A = a ) = number of event ( A = a ) occurs number of random experiments , which is however not focused in these models. Our proposed IPNN is fully based on event occurrence and is an analytical solution.

Deep Latent Variable Models. DLVMs are probabilistic models and can refer to the use of neural networks to perform latent variable inference [20]. Currently, the posterior calculation of continuous latent variables is regarded as intractable [11], V AEs [6, 21-23] use variational inference method [24] as approximate solutions. Our proposed IPNN is one DLVM with discrete latent variables and the intractable posterior calculation is now analytically solved with our proposed theory.

3 Background

Let's first introduce a small game coin toss: a child and an adult are observing the outcomes of 63 each coin toss and record the results independently (heads or tails), the child can't always record the 64 results correctly and the adult can record it correctly, in addition, the records of the child are also 65 observed by the adult. After several coin tosses, the question now is, suppose the adult is not allowed 66 to watch the next coin toss, what is the probability of his inference outcome of next coin toss via the 67 child's record? 68

Heads

Tails

Heads

Tails

X

A

Y

As shown in Figure 1, random variables X is the random experiment itself, and X = x k represent the 69 k th random experiment. Y and A are defined to represent the adult's record and the child's record, 70

respectively. And hd, tl is for heads and tails. For example, after 10 coin tosses, the records are 71 shown in Table 1. 72

We formulate X compactly with the ground truth, as shown in Table 2. 73

Through the adult's record Y and the child's records A, we can calculate P ( Y | A ) , as shown in 74 Table 3. We define this process as observation phase. 75

For next coin toss ( X = x 11 ), the question of this game is formulated as calculation of the probability 76 P A ( Y | X ) , superscript A indicates that Y is inferred via record A, not directly observed by the adult. 77 For example, given the next coin toss X = hd = x 11 , the child's record has then two situations: 78 P ( A = hd | X = hd = x 11 ) = 4 / 5 and P ( A = tl | X = hd = x 11 ) = 1 / 5 . With the adult's 79 observation of the child's records, we have P ( Y = hd | A = hd ) = 4 / 4 and P ( Y = hd | A = tl ) = 80 1 / 6 . Therefore, given next coin toss X = hd = x 11 , P A ( Y = hd | X = hd = x 11 ) is the summation 81 4 4 1 1

Let's go one step further, we can find that even the child's record is written in unknown language 83 (e.g. A ∈ { ZHENG,FAN } ), Table 3 can still be calculated by the man. The same is true if the 84 child's record is written from the perspective of attributes, such as color, shape, etc. 85

Hence, if we substitute the child with a neural network and regard the adult's record as the sample 86 labels, although the representation of the model outputs is unknown, the labels of input samples can 87 still be inferred from these outputs. This is the core idea of IPNN. 88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

4 Indeterminate Probability Theory

In this section, we propose a new general probability theory, which is derived from IPNN a neural network with discrete deep latent variables.

4.1 IPNN Model Architecture

Let X ∈ { x 1 , x 2 , . . . , x n } be training samples ( X = x k is understood as k th random experiment - select one train sample.) and Y ∈ { y 1 , y 2 , . . . , y m } consists of m discrete labels (or classes), P ( y l | x k ) = y l ( k ) ∈ { 0 , 1 } describes the label of sample x k . For prediction, we calculate the posterior of the label for a given new input sample x n +1 , it is formulated as P A ( y l | x n +1 ) , superscript A stands for the medium model outputs, via which we can infer label y l , l = 1 , 2 , . . . , m . After P A ( y l | x n +1 ) is calculated, the y l with maximum posterior is the predicted label.

Figure 2a shows IPNN model architecture, the output neurons of a general neural network (FFN, CNN, Resnet [25], Transformer [26], Pretrained-Models [27], etc.) is split into N unequal/equal parts, the split shape is marked as Equation (1), hence, the number of output neurons is the summation of the split shape ∑ N j =1 M j . Next, each split part is passed to 'softmax', so the output neurons can be defined as discrete random variable A j ∈ { a j 1 , a j 2 , . . . , a j M j } , j =

109

110

111

112

113

114

115

116

117

118

119

120

𝑎𝑖1

1

𝑠𝑜𝑓𝑡𝑚𝑎𝑥

𝑠𝑜𝑓𝑡𝑚𝑎𝑥

𝑠𝑜𝑓𝑡𝑚𝑎𝑥

…

N-dimensional

Joint Sample Space

𝑦1

…

…

𝑃(𝑦𝑙 |𝑎

𝑖 1

1

, 𝑎

𝑖 2

2

, … , 𝑎

𝑖 𝑁

𝑁 )

Joint Sample Point

(𝑎 𝑖 1

1

, 𝑎

𝑖 2

2

, … , 𝑎

𝑖 𝑁

𝑁 )

𝐴 1

𝐴 2

𝐴 𝑁

SPLIT into N-Parts

𝑎𝑖2

2

𝑎𝑖𝑁

𝑁

𝐴 1

𝐴 2

𝐴 𝑁

…

Neural

Network

…

𝑥1

𝑥𝑛

𝑥𝑘

…

…

Random Variable

Sample Space

…

𝑦2

𝑦𝑙

𝑦𝑚

(a) model architecture

𝑋

𝐴 1

𝑌

𝐴 2

𝐴 𝑁

…

(b) observation phase

𝑋

𝐴 1

𝑌

𝐴 2

𝐴 𝑁

…

(c) inference phase

1 , 2 , . . . , N , and each neuron in A j is regarded as an event. After that, all the random variables 104 together form the N-dimensional joint sample space, marked as A = ( A 1 , A 2 , . . . , A N ) , and 105 all the joint sample points are fully connected with all labels Y ∈ { y 1 , y 2 , . . . , y m } via condi106 tional probability P ( Y = y l | A 1 = a 1 i 1 , A 2 = a 2 i 2 , . . . , A N = a N i N ) , or more compactly written as 107 P ( y l | a 1 i 1 , a 2 i 2 , . . . , a N i N ) 1 . 2 108

4.2 Definition of Indeterminate Probability

In classical probability theory, perform a random experiment (or given a sample x k ), the event or joint event has only two states: happened or not happened. However, for IPNN, the model only outputs the probability of an event state and its state is indeterminate, that's why this paper is called IPNN. This difference makes the calculation of probability (especially joint probability) also different. Equation (2) and Equation (3) will later formulate this difference.

Given an input sample x k (perform the k th random experiment), with Assumption 1 the indeterminate probability (model outputs) is defined as:

Assumption 1. Given an input sample X = x k , IF ∑ M j i j =1 α j i j ( k ) = 1 and α j i j ( k ) ∈ [0 , 1] , k = 1 , 2 , . . . , n . THEN , { a j 1 , a j 2 , . . . , a j M j } can be regarded as collectively exhaustive and exclusive events set, they are partitions of the sample space of random variable A j , j = 1 , 2 , . . . , N .

In classical probability, α j i j ( k ) ∈ { 0 , 1 } , which indicates the state of event is 0 or 1.

For joint event, given x k , using Assumption 2 and Equation (2), the joint indeterminate probability is 121 formulated as: 122

125

126

127

128

129

130

Assumption 2. Given an input sample X = x k , A 1 , A 2 , . . . , A N is mutually independent. 123

Where it can be easily proved, 124

In classical probability, ∏ N j =1 α j i j ( k ) ∈ { 0 , 1 } , which indicates the state of joint event is 0 or 1.

Equation (2) and Equation (3) describes the uncertainty of the state of event ( A j = a j i j ) and joint event ( A 1 = a 1 i 1 , A 2 = a 2 i 2 , . . . , A N = a N i N ) .

4.3 Observation Phase

In observation phase, the relationship between all random variables A 1 , A 2 , . . . , A N and Y is established after the whole observations, it is formulated as:

Because the state of joint event is not determinate in IPNN, we cannot count its occurrence like 131 classical probability. Hence, the joint probability is calculated according to total probability theorem 132 over all samples X = ( x 1 , x 2 , . . . , x n ) , and with Equation (3) we have: 133

Because Y = y l is sample label and A j = a j i j comes from model, it means A j and Y come from 134 different observer, so we can have Assumption 3 (see Figure 2c). 135

Assumption 3. Given an input sample X = x k , A j and Y is mutually independent in observation 136 phase, j = 1 , 2 , . . . , N . 137

Therefore, according to total probability theorem, Equation (3) and the above assumption, we derive: 138

Substitute Equation (6) and Equation (7) into Equation (5), we have: 139

Where it can be proved, 140

4.4 Inference Phase 141

142

143

144

Given A j , with Equation (8) (passed experience) label y l can be inferred, this inferred y l has no pointing to any specific sample x k , incl. also new input sample x n +1 , see Figure 2b. So we can have following assumption:

145

146

147

Assumption 4. Given A j , X and Y is mutually independent in inference phase, j = 1 , 2 , . . . , N .

Therefore, given a new input sample X = x n +1 , according to total probability theorem over joint sample space ( a 1 i 1 , a 2 i 2 , . . . , a N i N ) ∈ A , with Assumption 4, Equation (3) and Equation (8), we have:

And the maximum posterior is the predicted label of an input sample: 148

4.5 Summary 149

Our most important contribution is that we propose a new general tractable probability Equation (10), 150 rewritten as: 151

152

153

154

155

156

157

158

159

160

161

162

163

164

Where X is random variable and X = x k denote the k th random experiment (or model input sample x k ), Y and A 1: N are different discrete or continuous [10] random variables. This equation can be applied to any random experiment, as long as the outcomes of random experiments are detected by some observers (neural networks, humans, or others).

Our proposed theory is derived from three our proposed conditional mutual independency assumptions, see Assumption 2 Assumption 3 and Assumption 4. However, in our opinion, these assumptions can neither be proved nor falsified, and we do not find any exceptions until now. Since this theory can not be mathematically proved, we can only validate it through experiment.

Finally, our proposed indeterminate probability theory is an extension of classical probability theory, and classical probability theory is one special case to our theory. More details to understand our theory intuitively, see Appendix B.

5 Training

5.1 Training Strategy

Given an input sample x t from a mini batch, with a minor modification of Equation (10): 165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

Where b is for batch size, ¯ t = ⌈ t b ⌉ , t = 1 , 2 , . . . , n . Hyperparameter T is for forgetting use, i.e., H and G are calculated from the recent T batches. Hyper-parameter T is introduced because at beginning of training phase the calculated result with Equation (8) is not good yet. And the ϵ on the denominator is to avoid dividing zero, the ϵ on the numerator is to have an initial value of 1. Besides, H and G are not needed for gradient updating during back-propagation. The detailed algorithm implementation is shown in Algorithm 1.

We use cross entropy as loss function: 181

Algorithm 1 IPNN training

Input : A sample x t from mini-batch Parameter : Split shape, forget number T , ϵ , learning rate η . Output : Posterior P A ( y l | x t )

1: Declare default variables: H,G,hList,gList

2: for ¯ t = 1 , 2 , . . . Until Convergence do

3: Compute h, g with Equation (14) and Equation (15)

4: Record: hList.append ( h ) , gList.append ( g )

5: if ¯ t > T then

6: Forget: H = H -hList [0] , G = G -gList [0]

7: Remove first element from hList, gList

8: end if

9: Compute posterior with Equation (13): P A ( y l | x t )

10: Compute loss with Equation (18): L ( θ )

11: Update model parameter: θ = θ -η ∇L ( θ )

12: Update for next loop: H = H + h, G = G + g

13: end for

14: return model and the posterior

With Equation (13) we can get that P A ( y l | x 1 ) = 1 for the first input sample if y l is the ground truth and batch size is 1. Therefore, for IPNN the loss may increase at the beginning and fall back again while training.

5.2 Multi-degree Classification (Optional)

In IPNN, the model outputs N different random variables A 1 , A 2 , . . . , A N , if we use part of them to form sub-joint sample spaces, we are able of doing sub classification task, the sub-joint spaces are defined as Λ 1 ⊂ A , Λ 2 ⊂ A , . . . The number of sub-joint sample spaces is:

If the input samples are additionally labeled for part of sub-joint sample spaces 3 , defined as Y τ ∈ 189 { y τ 1 , y τ 2 , . . . , y τ m τ } . The sub classification task can be represented as 〈 X, Λ 1 , Y 1 〉 , 〈 X, Λ 2 , Y 2 〉 , . . . 190 With Equation (18) we have, 191

Together with the main loss, the overall loss is L + L 1 + L 2 + . . . In this way, we can perform 192 multi-degree classification task. The additional labels can guide the convergence of the joint sample 193 spaces and speed up the training process, as discussed later in Appendix D.1. 194

182

183

184

185

186

187

188

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

5.3 Multi-degree Unsupervised Clustering

If there are no additional labels for the sub-joint sample spaces, the model are actually doing unsupervised clustering while training. And every sub-joint sample space describes one kind of clustering result, we have Equation (19) number of clustering situations in total.

5.4 Designation of Joint Sample Space

As in Appendix C proved, we have following proposition:

Proposition 1. For P ( y l | x k ) = y l ( k ) ∈ { 0 , 1 } hard label case, IPNN converges to global minimum only when P ( y l | a 1 i 1 , a 2 i 2 , . . . , a N i N ) = 1 , for ∏ N j =1 α j i j ( t ) > 0 , i j = 1 , 2 , . . . , M j . In other word, each joint sample point corresponds to an unique category. However, a category can correspond to one or more joint sample points.

Corollary 1. The necessary condition of achieving the global minimum is when the split shape defined in Equation (1) satisfies: ∏ N j =1 M j ≥ m , where m is the number of classes. That is, for a classification task, the number of all joint sample points is greater than the classification classes.

Theoretically, if model with 100 output nodes are split into 10 equal parts, it can classify 10 billion categories, validation result see Appendix D.1. Besides, the unsupervised clustering (Section 5.3) depends on the input sample distributions, the split shape shall not violate from multi-degree clustering. For example, if the main attributes of one dataset shows three different colors, and your split shape is { 2 , 2 , . . . } , this will hinder the unsupervised clustering, in this case, the shape of one random variable is better set to 3. And as in Appendix D also analyzed, there are two local minimum situations, improper split shape will make IPNN go to local minimum.

In addition, the latter part from Proposition 1 also implies that IPNN may be able of doing further unsupervised classification task, this is beyond the scope of this discussion.

6 Experiments and Results

6.1 Unsupervised Clustering

0

1

2

3

4

5

6

7

8

9

true labels

0

1

cluster id

7

89

5.6

7.2

92

8.7

10

95

15

94

93

11

94

93

7.6

91

90

5

85

6.3

0

20

40

60

80

100

percentage [%]

percentage

=

1

round

·

round

∑

i

=1

number of samples with label

l

in one cluster at

i

th

round

number of samples with label

l

As in Section 5.3 discussed, IPNN is able of performing unsupervised clustering, we evaluate it on MNIST. The split shape is set to { 2 , 10 } , it means we have two random variables, and the first random variable is used to divide MNIST labels 0 , 1 , . . . 9 into two clusters. The cluster results is shown in Figure 3.

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

6.2 Hyper-parameter Analysis

IPNN has two import hyper-parameters: split shape and forget number T. In this section, we have analyzed it with test on MNIST, batch size is set to 64, ϵ = 10 -6 . As shown in Figure 4a, if the number of joint sample points is smaller than 10, IPNN is not able of making a full classification and its test accuracy is proportional to number of joint sample points, as number of joint sample points increases over 10, IPNN goes to global minimum for both 3 cases, this result is consistent with our analysis. However, we have exceptions, the accuracy of split shape with { 2 , 5 } and { 2 , 6 } is not high. From Figure 3 we know that for the first random variable, IPNN sometimes tends to put number 1,4,7,9 into one cluster and the rest into another cluster, so this cluster result request that the split shape need to be set minimums to { 2 , ≥ 6 } in order to have enough free joint sample points. That's why the accuracy of split shape with { 2 , 5 } is not high. (For { 2 , 6 } case, only three numbers are in one cluster.)

Another test in Figure 4b shows that IPNN will go to local minimum as forget number T increases and cannot go to global minimum without further actions, hence, a relative small forget number T shall be found with try and error.

5

10

15

20

25

number of joint sample points

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

accuracy

1D Split Shape

2D Split Shape

(a) split shape

0

200

400

600

800

forget number T

0.4

0.5

0.6

0.7

0.8

0.9

1.0

accuracy

accuracy

(b) forget number T

6.3 Evaluation on Datasets

Further results on MNIST [29], FashionMNIST [30], CIFAR10 [31] and STL10 [32] show that our proposed indeterminate probability theory is valid, the backbone between IPNN and 'Simple-Softmax' is the same, the last layer of the latter one is connected to softmax function. Although IPNN does not reach any SOTA, the results are very important evidences to our proposed mutual independence assumptions, see Assumption 2 Assumption 3 and Assumption 4.

7 Conclusion

For a classification task, we proposed an approach to extract the attributes of input samples as random variables, and these variables are used to form a large joint sample space. After IPNN converges to global minimum, each joint sample point will correspond to an unique category, as discussed in Proposition 1. As the joint sample space increases exponentially, the classification capability of IPNN will increase accordingly.

We can then use the advantages of classical probability theory, for example, for very large joint sample space, we can use the Bayesian network approach or mutual independence among variables (see Appendix E) to simplify the model and improve the inference efficiency, in this way, a more complex Bayesian network could be built for more complex reasoning task.

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

References

Irving Biederman. Recognition-by-components: a theory of human image understanding. In Psychological review , pages 115-147, 1987. doi: 10.1037/0033-295X.94.2.115.

Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 951-958, 2009. doi: 10.1109/CVPR.2009.5206594.

Yanwei Fu, Tao Xiang, Yu-Gang Jiang, Xiangyang Xue, Leonid Sigal, and Shaogang Gong. Recent advances in zero-shot recognition: Toward data-efficient understanding of visual content. IEEE Signal Processing Magazine , 35(1):112-125, 2018. doi: 10.1109/MSP.2017.2763441.

Abdul Arfat Mohammed and Venkatesh Umaashankar. Effectiveness of hierarchical softmax in large scale classification tasks. In 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI) , pages 1090-1094, 2018. doi: 10.1109/ICACCI. 2018.8554637.

Yonghui Cao. Study of the bayesian networks. In 2010 International Conference on E-Health Networking Digital Ecosystems and Technologies (EDT) , volume 1, pages 172-174, 2010. doi: 10.1109/EDT.2010.5496612.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR , abs/1312.6114, 2014.

Chan Su and Chia-Jen Chou. A neural network-based approach for statistical probability distribution recognition. Quality Engineering , 18:293 - 297, 2006.

Ozan Kocada˘ glı and Barı¸ s A¸ sıkgil. Nonlinear time series forecasting with bayesian neural networks. Expert Systems with Applications , 41(15):6596-6610, 2014. ISSN 0957-4174. doi: https://doi.org/10.1016/j.eswa.2014.04.035. URL https://www.sciencedirect.com/ science/article/pii/S0957417414002589 .

Jorge Morales and Wen Yu. Improving neural network's performance using bayesian inference. Neurocomputing , 461:319-326, 2021. ISSN 0925-2312. doi: https://doi.org/10.1016/j. neucom.2021.07.054. URL https://www.sciencedirect.com/science/article/pii/ S0925231221011309 .

Anonymous. Continuous indeterminate probability neural network. ICCV 2023 Submission ID 4297, Supplied as additional material cipnn.pdf .

Diederik P. Kingma and Max Welling. 2019.

YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. oct 2020. URL http://starai.cs.ucla.edu/ papers/ProbCirc20.pdf .

Meihua Dang, Anji Liu, and Guy Van den Broeck. Sparse probabilistic circuits via pruning and growing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 28374-28385. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/ 2022/file/b6089408f4893289296ad0499783b3a6-Paper-Conference.pdf .

Adnan Darwiche. A logical approach to factoring belief networks. In Proceedings of the Eights International Conference on Principles of Knowledge Representation and Reasoning , KR'02, page 409-420, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 1558605541.

Daniel Lowd and Pedro Domingos. Learning arithmetic circuits. In Proceedings of the TwentyFourth Conference on Uncertainty in Artificial Intelligence , UAI'08, page 383-392, Arlington, Virginia, USA, 2008. AUAI Press. ISBN 0974903949.

Hoifung Poon and Pedro Domingos. Sum-product networks: A new deep architecture. In 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops) , pages 689-690, 2011. doi: 10.1109/ICCVW.2011.6130310.

Tahrima Rahman, Prasanna Kothalkar, and Vibhav Gogate. Cutset networks: A simple, tractable, 313 and scalable approach for improving the accuracy of chow-liu trees. In Machine Learning and 314 Knowledge Discovery in Databases , page 630-645, Berlin, Heidelberg, 2014. Springer-Verlag. 315 ISBN 978-3-662-44850-2. doi: 10.1007/978-3-662-44851-9_40. URL https://doi.org/ 316 10.1007/978-3-662-44851-9_40 . 317

318

319

320

Radu Marinescu and Rina Dechter. And/or branch-and-bound for graphical models. In Proceedings of the 19th International Joint Conference on Artificial Intelligence , IJCAI'05, page 224-229, San Francisco, CA, USA, 2005. Morgan Kaufmann Publishers Inc.

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

Doga Kisa, Guy Van den Broeck, Arthur Choi, and Adnan Darwiche. Probabilistic sentential decision diagrams. In Proceedings of the Fourteenth International Conference on Principles of Knowledge Representation and Reasoning , KR'14, page 558-567. AAAI Press, 2014. ISBN 1577356578.

Yoon Kim, Sam Wiseman, and Alexander M. Rush. A tutorial on deep latent variable models of natural language, 2018.

Michalis Titsias and Miguel Lázaro-Gredilla. Doubly stochastic variational bayes for nonconjugate inference. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 1971-1979, Bejing, China, 22-24 Jun 2014. PMLR. URL https://proceedings.mlr.press/v32/titsias14.html .

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32 , ICML'14, page II-1278-II-1286. JMLR.org, 2014.

Karol Gregor, Ivo Danihelka, Andriy Mnih, Charles Blundell, and Daan Wierstra. Deep autoregressive networks. ArXiv , abs/1310.8499, 2013.

Michael I. Jordan, Zoubin Ghahramani, Tommi S. Jaakkola, and Lawrence K. Saul. An introduction to variational methods for graphical models. Mach. Learn. , 37(2):183-233, nov 1999. ISSN 0885-6125. doi: 10.1023/A:1007665907178. URL https://doi.org/10.1023/ A:1007665907178 .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, 2016. doi: 10.1109/CVPR.2016.90.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv , abs/1810.04805, 2019.

Edward Raff and Charles Nicholas. An alternative to ncd for large sequences, lempel-ziv jaccard distance. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '17, page 1007-1015, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450348874. doi: 10.1145/3097983.3098111. URL https://doi.org/10.1145/3097983.3098111 .

Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine , 29(6):141-142, 2012. doi: 10.1109/MSP.2012. 2211477.

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsu363 pervised feature learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík, editors, 364 Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , 365 volume 15 of Proceedings of Machine Learning Research , pages 215-223, Fort Lauderdale, FL, 366 USA, 11-13 Apr 2011. PMLR. URL https://proceedings.mlr.press/v15/coates11a. 367 html . 368

369

370

371

372

Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with relu activation. In Proceedings of the 31st International Conference on Neural Information Processing Systems , NIPS'17, page 597-607, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new probability theory named indeterminate probability theory and a new model name Indeterminate Probability Neural Network (IPNN). For the new probability theory, it is an extension of the classical probability theory, with which some intractable probability problems become tractable (analytical solution). For the new model IPNN, it can perform unsupervised clustering while doing classification and make very large classifications with very small neural networks.\n\nSTRENGTHS:\n1. This paper is well organized. The intuition of the new concept of indeterminate probability is clearly demonstrated from a simple example. The theory of indeterminate probability theory is well formulated and all assumptions are clearly numbered.\n\n2. To the best of my knowledge, the indeterminate probability theory is proposed for the first time.\n\nWEAKNESSES:\n1. The proposed new probability theory and the IPNN model are interesting contributions of this paper, but not the CIPNN. As listed in the second contribution in the introduction part, the authors claim CIPNN as the contribution of this paper. However, CIPNN is the main contribution of the authors' other papers. Thus, the authors should either delete this contribution and introduce CIPNN in other places (such as related work) or contain CIPNN in this paper (then the ICCV paper should be withdrawn).\n\n2. There are many grammar mistakes in the use of articles. For example, 'as example' on line 46 should be 'as an example', and 'an unique category' on line 257 should be 'a unique category'.\n\n3. On lines 110 to 112, the authors mention that the event in classical probability theory can only be happened or not happened, while for IPNN, we can have the probability of an event state. However, the authors do not consider (or demonstrate) the properties of the probability in IPNN. In classical probability theory, the happening of the event is an unbiased estimate of the probability, which means that we can approximate the true probability through plenty of experiments. But what about the probability in IPNN? What is the quality of the probability output by IPNN? Typically, the softmax output of a neural network can only be treated as a probability distribution (since the sum of all coordinates is 1) but does not indicate the true probability of one class.\n\n4. On line 120, the authors mention that $\\alpha_{ij}^j (k) \\in \\{ 0 , 1 \\}$ in classical probability. But as defined in Eq. (2), $\\alpha_{ij}^j (k)$ is a conditional probability. Then it can be any decimal between 0 and 1 if there is no more constraint. Can authors provide more detailed explanations for the claim on this line?\n\n5. As mentioned in the abstract, IPNN 'is capable of making very large classification with very small neural network'. And the idea is using binary encoding ('the binary vector is the model inputs from 000000000000 to 111111111111, which are labeled from 0 to 4095') as mentioned in appendix D.1, which is a basic concept in information theory. But in practical implementations, one-hot encoding is preferred since it does not introduce a redundant distance between different labels. For example, the binary code 001 is closer to 000 than to 111. If binary encodings can successfully reduce the network size, this would be a great contribution to this paper. Can authors provide a more detailed introduction about output encodings (especially the comparison between binary encodings and one-hot encodings in history) and more discussions about the key tricks to making binary encodings work in IPNN?\n\n6. As mentioned in the abstract, the indeterminate probability theory makes 'some intractable probability problems have now become tractable (analytical solution)'. It seems that the authors mix up the concept of tractable and analytical solutions. A tractable problem is a problem that can be solved with acceptable complexity (usually polynomial time and space complexity). Tractability has no relation with analytical solutions. An analytical solution can be intractable when there are exponential operations in the solution, and a tractable problem may have no analytical solution (there is no analytical expression of the error function but we can approximate the error function efficiently).\n\n7. To my understanding, the contribution of this paper is IPNN as a new neural network (architecture, engineering trick, or training algorithm). The indeterminate probability theory is far from an extension of the classical probability theory. To extend the classical probability theory, the authors should at least formulate the new theory from measure-theoretic probability theory.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe main contribution is the proposal of the inference architecture which creates parallel softmax outputs (the authors call “splits”), which are combined into a joint soft-label space to make classification decisions under MAP rule; the joint label space can help with sub-classification tasks.\n\nSTRENGTHS:\nThe architecture tries to create more Bayesian information before making final classification, which is a potential Bayesian neural network approach that can be developed in the future. On the other hand, this work may be inspiring to people who are interested in large-dimensional label representation.\n\nWEAKNESSES:\n1.\tIt is not clear to see in this paper what is new to the “classical theory of probability”;\n2.\tThe novelty is limited.\n3.\tThe assumptions may not be very reasonable:\nThe assumption 2 and 3 are ok at initialization, however when the weights are updated, A and Y are not generally independent;\nThe assumption 4 is very counterintuitive. Normally we have a joint distributions between X and Y, but assuming that X and Y are not independent mutually, then they are not given A; and if X and Y are independent (Y is random label for example), then they are independent given A. \n4.\tThe main contribution of this paper is the splitting part of the architecture, creating parallel softmax outputs and combine them to make MAP decision. I think the paper should emphasize on the reasoning of this mechanism, for example, the ensemble of different likelihoods that contributes to the performance, or the geometric interpretation of the proposed label embedding/representation that makes sense.\n5.\tLacking of the verification of the advantage of the new method over original softmax. After reading this paper I still do not know if this method creates more or reduce a little uncertainty in the classification compared to original softmax approach.\n\nQUESTIONS:\n1.\tCan you compare your work with the literature on label embedding/representation?\n2.\tCan you make some theoretical explanation on the advantage of your methods?\n3.\tCan you give us better explanation on Assumption 4?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes what I consider to be a type of neuro-symbolic AI model involving neural networks for multi-class classification problems; the authors refer to the model as an indeterminate probability neural network (IPNN). Frankly I did not fully understand the model, but the main intent appears to be a form of latent variable modeling for multi-class classification. An important line to summarize the paper is mentioned in Section 2: “our proposed IPNN is one DPVM (deep latent variable model) with discrete latent variables and the intractable posterior calculation is now analytically solved with our proposed theory”. I have reviewed a version of this paper previously, and it looks like the paper has not changed much; this is unfortunate, because I did not understand the paper clearly at that time, and I feel that I still do not understand it well.\n\nSTRENGTHS:\nThe paper claims to blend neural networks and probability theory in a novel way; if this is true then it can be seen as an innovation in neural network modeling as well as neuro-symbolic AI. Another strength is that the paper is non-standard in that it tries to do something new around modeling multi-class classification problems.\n\nWEAKNESSES:\nIn current form, the paper suffers from a number of weaknesses. A major weakness is that it remains unclear why a new theory of probability is needed in the first place. Note that the example in Section 3 can be studied using standard Bayesian modeling where X_i is the true coin toss result, A_i the adult’s reading and Y_i is the child’s reading, all for the i^{th} coin toss. Here X_i are i.i.d random variables, and A_i and Y_i are conditionally independent given X_i. Then the query of interest can be posed in terms of these random variables. I did not understand what the new theory is and why it is even needed. Also, I find it hard to follow the paper and feel it is not appropriately positioned in the literature. For instance, to me, it appears that the proposed approach is a form of neuro-symbolic AI, yet this is not even mentioned in the paper. I feel there is far too much lack of clarity in the paper in general.\n\nQUESTIONS:\nSome questions, comments and suggestion follow:\n\nWhat exactly is the “indeterminate” aspect of IPNN? Why is it needed? Could you please explain in the paper with an example?\n\nThe distinction between the submission and the work under review on CIPNN is unclear from Section 1.\n\nThe related work section is far too small – I’m sure there is a lot of other relevant work in related areas.\n\nAm I correct in assuming that multiple labels are not allowed for a specific sample? Could you please confirm? If so, how is this constraint enforced besides just choosing the label with maximum posterior P(y_l|x_{n+1})?\n\nThe authors claim that very large classification problems can be solved with small neural networks. However, from my limited understanding of the paper, I understood that the network splits up based on the number of training samples N. How is this then a small neural network? Doesn’t it depend on N? What is the size of the neural network (in terms of number of edges) in the proposed approach, and how is it different from the baselines?\n\nWhat is the distinction between observation and inference? Is this just the distinction between learning a model and deploying it for a query?\n\nIn my opinion, the experimental section does not provide much evidence for the usefulness of the proposed approach.\n\nI noticed several typos and grammatical errors in the paper. I suggest the authors review it carefully and fix these errors.\n\nSome references seem incomplete.\n\nI suggest using capital letter “B” for “Bayesian”.\n\nLIMITATIONS:\nThe authors need to write more about the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}