Title: Transforming to Yoked Neural Networks to Improve ANN Structure
Venue: NeurIPS 2023
Accepted: no
Keywords: Deep Learning

Neural Networks
Abstract: Artificial neural networks (ANNs) are typically designed as a tree structure to mimic neural networks. However, this structure has limitations as nodes on the same level cannot communicate with each other, resulting in deficiencies. To address this, we proposed the Yoke neural networks (YNN) model, which connected nodes bidirectionally in a complete graph, forming a neural module. YNN improved information transfer and eliminated structural bias more effectively than traditional ANNs. Although ANN structures have advanced to more complex structures such as Directed Acyclic Graph (DAG) in recent years, these methods also exhibit unidirectional and acyclic biases for ANN. Compared to traditional ANN, our YNN can emulate neural networks more effectively. In this study, we analyzed the limitations of existing ANN structures and attached an auxiliary sparsity constraint to the distribution of connectedness to focus on critical connections. Unlike traditional structures which are a flow of tensors, with each unit representing a model level and each level being a tensor with independent elements, YNN treated each tensor in the flow as a whole to represent an information unit since the elements in the tensor interacted with each other. Moreover, based on the optimized structure, we designed a neural module structure using the minimum cut technique to reduce the calculation of the YNN model. This learning process was compatible with existing networks and various tasks that efficiently eliminated structural bias. The quantitative results of our experiments indicated that the learned connectivity was superior to the traditional neural network structure.

Abstract

Most existing classical artificial neural networks (ANN) are designed as a tree structure to imitate neural networks. In this paper, we argue that the connectivity of a tree is not sufficient to characterize a neural network. The nodes of the same level of a tree cannot be connected with each other, i.e., these neural unit cannot share information with each other, which is a major drawback of ANN. Although ANN has been significantly improved in recent years to more complex structures, such as the directed acyclic graph (DAG), these methods also have unidirectional and acyclic bias for ANN. In this paper, we propose a method to build a bidirectional complete graph for the nodes in the same level of an ANN, which yokes the nodes of the same level to formulate a neural module. We call our model as YNN in short. YNN promotes the information transfer significantly which obviously helps in improving the performance of the method. Our YNN can imitate neural networks much better compared with the traditional ANN. In this paper, we analyze the existing structural bias of ANN and propose a model YNN to efficiently eliminate such structural bias. In our model, nodes also carry out aggregation and transformation of features, and edges determine the flow of information. We further impose auxiliary sparsity constraint to the distribution of connectedness, which promotes the learned structure to focus on critical connections. Finally, based on the optimized structure, we also design small neural module structure based on the minimum cut technique to reduce the computational burden of the YNN model. This learning process is compatible with the existing networks and different tasks. The obtained quantitative experimental results reflect that the learned connectivity is superior to the traditional NN structure.

1 Introduction

Deep learning successfully transits the feature engineering from manual to automatic design and

enables optimization of the mapping function from sample to feature. Consequently, the search 26

for effective neural networks has gradually become an important and practical direction. However, 27

designing the architecture remains a challenging task. Certain research studies explore the impact 28

of depth [1,2,3] and the type of convolution [4,5] on performance. Moreover, some researchers 29

have attempted to simplify the architecture design. VGGNet [6] was directly stacked by a series 30

of convolution layers with plain topology. To better adapt the optimization process of gradient 31

descent process, GoogleNet [7] introduced parallel modules, while Highway networks [8] employed 32

gating units to regulate information flow, resulting in elastic topologies. Driven by the significance 33

of depth, the residual block consisted of residual mapping and shortcut was raised in ResNet [9]. 34

Topological changes in neural networks successfully scaled up neural networks to hundreds of layers. 35

The proposed residual connectivity was widely approved and was subsequently applied in other 36

works such as MobileNet [10,11] and ShuffleNet [12]. Divergent from the relative sparse topologies, 37

DenseNet [13] wired densely among blocks to fully leverage feature reuse. Recent advances in 38 computer vision [25,26] also explored neural architecture search (NAS) methods [14,15,16] to search 39 convolutional blocks. In recent years, Yuan proposed a topological perspective using directed acyclic 40 graph (DAG) [29] to represent neural networks, enhancing the topological capabilities of artificial 41 neural networks (ANNs). However, these approaches suffer from the bias of unidirectional and 42 acyclic structures, limiting the signal's capability for free transmission in the network. 43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

The existing efforts in neural network connectivity have primarily focused on the tree structures where neural units at the same level cannot exchange information with each other, resulting in significant drawbacks for ANNs. This limitation arises due to the absence of a neural module concept. In this paper, we argue that the current connectivity approaches fail to adequately capture the essence of neural networks. Since the nodes at the same level of a tree cannot establish connections with each other, it hampers the transfer of information between these neural units, leading to substantial defects for ANNs. We argue that the nodes in the same level should form a neural module and establish interconnections. As a result, we introduce a method to build up a bidirectional complete graph for nodes at the same level of an ANN. By linking the nodes in a YOKE fashion, we create neural modules. Furthermore, when we consider all the nodes at the same level, we would have a chance to construct a bidirectional complete graph in ANNs and yields remarkable improvements. We refer to our model as Yoked Neural Network, YNN for brevity. It is important to note that if all the edge weights in the bidirectional complete graph become vestigial and approach to zero, our YNN would reduce to a traditional tree structure.

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

In this paper, we analyze the structural bias of existing ANN structures. To more accurately mimic neural networks, our method efficiently eliminates structural bias. In our model, nodes not only aggregate and transform features but also determine the information flow. We achieve this by assigning learnable parameters to the edges, which reflect the magnitude of connections. This allows the learning process to resemble traditional learning methods, enhancing the overall performance of our model in imitating neural networks. As the nodes are relied on the values of other nodes, it is a challenging task designing a bidirectional complete graph for nodes at the same level. We address this challenge by introducing a synchronization method specifically tailored for learning the nodes at the same level. This synchronization method is crucial for ensuring the effective coordination and learning of these interconnected nodes.

Finally, to optimize the structure of YNN, we further attach an auxiliary sparsity constraint that influences the distribution of connectedness. This constraint promotes the learned structure to prioritize critical connections, enhancing the overall efficiency of the learning process.

The learning process is compatible with existing networks and exhibits adaptability to larger search spaces and diverse tasks, effectively eliminating the structural bias. We evaluate the effectiveness of our optimization method by conducting experiments on classical networks, demonstrating its competitiveness compared to existing networks. Additionally, to showcase the benefits of connectivity learning, we evaluate our method across various tasks and datasets. The quantitative results from these experiments indicate the superiority of the learned connectivity in terms of performance and effectiveness.

Considering that the synchronization algorithm for nodes at the same level may be computationally intense, we also propose a method to design small neural modules to simplify our model. This approach significantly reduces the computational burden of our model while maintaining its effectiveness.

To sum up, our contributions in this paper are as follows:

We provide an analysis of the structural bias present in existing ANN networks.

We propose the YNN model which involves YOKING the nodes at the same level together to simulate real neural networks.

We develop a synchronization method to effectively learn and coordinate the nodes at the same level, introducing the concept of neural modules.

We design a regularization-based optimization method to optimize the structure of the YNN model.

We propose the design of small neural modules to significantly reduce the computational complexity of our model, improving its efficiently.

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

2 Related Works

We firstly review some related works on the design of neural network structures and relevant optimization methods. The design of neural network has been studied widely. From shallow to deep, the shortcut connection plays an important role. Before ResNet, an early practice [17] also added linear layers connected from input to output to train multi-layer perceptrons. [7] was composed of a shortcut branch and a few deeper branches. The existence of shortcut eases the vanishing or exploding gradients [8,9]. Recently, Yuan [29] explained from a topological perspective that shortcuts offer dense connections and benefit optimization. Many networks with dense connections exist On macro-structures also. In DenseNet [13], all preceding layers are connected. HRNet [18] was benefited from dense high-to-low connections for fine representations. Densely connected networks promote the specific task of localization [19]. Differently, our YNN optimizes the desired network from a bidirectional complete graph in a differentiable way.

For the learning process, our method is consistent with DARTS [22], which is differentiable. Different from sample-based optimization methods [29], the connectivity is learned simultaneously through the weights of the network using our modified version of the gradient descent. A joint training can shift the transferring step from one task to another, and obtain task-related YNN. This type was explored in [20,21,22,23,24] also, where weight-sharing is utilized across models at the cost of training. At the same time, for our YNN model, we also propose a synchronization method to get the node values in the same neural module.

In order to optimize the learned structure, a sparsity constraint can be observed in other applications, e.g., path selection for a multi-branch network [27], pruning unimportant channels for fast inference [28], etc. In a recent work, Yuan used L1 regularization to optimize a topological structure. In this paper, we also use L1 as well as L2 regularization to search a better structure.

Secondly, many deep learning works deal with the geometric data in these years[40]. They make neural network better cope with structure. Graph neural networks (GNNs) are connectivity-driven models, which have been addressing the need of geometric deep learning[30,31]. In fact, a GNN adapts its structure to that of an input graph, and captures complex dependencies of an underlying system through an iterative process of aggregation of information. This allows to predict the properties of specific nodes, connections, or of the entire graph as a whole, and also to generalize to unseen graphs. Due to these powerful features, GNNs have been utilized in many relevant applications to accomplish their tasks, such as recommender systems [33], natural language processing [34], traffic speed prediction [35], critical data classification [36], computer vision [25,26,37], particle physics [38], resource allocation in computer networks [39], and so on.

3 Methodology

3.1 Why YNN is Introduced?

NNstands for a type of information flow. The traditional structure of ANN is a tree, which is a natural way to describe this type of information flow. Then, we can represent the architecture as G = ( N,E ) , where N is the set of nodes and E denotes the set of edges. In this tree, each edge e ij ∈ E performs a transformation operation parameterized by w ij , where ij stands for the topological ordering from the node n i to node n j with n i , n j ∈ N . In fact, the importance of the connection is determined by the weight of e ij . The tree structure as a natural way to represent such formation flow is most frequently used in ANN.

A tree is a hierarchical nested structure where a node can be influenced only by its precursor node, thereby causing transformation of information between them. In a tree structure, the root node has no precursor node, while each other node has one and only one precursor node. The leaf node has no subsequent nodes. The number of subsequent nodes of each other node can be one or multiple. In addition, the tree structure in mathematical statistics can represent some hierarchical relationships. A tree structure has many applications. It can also indicate subordinating relationships.

In recent years, some researchers attempted to generalize this structure. In those works, except the root node, all other nodes are made to have multiple precursor nodes, i.e., the hierarchical information flow is made to form a directed acyclic graph (DAG).

However, a tree or a DAG is a hierarchical nested structure where a node can be influenced only by 142 its precursor node, which makes the transformation of information quite inadequate. Moreover, we 143 find that this structure is far more inferior in its strength compared with those of real neural networks, 144 which connect far more complex structures than a tree or DAG structure as shown in Fig 1. In fact, 145 a tree or a DAG structure is used just because its good mathematical properties which can apply 146 backward propagation conveniently. 147

Softmax

Softmax

outputlayer

Hiddenlayer

1

(e.g.,ReLU)

Input

1

layer

X

X2

In this paper, we represent the neural network as a bidirectional complete graph for the nodes of 148 the same level to make the description of NN is much better compared with the traditional ANN. 149 Further, the connections between nodes are represented as directed edges, which determine the flow 150 of information between the connected nodes. We consider that any two nodes n i and n j of the 151 same level construct an information clique if there exists a path between them. Compared with the 152 traditional tree structure, we yoke the nodes of the same level to form a bidirectional complete graph. 153 We call this structure as YNN, which will be introduced in the next section. 154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

3.2 Structure of YNN

Inspired by the neural network of human beings as shown in the Fig 2. In order to enhance the ability of NN to express information, we design cliques for the nodes of each level of a neural network.

Definition 1 A clique is a bidirectional complete graph which considers that for any two nodes n i and n j , an edge exists from n i to n j .

According to this definition, the model in our framework is considered as a bidirectional complete graph for the nodes of the same level. These nodes construct a clique, where every node is not only influenced by its precursor nodes but also by all other nodes of its level. The cliques are represented as information modules which greatly enhance the characterization of NN.

According to the definition of clique, a neural network can also be represented as a list of cliques. Further, we can also introduce a concept of neural module.

Definition 2 A neural module is a collection of nodes that interact with each other.

According to the definition, a neural module can be part of clique. In fact, if all the weights in a clique becomes zero, then the YNN model is reduced to the traditional tree structure.

In each clique of our model, the nodes are first calculated by using their precursor nodes, which only

distribute features. The last one is the output level, which only generates final output of the graph. 170

171

Secondly, each node is also indicated by the nodes of the same level and their values are influenced

level, and the second part contains the nodes of the corresponding clique of the same level. Then,

In summary, according to the above definitions, each YNN is constructed as follow. Its order of 178 outputs is represented as G = { N,E } . For the nodes in the same level, bidirectional complete graphs 179 are built as clique C . Each node n in C is first calculated by using the precursor nodes without the 180 nodes in the clique, which is called as the meta value ˆ n of the node. Then, we calculate its real value 181 n by using the nodes of the clique. 182

According to the meta value and the real value as introduced before, the structure of YNN is shown 183 in the Fig 3. 184

In the next section, we will explain how to calculate the values of the nodes by using the precursor node as well as the nodes in the clique.

3.3 Forward Process

Let we have n elements:

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

̸

̸

as the input data to feed for the first level of ANN. Then, the meta value ̂ N 1 of the first level can be calculated as:

where W 01 is the fully connected weight of the edges between level 1 and input nodes. Then, similarity in nature, for meta value, the full connection between the levels makes the information to flow as:

where N i -1 = { 1 , n i -1 1 , n i -1 2 , ... } , n i -1 j is the real value of the j th node in the ( i -1) th level, number 1 indicates for the bias of the value between the ( i -1) th and i th levels as well as the activation function f .

Then, by introducing weight W i in the i th level and considering the bidirectional complete graph of that level as a clique, we propose a method to calculate the real value N i based on the meta value ̂ N i as introduced in the previous section. Suppose, there are m nodes in the clique and they rely on the values of other nodes. Hence, we need a synchronization method to solve the problem. Here, we take the problem as a system of multivariate equations as well as an activation function f . Then, for the real value of n i j in N i based on the meta value ̂ n i j in ̂ N i , the equations can be summarized as follow:

̸

In the above equations, w i 01 , w i 02 , ..., w i 0 m are the bias of the real values of the nodes in the i th level. 203 Note that, for the mata value, the bias is a value between the levels; while for a real value, the bias is 204 a value in the individual level only. 205

206

207

208

209

Existing numerical methods would be able to solve the above equations efficiently. In the real applications, the efficiency can also be well optimized. In fact, for too large equations, we also propose a method to reduce the calculation scale efficiently. This method is introduced in the following section.

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

3.4 Backward Process

In this section, we introduce the backward process of our model. Firstly, let the gradient of the output be the gradient of the meta value of the last level. We calculate the node gradient for the i th level as:

The meta value of ̂ N i is calculated by using the real value of N i -1 according to the system of equations.

Then, to get the value of d ( ̂ N i -1 ) , we need to consider the nodes as the variables in the system of equations. For convenient, we introduce operator C i to represent the derivatives for the i th level, which can be expressed as:

where W i is the adjacency matrix of the clique in the i th level, diag ( W i ) is the diagonal matrix of W i , eye ( W i ) is the identity matrix whose size is the same as that of w i , and operator C i represents the transfer of other nodes for each node in the clique according to the system of equations. In the clique, the identity matrix is for the node itself.

According to the system of equations, the meta value of a node is connected to its real value through the diagonal matrix of W i . Note that each node is calculated by using the activation function f . As a result, after the transfer through the bidirectional complete graph, the gradient of the meta value of the nodes becomes:

Now, we have got the gradient of the meta value as well as that of the real value of each node. Finally, the gradient weight of the fully connected level W i ( i +1) between the i th and ( i +1) th level can be expressed as:

Now, we need to calculate the gradient of W i for the clique in the i th level. According to the system of equations, we need to consider the weights of all the connected nodes. For any j th node in the clique, its connected weight is the j th column of the matrix. Similarly, for convenient, we introduce the following operator:

which can be found in the system of equations. Then, by the gradient of real value of the j th node n i j in N i , the following becomes the corresponding gradient of the clique:

3.5 YNN Structure Optimization

Consider that for the nodes in the same level, we construct a clique as stated before. Here, we consider a clique just as a universal set for all the possible connections. In our work, we can optimize the YNN structure to let our model to focus on important connections only. The optimization process can be L1 or L2 regularization as usual, which can be parameterized L 1 and L 2 , respectively.

For the j th node in the i th level, the process can be formulated as follow:

According to the L1 and L2 regularization, the L1 parameter can make our YNN to focus on important 241 connections in the clique, and the L2 regularization makes the weight in the clique to be low to make 242 our model to have better generation. 243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

3.6 Structure of Neural Module

According to the forward process of YNN as stated earlier, it solves a system of equations. A large number of nodes in the same level would bring too much computational burden to solve a large system of equations. In Fact, we can optimize the graph of any level by L1 and L2 regularization, and then turn to a minimum cut technology, e.g., the NE algorithm, to reduce the computation significantly. For each cut subgraph, we design a neural module structure according to definition 2 to simplify the system of equations as shown in Fig. 4. Since the nodes are influenced only by the nodes in the subgraph, the system of equations can be reduced to the number of the nodes in the cut subgraph, which is formulated as a neural module as definition 2 in this paper.

In summary, the structure of the neural module can be constructed as follows:

Construct the clique for the nodes in the same level;

Optimize the clique by using the L1 and L2 regularization;

Cut the optimized graph using the NE algorithm;

Construct system of equations by taking each cut subgraph as a neural module.

As explained before, in this way the system of equations can be reduced to Ns-ary equations, where Ns is the number of nodes in each neural module. Of course, if the calculation of our model can be accept for our model, take the clique itself as Neural Module is most accurate, since clique considers all connection in the level.

4 Experiments

4.1 Optimization of Classical ANN

In this section, we will show the experiments with our method. Here, we compare our method with the traditional NN method, stacked auto encoder(SAE), as well as the generalized traditional NN which is a topological perspective to take NN as a DAG graph proposed in recent years.

We show our results for three real data sets. The first dataset contains the codon usage frequencies in the genomic coding DNA of a large sample of diverse organisms obtained from different taxa tabulated in the CUTG database. Here, we further manually curated and harmonized the existing entries by re-classifying the bacteria (bct) class of CUTG into archaea (arc), plasmids (plm), and bacteria

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

proper (keeping with the original label 'bct'). The second dataset contains optically recognized handwritten digits made available by NIST using preprocessing programs to extract normalized bitmaps of handwritten digits from a preprinted form. Out of a total of 43 people. The third dataset is Connect-4 that contains all the legal 8-ply positions used in the game of connect-4, in which neither player has won yet, and the next move is not forced. The outcome class is the theoretical value of the first player in the game.

Here, we compared our method with other methods in terms of a variety of nodes. In this way, we can examine the effectiveness of our model at different levels of complexity of the traditional structure. These nodes are constructed by the NN, SAE, and DAG models. We compared these models in terms of the percentage error. The obtained results are organized in the following Tables, where we can see that our YNN model achieves much better results in most of the cases.

In fact, for all the data sets and a variety of nodes in the same level, our YNN model could tend to get better results after the nodes are yoked together. The effect of our YNN could be improved by optimizing the structure as explained before. All of the first four lines of the Tables are for the results that do not be optimized by the L1 or L2 regularization. We can see that our YNN structure is more efficient even without regularization, compared with the traditional structure.

4.2 Optimization of Structure

In this section, we optimize the structure of our model. Since every structure is a subgraph of a fully connected graph, the initial clique can be a search space for our model. Our model is optimized by using the L1 and L2 regularization, which are effective tools for optimizing structures. The obtained results show that such optimizations can yield better effect.

Here, we study the structure of the model for different L1 and L2 parameters, as shown in Fig. 5. In the figure, the green line represents the results of YNN without optimization, while the blue and red lines are the results for a variety of L1 and L2 parameters, respectively. We can see that such optimization is effective for our YNN in most cases.

We also show the pixel map of the matrix for the clique. In the figure, the black-and-white graph 296 represents the matrix of the fully connected graph for the nodes in the same level. The more black of 297 the pixel means a lower weight for the corresponding edge. 298

According with the decline of the error, we can always seek a better structure compared with the 299 bidirectional complete graph used in our YNN. Besides the L1 regularization, the L2 regularization is 300 also an effective tool to optimize the structure of our model. A larger L2 regularization lowers the 301 weights of all the edges, thus yields more black pixels. However, from the decline of error, we can 302 find that the L2 regularization is also effective to optimize our YNN structure. 303

1

2

3

4

5

6

7

parameter *e-5

0.15

0.2

0.25

0.3

0.35

0.4

error

YNN

L1 regular

L2 regular

1

2

3

4

5

6

7

parameter *e-5

0.1

0.12

0.14

0.16

0.18

error

YNN

L1 regular

L2 regular

1

2

3

4

5

6

7

parameter e-5

0.255

0.26

0.265

0.27

0.275

0.28

0.285

error

YNN

L1 regular

L2 regular

(a) L1 regularization

(b) L2 regularization

5 Conclusion

304

In this paper, we propose a YNN structure to build a bidirectional complete graph for the nodes in 305 the same level of ANN, so as to improve the effect of ANN by promoting the significant transfer 306 of information. In our work, we analyse the structure bias. Our method eliminates structure bias 307 efficiently. By assigning learnable parameters to the edges, which reflect the magnitude of connections, 308 the learning process can be performed in a differentiable manner. For our model, we propose a 309 synchronization method to simultaneously calculate the values of the nodes in the same level. We 310 further impose an auxiliary sparsity constraint to the distribution of connectedness by L1 and L2 311 regularization, which promotes the learned structure to focus on critical connections. We also propose 312 a small neural module structure that would efficiently reduce the computational burden of our model. 313 The obtained quantitative experimental results demonstrate that the learned YNN structure is superior 314 to the traditional structures. 315

References

Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. 317 In: Advances in neural information processing systems. pp. 1097-1105 (2012) 318

Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In:Proceedings of the fourteenth 319 international conference on artificial intelligence and statistics. pp. 315-33 (2011) 320

Perez-Rua, J.M., Baccouche, M., Pateux, S.: Efficient progressive neural architecture search. arXiv preprint 321 arXiv:1808.00391 (2018) 322

323

324

Dai, J., Qi, H., Xiong, Y ., Li, Y ., Zhang, G., Hu, H., Wei, Y .: Deformable convolutional networks. In: Proceedings of the IEEE international conference on computer vision. pp. 764-773 (2017)

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1-9 (2015)

Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510-4520 (2018)

Howard, A., Sandler, M., Chu, G., Chen, L.C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al.: Searching for mobilenetv3. arXiv preprint arXiv:1905.02244 (2019)

Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6848-6856 (2018)

Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700-4708 (2017)

Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V .: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8697-8710 (2018)

Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055 (2018)

Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., Le, Q.V.: Mnasnet: Platform-aware neural architecture search for mobile. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2820-2828 (2019)

Venables, W.N., Ripley, B.D.: Modern applied statistics with S-PLUS. Springer Science Business Media (2013)

Sun, K., Zhao, Y., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., Wang, J.: High-resolution representations for labeling pixels and regions. arXiv preprint arXiv:1904.04514 (2019)

Tang, Z., Peng, X., Geng, S., Wu, L., Zhang, S., Metaxas, D.: Quantized densely connected 361 u-nets for efficient landmark localization. In: Proceedings of the European Conference on Computer 362 Vision (ECCV). pp. 339-354 (2018) 363

Ahmed, K., Torresani, L.: Maskconnect: Connectivity learning by gradient descent. In: Proceed364 ings of the European Conference on Computer Vision (ECCV). pp. 349-365 (2018) 365

366

367

Xie, S., Kirillov, A., Girshick, R., He, K.: Exploring randomly wired neural networks for image recognition. arXiv preprint arXiv:1904.01569 (2019)

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

Rauschecker, J.: Neuronal mechanisms of developmental plasticity in the cat's visual system. Human neurobiology (1984)

Bender, G., Kindermans, P.J., Zoph, B., Vasudevan, V., Le, Q.: Understanding and simplifying one-shot architecture search. In: International Conference on Machine earning. pp. 550-559 (2018)

Guo, Z., Zhang, X., Mu, H., Heng, W., Liu, Z., Wei, Y., Sun, J.: Single path one-shot neural architecture search with uniform sampling. arXiv preprint arXiv:1904.00420 (2019)

Ghiasi, G., Lin, T.Y ., Le, Q.V .: Nas-fpn: Learning scalable feature pyramid architecture for object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 7036-7045 (2019)

Liu, C., Chen, L.C., Schroff, F., Adam, H., Hua, W., Yuille, A.L., Fei-Fei, L.: Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 82-92 (2019)

Huang, Z., Wang, N.: Data-driven sparse structure selection for deep neural networks. In: Proceedings of the European conference on computer vision (ECCV). pp. 304-320 (2018)

Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems. pp. 1135-1143 (2015)

Kun Yuan, Quanquan Li,Jing Shao, and Junjie Yan Learning Connectivity of Neural Networks from a Topological Perspective. ECCV 2020

Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in Graph domains. International Joint Conference on Neural Networks 2 (2005), 729-734.

Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and G. Monfardini. 2009. The Graph Neural Network Model. IEEE Trans. Neural Networks 20, 1 (2009), 61-80.

Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. 2017. Protein interface prediction using graph convolutional networks. Advances in Neural Information Processing Systems 2017Decem, Nips (2017), 6531-6540.

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019. Graph neural networks for social recommendation. The Web Conference 2019, WWW 2019 (2019), 417-426.

T. Young, D. Hazarika, S. Poria, and E. Cambria. 2018. Recent Trends in Deep Learning Based Natural Language Processing. IEEE Computational Intelligence Magazine 13, 3 (2018), 55-75.

Zhipu Xie, Weifeng Lv, Shangfo Huang, Zhilong Lu, and Bowen Du. 2020. Sequential Graph Neural Network for Urban Road Traffic Speed Prediction. IEEE Access 8 (2020), 63349-63358.

Sweah Liang Yong, Markus Hagenbuchner, Ah Chung Tsoi, Franco Scarselli, and Marco Gori. 2006. Document mining using graph neural network. In International Workshop of the Initiative for the Evaluation of XML Retrieval. Springer, 458-472.

Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local Neural Networks. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2018), 7794-7803.

Xiangyang Ju, Steven Farrell, Paolo Calafiura, Daniel Murnane, Prabhat, Lindsey Gray, et al. 2019. Graph Neural Networks for Particle Reconstruction in High Energy Physics detectors. In Second Workshop on Machine Learning and the Physical Sciences (NeurIPS 2019).

Krzysztof Rusek and Piotr Cholda. 2019. Message-Passing Neural Networks Learn Little's Law. IEEE Communications Letters 23, 2 (2019), 274-277.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new method called YNN that transforms traditional ANN structures into yoked neural networks, promoting information transfer and improving performance. The authors analyze the existing structural bias of ANN and propose a model YNN to efficiently eliminate such structural bias. In their model, nodes carry out aggregation and transformation of features, and edges determine the flow of information. They further impose auxiliary sparsity constraints to the distribution of connectedness, which promotes the learned structure to focus on critical connections. Finally, based on the optimized structure, they also design a small neural module structure based on the minimum cut technique to reduce the computational burden of the YNN model. The learning process is compatible with the existing networks and different tasks. The obtained quantitative experimental results reflect that the learned connectivity is superior to the traditional NN structure.\n\nSTRENGTHS:\n1. YNN promotes information transfer significantly which helps in improving the performance of the method.\n2. The authors propose a model that efficiently eliminates structural bias in ANN.\n3. The authors design a small neural module structure based on the minimum cut technique to reduce the computational burden of the YNN model.\n\nWEAKNESSES:\n1. There is a lack of ablation study, e.g., comparing the model performance using different clique size/number of cuts\n2. The equations presented in Section 3.3 are excessively complex and challenging to comprehend. The authors have employed \"W\" and \"w\" for too many different variables in their notations, leading to confusion.\n\nQUESTIONS:\nThe authors seem to be putting forth a novel technique for calibrating hidden states within each layer, which is delineated through the solution of a system of equations, as presented at the bottom of page 5. One might question how this proposed approach contrasts with conventional techniques such as Graph Neural Networks (GNN) or self-attention mechanisms.\n\nDrawing a parallel, it appears that the essence of this newly proposed method bears resemblances to the fundamental principles of GNNs and self-attention methods, notably the propagation of information from one node to its neighboring nodes. Therefore, a comprehensive analysis of the contrasts and similarities of these approaches would be beneficial for a deeper understanding of this proposition.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposed a module called YNN that could exchange the information of the neurons within the same layer. The proposed module can be combined with MLP. The experiments on several small-scale datasets show that their method achieves good performance compared to previous networks.

STRENGTHS:
- The motivation of this paper is valid and interesting.

WEAKNESSES:
- I don't think a fundamental difference between the proposed YNN and graph neural networks. This method can be a special case by assigning a fully connect adjacent matrix to a GNN.
- The evaluation is only conducted on small-scale datasets.

QUESTIONS:
- How deep can this YNN be? Typically this kind of network will suffer the information diminishing problem since the feature will be over-smoothed with depth increasing.
- Can this network generalize to large-scale datasets such as imagenet? If so, could the author show some experimental results on it.
- What's the fundamental difference between YNN and GNN?

LIMITATIONS:
Please refer to the weakness section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a 'yoked' neural architecture where neurons at the same level are bidirectionally linked. They claim that optimizing this complete graph is superior to current deep neural network architectures that impose a structural bias due to the transfer of knowledge in a way that prevents structural bias.\n\nSTRENGTHS:\n* the proposed optimization changes is simple in that it is similar to other methods like DARTS that grow neural networks and regulate their connections, assigning weights to them using ANN optimization algorithms with regularization term \n* it is clear how the forward propagation is done with the addition of the clique nodes that are computed in addition to the regular precursor nodes at each layer\n\nWEAKNESSES:\n* some of the terminology and abbreviations need to be defined / explained in the first appearance in the intro paragraphs (e.g. 'yoked', ANN, and DAG)\n* the method of optimization doesn't seem particularly novel, employing both an L1 and L2 term to search for the best architecture. \n* the figure 2 presented does not particularly show much about the method or its justification\n* Is it possible to approximate the non-differentiable minimum cut algorithm and absorb it into the training procedure? This would be similar to progressive training methods like http://proceedings.mlr.press/v119/evci20a/evci20a.pdf and other related works)\n* please proofread for more typos and such (e.g. 'mata' on l204, other grammatical errors)\n* results are on quite toy problems and training with regularization is not necessarily yielding the best results\n\nQUESTIONS:\n* Can the authors comment on the tradeoff between running more gradient updates on a non-yoked network versus the potential representation learning benefits from a yoked version? \n* Is the influence of information flow between nodes of the same level not potentially reflected intrinsically in the next leading level? \n* How does this method compare to other progressing growing methods like the lottery ticket methods or pruning networks?\n* Why is the 25 nodes table 3 DAG result so different from the rest?\n\nLIMITATIONS:\nAuthors do not discuss limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Yoked Neural Networks (YNN) - an extension of neural networks, which, when calculating the value of a node, in addition to the information from the previous layer of the network, uses information from the nodes on the same layer (i.e. it \"yokes\" nodes from the same layer together).\n\nSTRENGTHS:\nThe paper describes the approach well, it is clear how it works.\n\nCode has been provided as an attachment, so that it should be reproducible (I have not ran the code or looked at it carefully).\n\nWEAKNESSES:\n(Details about the mentioned weaknesses are given per line, in the field \"Questions\".)\n\nThe paper would benefit from describing in more details the contributions of the proposed method. Particularly, across the paper, some strong statements have been used, but they have not been motivated with evidence. \n\nOverall, the idea and the benefits of using the method needs to be better motivated.\n\nThe description of the experiments is not very clear.\n\nQUESTIONS:\nLine 2: It is not clear why \"the connectivity of a tree is not sufficient to characterize a neural network\", Since this is one of the most important motivations for the work, it would be beneficial to give details about it.\n\nLine 12: It is not obvious why the method improves, maybe instead of using the word \"obviously\", some proof of the statement could be given - how does the method improve over existing methods, by how much?\n\nLine 12-13: \"YNN can imitate neural networks much better\" - Which properties do they imitate better? What does better mean in this context?\n\nLine 43: \"limiting the signal’s capability for free transmission\" - What does this mean? What is free transmission and why is it limited?\n\nLine 45-46: \"significant drawbacks\" - Can you give a citation to prove this claim, what kind of drawbacks? In this paragraph, it is not really clear what limitations of ANNs you are addressing, can you please be more specific. What \"substantial defects\" do you refer to?\n\nLine 52: It is not obvious what \"YOKE fashion\" is, can you please provide a short explanation?\n\nLine 54: Instead of using \"remarkable improvements\", could you please give details about what kind of improvements you have obtained - in what measures, and by how much.\n\nLine 59: \"our method efficiently eliminates structural bias\" -  How does it eliminate it? Please define structural bias, how it is eliminated and what does \"efficiently\" mean. Again, please give details of what exactly is improved and how.\n\nLine 94: Please provide a citation for ResNet.\n\nLine 139: \"some researchers attempted to generalize this structure\" - Please provide a citation.\n\nLine 143-144: \"makes the transformation of information quite inadequate\", \"makes the transformation of information quite inadequate.\" - What does inadequate mean in this context? What needs to be improved? Why is the structure inferior inferior, what kinds of properties are missing or need to be improved?\n\nLine 149: (very minor, typo) \"is\" is not necessary\n\nLine 156: It is not really clear what properties of a neural network have inspired the YNNs. Please give more details and some specifics.\n\nLine 163: It is not clear what you mean by \"greatly enhance the characterization\". Please explain.\n\nLine 181: Is the \"meta value\" the value as it would be from a standard NN pass, without yoking the same layer nodes? And the \"real value\" is the value from the forward pass, together with the values calculated from the other nodes at the same level?\n\nLine 248: \"NE algorithm\" should be explained and a citation given.\n\nLine 249: (minor) The reference to \"Definition 2\" is not linking to it.\n\nLine 259-261 - The last sentence of the paragraph is hard to follow.\n\n\n\nQuestions about the experiments:\n\nIt is not very clear how the experimental setup is done. Are you doing classification, and measuring classification accuracy on the three mentioned datasets? Please give more information about the task(s) you are addressing and what you are measuring.\n\nLine 265: It is not clear what the compared models are. Please give details of the structure of the used \"traditional NN\", \"SAE\" and \"generalized traditional NN\". Also, it is not clear what is the structure of the neural network that you are using for your approach. Is it based on a feed-forward neural network, but with yoked nodes? \nFor all the compared models, please give details about network structure, number of layers, training method, training details. (Maybe this can be seen in the code, but it would greatly increase the understanding of the paper if these details are included in the experiments section.)\n\nIn the tables with results, what is the meaning of the number of nodes in the columns? Is this the number of nodes in one NN layer? How many layers are there? \n\nLine 267 and below: Could you provide a reference to the used CUTG dataset? Can you please give a little more description: is this an existing dataset which you annotated further? Or did you collect this data - if so, can ou please give more details of how it was collected and annotated?\nCan you please give citations for \"the second dataset\" and \"Connect-4\"?\n\nLine 280: Please give a reference to the tables with results for more clarity.\n\nLine 281: Please give a little more detailed comment on the results - how much better are your results, in which cases. What conclusions can be derived from these results?\n\nLine 290: How do you optimize with L1 and L2 regularization? Please give some details. It is not obvious how this optimization is performed and what is the result from it.\n\nLine 295: What does \"effective\" mean in this context - better performance on the classification task, or execution time?\n\nLine 307-308: \". Our method eliminates structure bias efficiently.\" - It is not clear how this is done. The details of what exactly you mean by \"structure bias\", how your method eliminates it, and how this improves the model needs to be more clearly described in the paper.\n\nLIMITATIONS:\nNo limitations have been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a novel neural network model that exploits a connection between the nodes of a layer. The authors’ goal is to develop a model that overcomes the structural bias posed by the classical layer structure of the NN. To do this they propose to consider the model as a bidirectional complete graph for the nodes of the same level and to define for each layer a clique. \nThe authors then test the proposed architecture and compare it with the traditional NN model considering 3 datasets.\n\nSTRENGTHS:\nThe paper discusses an interesting problem and proposes a novel methodology that seems promising.\n\nWEAKNESSES:\nOverall, the paper is challenging to read, and at times, the concepts being discussed are not adequately introduced, causing difficulty for the reader to comprehend the discussion's progression. Additionally, several critical concepts are unclearly defined.\n\nIn the introduction, the authors discuss the neural module without explaining what it is. Even the concept of “yoke”, which is central to the discussion, is not adequately introduced and explained in the context of neural networks.\nIn the introduction the authors also discuss the impact of the sparsity constraints, also in this case this concept has to be defined and explained to the reader.\nIn the list of contributions, point 4 says that the authors designed a regularization-based optimization, which at this point of the paper is very difficult to understand. Point 5 of the same list discusses the problem of computational complexity, which also in this case is not discussed before.\n\nAnother issue is the experimental campaign where the experimental setting and the metric used to perform comparison are not explained. Indeed the authors use the “variety of nodes” as a metric, but they do not explain why it has to be significant to show the advantage of the proposed approach.\nFrom the tables, 1,2,3 seems the authors fix the number of nodes for the various architectures and train them. In general to me, it does not seem a fair way to compare the models, mainly for 2 reasons: (i) the architectures of the baselines have to be validated (in particular in terms of the number of neurons, but also considering the other hyperparameters of the model and of the optimization algorithm) in order to find the most suitable setting for the task. (ii) the comparison has to consider the number of parameters of the model since the structure of the  YNN will have many more weights than a standard model (fixing the number of neurons).\nEven a description of the setting of the three proposed approaches (YNN, YNN&L1 ,YNN&L2) would make it easier for the reader to understand the proposed results.\nFinally, a discussion about the computational burden and a comparison with standard NN is missing.\nThe experimental evaluation and the discussion of the obtained results should be significantly improved and extended\n\nQUESTIONS:\nIn Section 2 the first part discuss models that have an architecture that is slightly different than the standard ANN but seems none of them exploit the connection between neurons of the same level, thus I wonder how these cited models are placed in comparison to the YNN (theoretically and in terms of performance) because to me it is not clear if the benefit of the proposed architecture come from the differentiable structure or/and from the richest connection pattern between neurons.\nIn line 103 the authors state that the proposed learning process method is consistent with DARTS. In my opinion, the authors should provide further elaboration on this point as it appears unclear to me.\n\nIn general section 2 suffers from the same problem of the introduction, since it cites the proposed approach and algorithm, but who read the paper do not know anything about them at this point, therefore my suggestion is to move the entire section two after the explanation of the proposed model.\nIn section 3, from lines 133 to 138 the authors discuss the similarity between the ANN structure and the tree structure, but in general, I find it not correct to state that a node in a NN is influenced only by its precursor node, since in a standard NN all the neurons on the previous layers influence the output of the current node. The authors should clarify this point to make it more clear how this description fits the typical NN architecture. \n\nIn section 3.2 the authors use the concept of clique,  which is a concept related to graph theory but not that common in ML in general, therefore in my opinion definition 1 should be extended to explain this concept a bit more in depth.\nEven the concept of “node” (that in the first part seems to be synonymous with “neuron” ) should be defined more precisely. \nFor what concerns the forward pass to me it is not clear how the authors solve the system proposed at the end of page 5 considering that the values of n_j^i, which are part of the summation of each row, are also the result of the other system equations.\nIn the backward pass, in particular, in eq. 4, the authors define how they compute gradients for each level. Since in this equation appears f^{-1} I am wondering if this method works only with an invertible activation function  (that could be a strong constraint)\n\nIn section 3.5 it is not discussed why the use of L1 and L2 is important to perform structure optimization.  Moreover, the meaning of w^i in eq.18 is not clear. previously they were defined as the bias of the real values of the nodes in the i-th level while here they are applied as a function.\nIn section 3.6: missing definition (or reference to) NE algorithm.\nThe authors also state that imposing auxiliary sparsity constraints to the distribution of connectedness optimization promotes the learned structure to focus on critical connections. This is a very interesting point but to me seems a theoretical or empirical proof is missing.\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}