Polarity is all you need to learn and transfer faster
Authors: Qingyang Wang
Michael Alan Powell
Ali Geisa
Eric W Bridgeford
Joshua T Vogelstein
Venue: Submitted to ICLR 2023 (2023)
Keywords: Weight Polarity
Learning Efficiency
Transfer Learning
Bio-inspired AI

Abstract: Natural intelligences (NIs) thrive in a dynamic world – they learn quickly, sometimes with only a few samples. In contrast, Artificial intelligence (AI) has achieved supra (-human) level performance in certain AI settings, typically dependent on a prohibitive amount of training samples and computational power. What design principle difference between NI and AI could contribute to such a discrepancy? Here, we propose a research avenue based on a simple observation from NIs: post-development, neuronal connections in the brain rarely see polarity switch. Why? Our answer is: to learn and transfer more efficiently. We demonstrate with theory and simulations that if weight polarities are adequately set $\textit{a priori}$, then networks learn with less time and data. We extend such findings onto image classification tasks and demonstrate that polarity, not weight, is a more effective medium for knowledge transfer between networks. We also explicitly illustrate situations in which $\textit{a priori}$ setting the weight polarities is disadvantageous for networks. Our work illustrates the value of weight polarities from the perspective of statistical and computational efficiency for both NI and AI.

Under review as a conference paper at ICLR 2023
POLARITY IS ALL YOU NEED TO LEARN AND TRANSFER FASTER
Anonymous authors Paper under double-blind review
ABSTRACT
Natural intelligences (NIs) thrive in a dynamic world - they learn quickly, some- times with only a few samples. In contrast, Artificial intelligence (Al) has achieved supra (-human) level performance in certain AI settings, typically dependent on a prohibitive amount of training samples and computational power. What design prin- ciple difference between NI and Al could contribute to such a discrepancy? Here, we propose an angle based on simple observation from NIs: post-development, neuronal connections in the brain rarely see polarity switch. We demonstrate with simulations that if weight polarities are adequately set priori, then networks learn with less time and data. We extend such findings onto image classification tasks and demonstrate that fixed polarity, not weight, is more effective medium for knowledge transfer between networks. We also explicitly illustrate situations in which priori setting the weight polarities is disadvantageous for networks. Our work illustrates the value of weight polarities from the perspective of statistical and computational efficiency during learning.
INTRODUCTION
Natural intelligences (NIs), including animals and humans, thrive in dynamic world. Often, NIs learn quickly with just a few samples. Artificial intelligences (Als), specifically deep neural networks (DNNs), can now compete with even surpass humans in certain tasks, e.g., GO playing (Silver ct al. 2017), object recognition (Russakovsky et al. 2015), protein folding analysis (Jumper et al. 2021). ) etc. However, DNN is only capable of achieving such when a prohibitive amount of data and training resources are available. Such gap on learning speed and data efficiency between NI and AI has baffled and motivated many AI researchers. A subfield of AI is dedicated achieving few-shot learning using DNNs (Hoffer & Ailon 2015 van der Spoel et al. [015]] Vinyals et al. 2016) Snell et al. [2177] Finn et al. (017). Many research teams have achieved amazing performances on benchmark datasets (Lazarou et al. 2022: Bendou et al. (022). However, the products of such engineering efforts greatly deviate from the brain. What are the design principle differences between NIs and AIs that contribute to such learning efficiency gap? In this paper, we propose one possiblility of such a design difference we could move AI one step closer to NI-level learning efficiency by applying just one simple design principle from NI.
NIs are blessed with hundreds of millions of years of optimization through evolution. Through trial and error, the most anvvival-advantageeee into the form that can thrive in an ever-changing world. Such circuit configurations get embedded into our genetic code, establishing blueprint to be carried out by development. Among the many configurations, circuit rules, and principles that formed through evolution, one theme stands out, one that neuroscientists celebrate and yet is overlooked by the machine learning community: post- development, neuronal connections in the brain rarely see polarity switch (Spitzer 2017). After development, NIs learn and adapt through synaptic plasticity - a connection between a pair of neurons can change its strength but rarely its excitatory or inhibitory nature; on the contrary, connection (weight) between a pair of units in DNN can freely change its sign (polarity). In fact, polarity change in the adult brain is hypothesized to be associated with depression, schizophrenia, and other illnesses (pitzer 2017). For the rare times such phenomenon have been observed, they never appeared in sensory and motor cortices (Spitzer 2017) where visual, auditory and motor processing take place. It seems a rather rigid design choice to fix a network's 's connection polarity. We wonder why the biological networks settled into such learning strategy: Is it mere outcome of inplementation-
circuit configurations emerge, refine, and slowly come

---

Under review as a conference paper at ICLR 2023
level constraint? It could be just hard for synapses to switch polarities. Or could it be that NIs found out polarity pattern is an effective medium for transferring knowledge across generations?
This paper provides some thoughts and evidence relevant to these questions. We first investigate why brains pre-set neuronal connection polarities by assessing what we gain through setting weight polarity priori (Sec 2 We discuss in theory the trade-off' between representation capacity and learning speed when weight polarity is fixed for networks. We then propose an SGD-base based polarity-fixed learning algorithm: Freeze-SGD. We experimentally show that if the weight polarities are adequately set a priori, then networks can learn with less time and data (simulated task (Sec 2 + two image classification tasks (SecB).. We call a network with fixed polarity Frozen-Net, and we discuss how the quality of the polarity configuration affects a DNN's learning efficiency (Sufficient- Polarity vs. RAND-Polarityy. We further find transferring and fixing polarities is even superior to transferring weights (Sec Our results point to an unexplored direction in the machine learning community: polarity, not weight, may be the more effective and compressed medium for transferring knowledge between networks. To complete our discussion, we further discuss what we lose when weight polarities are set a priori ( Frozen-Nets have reduced representation capacity; therefore, randomly configured network may not even have the capacity represent a simple problem such as XOR (Def (2.2). We theoretically prove and experimentally show that if the polarities are set randomly, the probability of higdee--ddee-e network learning XOR increases exponentially as a function of its size (i.e., number of hidden units). We also experimentally show that sufficiently sized network, even when its polarities are randomly picked, can learn with less time and data than an equally sized network without fixed polarities.
By discussing both the advantages and disadvantages of fixing weight polarity, we provide some insights on how to make AI more statistically and computationally efficient; we also show polarity pattern is an effective medium for transferring knowledge.
WHAT DO WE GAIN BY SETTING WEIGHT POLARITY a priori?
Networks need both positive and negative weights to funcion (Wang et al. 2022 DNN with all non-negative weights is not a universal approximator. Constraining network's 's weight polarity pattern limits its representation capacity: when only half of the range is available to each connection, the reduction in total possible network patterns grows exponentially with more edges in the network. seems counter-intuitive for any network to have willingly chosen to give up on a vast portion of its representation capacity. Are they gaining elsewhere? Our thought is: maybe they learn faster. Below we provide theoretical discussions and experimental evidence.
Definition 2.1 (DNN). Let ((I) E B be the input weights of layer I, ny be the number of units in layer (0) E "R be the bias terms of layer I, for E (I,..., L}. 1 is the input layer, no = dim(x:). Let layer 1 be (00) (x) o oW b(0).
P(x) = (5) ( (1- .... o (x), e R"
F(x) softmax softmax(P(x))
DNN is F(x). For the non-rinearity a. we define it to be ReLU throughout the paper.
0
0
a (x,)
= (o)=))
x
I 00
Lemma 2.1 (capacity-speed trade-off). If the weight polarities are set a priori, such that the function is still representable, then the network can learn faster.
We prove Lemma 2.1 for single-hidden-layer networks, with the following assumptions:
Assumption 1: The weights take on discrete values. This is essentially true for all DNNs implemented on silicon chips where all continuous variables are discretized;
Assumption 2: Exhaustive search as the learning algorithm.
See proof on page 20
The theory, albeit proved under constrained settings, argues for a trade-off between network rep- resentation capacity and learning speed. Next, test with simulation show that networks
2

---

Under review as a conference paper at ICLR 2023
learn more quickly when polarities are set priori in such way that the function is still repre- sentable (Figure panel B). With input space X', a function f is representable by DNN F when Vrr E x > 0, (z)) F(x) <.
We design our freeze training procedure to be exactly the same as SGD (Adam optimizer) except for one single step: after each batch, all weights are compared to the preset polarity template, and if any of the weight has switched polarity in the last batch, it is reverted back to the desired sign (see algorithm|] DT. As our goal is to see the pure effect of fixing weight polarity, we did not adopt any bio-plausible learning algorithms as they may introduce confounding factors.
We compared four training procedures in general:
1. Frozen-Net Pefficient-Polarride
Weight polarities were set priori. The polarity pattern was chosen based on a rule that ensures the configuration is adequate for learning the task, i.c., the polarity pattern carries expert knowledge about the data.
2.
Frozen-Net RAND-Polarity: Weight polarities were set a priori randomly Bernoulli(0.5) The polarity pattern does not carry any information about the data.
3. Fluid-Net RAND-Polarity: Weights (containing polarities) were initialized randomly;
weight polarities were free to change throughout the training procedure.
4. Fluid-Net Sefficeent-Pllarit Weight polarities (not magnitudes) were initialized with prior knowledge; weight polarities were free to change throughout the training procedure. This scenario will only be discussed in the next section 3 on image classification tasks.
We used 5-dimensional XOR (XOR-5D) as our simulation binary classification task; only the first two dimensions relevant to the task, and the remaining three dimensions are noise following a normal distribution template is pre-set this way: for each hidden unit, the polarity of the output edge is the sign product of the first two dimensions' input weights.
(Figure [ panel A). For Frozen-Net sufficient-Polariiy the polarity
Definition 2.2 (XOR). Let x E R2 and f1, E R.
f(x) - {
[ 2 41 and D2 2 [2] or
[r] < E and D2 < [2] [ < 11 and 22 2 (2).
(1)
[ 2 4 and
12] or
To have the best controlled experiments, we fixed the magnitude distribution, architecture, training samples (n varies, see figure), batch sequence of the training data, and the validation samples (n=1000) across all scenarios. More details about the experiments can be found in the appendix See[C] We first tried four different weight reset methods (details in Algo D and found they give us similar results and thus did not matter for our primary questions (supplementary figure B.1 For the rest of this paper, we chose the posRand method whereby we set weights to a small random number of the correct sign.
When the polarities are fixed in such a way that it is sufficient to learn the task (red), networks always learn faster than networks without polarity constrains (blue) (Fig I panel B). This advantage is true across all data scarcity levels and is particularly valuable when data is scarce. When only 60 or 72 training samples were available, Frozen-Net sufficient- poolariyy on average take 58% and 48% of the standard Fluid-Net training time respectively reach the same level of accuracy, When weight polarities are randomly chosen (green), networks learn as fast as their Fluid-Net counterparts (blue), and sometimes faster (e.g., training samples 80 and 92).
Frozen-Net sufficient-Pllarity not only saves time, but also saves data (Figl I panel C). At convergence, Frozen-Net sufficient-Pollae Polarity (red) takes fewer samples to reach the same level of accuracy when compared to the standard Fluid-Net (blue). Randomly configured Frozen-Net (green) uses similar and oftentimes less data than Fluid-Net (blue) (e.g., to reach 80% accuracy, Frozen-Net RAND-polarrty uses less data than Fluid-Net).
We showed that setting weight polarity a priori makes networks learn in less time, with less data, provided the function is still representable. Even randomly configured Frozen-Nets show comparable and sometimes better performance than Fluid-Nets. Such a result is striking from an optimization perspective: Frozen-Ne Net is at disadvantage by design because the weight resetting step in Freeze- SGD fighht against the gradient update- part of the error update information lost in this process. Regardless of this disadvantage during optimization, our Frozen-Net sufficient-polarity consistently
3

---

Under review as a conference paper at ICLR 2023
outperforms Fluid-Net; even Frozen-Net RAND-Polaris is never worse than Fluid-Net. Combined, these results show we may help AIs to learn quickly and with fewer samples by doing two things: 1) fix weight polarities, and 2) choose polarity pattern wisely. We will tease apart the effect of these two factors in the next section.
A]
B)
Computational Efficiency C)
Statistical Efficiency
reach 90% accuracy
50
Fluid
Freeze sufficient-ellaity
Validation error %
50
#epochs to
freeze HaND-Palanity
X3. X4, Xs
25
25
5
#
100 # samples
500
8
100
500
samples
Figure 1: Adequately setting weight polarity a priori makes networks learn more quickly (fewer epochs) and with less data (fewer training samples). Single hidden layer networks with 64 hidden units are trained to learn XOR-SD. In all scenarios, networks were trained for 100 epochs. A) XOR-5D, a binary classification problem where only the first two dimensions task (XOR), and the other three dimensions are noise following normal distribution (1,, 1). B) Computational efficiency: Setting weight polarity in a data intormnnfrmee way (red) makes networks learn more quickly this is true across all data-scarcity levels; when weight polarities are randomly fixed (green), networks learn as fast as their fluid counterparts (blue), and sometimes faster (e.g., training sample sizes 80 and 92). C) Statistical efficiency: At convergence, adequately configured freeze networks (red) achieve the same level of performance (% error) with less data; randomly configured freeze networks (green) use similar and oftentimes less data than fluid (blue), e.g., to reach 80% accuracy (20% error). For all experiments, n=50 trials. All curves correspond to medians with shaded regions 25th percentiles.
relevant to the
EFFECTIVENESS OF FROZEN-NET IN IMAGE CLASSIFICATION TASKS
To prove the effectiveness of our Frozen-Net strategy for more complex tasks, we extended the experiments in Fig [oo image classification tasks (Fig 2 Such complex learning tasks do not have simple and explicit polarity configuration rules XOR-5D. We exploited an alternative strategy by using ImageNet trained polarities (IN-Polarity). A Frozen-Net IN-Polarity has its weight magnitudes initialized randomly, its weight polarities initialized and fixed to match the IN-polarity pattern. We also tested Fluid IN-Polarity where networks were initialized to the exact same pattern Frozen-Net IN-Polarity, except polarities are free to switch this scenario. This pair of comparison help us to understand which of the two factors contributed more the performance gain: fixing polarities or knowledge transfer through polarity pattern. We trained and tested networks on the Fashion-MNISTT (grayscale) and CIFAR-I0 (RGB-color) datasets, using AlexNet network architecture (Krizhevsky et al. 2017). For both datasets, we trained for 100 epochs, with 1r=0.001. The AlexNet IN-weights were obtained here Worth emphasizing, controlled all conditions to follow the same weight magnitude distribution at initialization (supp Fig B.6) Specifically, we randomly initialized the networks following normal procedures: Glorot Normal for conv layers, Glorot Uniform for fe layers; then either fixed the polarities as is (RAND-Pot (AND--tolrity or flipped the polarities according to IN template (IN-Polarity), introducing no change the magnitude distributions.
Across the board, Frozen-Nets IN-polarity (red) always learn with fewer samples than Fluid-Net (blue) (Fig 2 first column). When only 100 training images were available (across the classes), Frozen-Net IN-Polarity (red) yields 7% less validation error at convergence compared to Fluid-Net (blue) in the Fashion-MNIS" task and 9.4% less error for CIFAR-10. Such a gain is mostly brought by knowledge transferred through polarity pattern (pink vs. blue, 6% gain for Fashion-MNIST 8.4% gain for CIFAR-10). Fixing the polarities can further bring performance gain: for CIFAR-10. 1% gain 100 training samples, and up 3% 50000 training samples. The gain from Fluid IN-Polarity
4

---

Under review as a conference paper at ICLR 2023
Freeze-Net IN-Polarity be statistically significant (Fig pink line). When the polarities fixed randomly (RAND-pone -oll-rto,,,ty, green), networks never perform worse than Fluid-Net (blue).
A)
Statistical Efficiency
Success Rate
Computational Efficiency
Volidation error %
50
Fluid RAND Polarity
IN-Polanty
validation accuracy
100
reach 80% accuracy
100
Fashion-MNIST
RAND Polarity
% trials reached
Fluid IN- Polarity
Wepochs to
25
50
2oo
1000
6000
100
1000
6000
900
1000
6000
samples
samples
samples
B)
100
50% validation accuracy
100
reach 50% accuracy
Validation error
% trials reached
CIFAR-10
50
#egochs
30
200
5000
50000
100
5000
50000
100
5000
50000
samples
samples
samples
Figure 2: DNNs with frozen IN-Polarity learn more quickly and with less data in image classifi- cation tasks. A) Experiments on MAshIT image classification dataset. From left to right: 1) Statistical efficiency: Frozen-Nets with IN-polarity (red) always learn with fewer samples than Fluid-Net (blue); the majority of the gain contributed by the knowledge transferred from initial polarity configuration (pink blue); Frozen-Nets RAND-Polarie (green) never perform worse than Fluid-Net (blue). 2) Frozen-Net IN-Polarity always have a higher chance of reaching 80% validation accuracy than Fluid-Net; Frozen-Net RAND-Polarity have comparable and sometimes a higher chance of reaching 80% validation accuracy than Fluid-Net. 3) Computational efficiency: Frozen-Net IN-Polarity always take less time than Fluid-Net reach 80% validation accuracy; again, effective knowledge transfer from preset polarity pattern is the major contributing factor (pink vs. blue): RAND-Polarity takes similar number of computational iterations as Fluid-Net. B) Same as A except experiments were CIFAR- 10 dataset and the validation accuracy threshold 50%. Gray lines the first column correspond to the validation accuracy thresholds used to plot the next two columns. For comprehensive view of performances at different thresholds, see Supp Fig B.2 For statistical significance the difference, see Fig Both datasets: n=20 trials, 100 epochs, 0.001. No data augmentation performed.
Across the board, Frozen-Net IN-Polarity always learns with less time than Fluid-Net (Fig third column). Majority of such gain is brought by polarity pattern knowledge transfer. Frozen-Net RAND-Polarity takes a comparable number computational iterations as Fluid-Net. Furthermore. not every network is able to reach the specified accuracy threshold (Fashion-MNIST) 80%, CIFAR- 10: 50%; Fig 2 second column). Across the board, Frozen-Ne IN-Polarity has a higher chance of passing the specified accuracy threshold than Fluid-Net; Frozen-Ne RAND-Polarity has an equal and sometimes higher chance than Fluid-Net (Fashion-MNIST 1000 2000 samples; CIFAR-10 10 10000 & 25000 samples). These observations are true across different validation accuracy thresholds (Supp Fig B.2
Worth noting, IN-Polarity initialized networks in general shows more consistent performance across trials, compared to the Fluid setting, for both Frozen-Nee and Fluid-Net. This especially obvious for the statistical efficiency plots: across the board, IN-Polarity always shows less variation in its performance on validation error (shaded area marks 25th_75" percentiles).
To provide lens into the dynamic of polarity switch throughout learning, analyzed the ratio of weight parameters (excluding bias terms) that had polarity switch between two epochs for Fluid RAND-Polarity for the first 50 epochs indeed, there more polarity switch early in training
5

---

Under review as a conference paper at ICLR 2023
and the ratio decays throughout training. Such trend is true across layers and across training sample sizes. These suggest polarities are mostly learnt early on during training but also remain dynamic throughout the learning process.
In sum, we are able to extend our simulation results in Sec to more complex image classification tasks: Frozen-Ner Net IN-polarity consistently learns with less data and time and does so with higher probability of success compared to Fluid-Net; the majority of the performance gain is brought by knowledge embedded in the initialized polarity pattern, with further gain possible by fixing weight polarities; Frozen-Net RAND-polarity perform as well as Fluid-Net, sometimes better.
4 TRANSFERRING AND FIXING POLARITY IS SUPERIOR TO TRANSFERRING WEIGHTS
From a transfer learning perspective, our Frozen-Net strategy essentially transfers weight polarities instead of weights per se. How does polarity transfer compare the traditional finetune (weight transfer) strategy? This time, instead of randomly initializing the weight magnitudes, we initialized the weight magnitudes based on the ImageNet-trainee trained weights (IN-Weine Weight), We compared them with Frozen-Net IN- IN-Polarity by plotting their differences (A=(LEGEND): [Freeze IN-Polarity)) }) in Fig (the original curves before taking the differences can be found Supp Fig B.3)
The orange curves compare weight transfer (Fluid-Net IN- Ieightght) with our polarity transfer strategy (Frozen-Net IN-Polarity). Across almost all data scarcity levels, our polarity transfer strategy achieves higher accuracy than finetune (first column, orange> 0). Such a superiority is statistically significant
Statistical Efficiency
Success Rate
Computational Efficiency
A)
A Valication error %
10-
80% validation accuracy
100
A #epochs to 80% accuracy
20
trials reached
Fashion-MNIST
0
100
1000
6000
100
1000 samples
6000
100
1000
6000
samples
samples
B)
Validation error %
valicaation lio eccuracy
100
Frouse Folur by Fuic RAND Polartty
a #apochs to reach 50% accuracy
30-
Fuld Polority trusze niwwiight
50% trials reached
25
Ful Weight
CIFAR-10
15
100
5000
50000
100
5000
50000
100
5000
50000
sam ples
samples
samples
Figure 3: Across all sample sizes, Frozen-Net with ImageNet-polaritie (IN-polarities) learns more quickly than Fluid-Net with -etight (IN-weight) initialization. In first and third columns, curves are the median difference (=(LEGEND) {Freeze IN-Polarity) (the second term is the red curve in Fig 2). Asterisks (+) indicate statistical significance with Mann- WhinnWW U two-tail test (a=0.05). 1) First columm: Transferring and fixing polarity is more effective than transferring weights in terms of data efficiency. This is indicated by orange curve above zero across most sample sizes, and it is statistically significant for small sample sizes. Note the zero line here means LEGEND Freeze IN- IN-Polarity. 2) Second column: Frezen-Net IN-Polarity has higher chance of reaching 80% (Fashion-MNNSTT / 50% (CIFAR- 10) validation accuracy compared to weight transfer (orange curve); 3) Third column: Frozen-Net IN-Polarity always takes less epochs to reach high validation accuracy than weight transfer (orange curve). Both datasets: n=20 trials, 100 epochs, 1r=0.001. For the right two columns, the validation accuracy thresholds are the same as Fig

---

Under review as a conference paper at ICLR 2023
(Mann-Wiil Whitney U two-taill when training data is limited (FaShion-MNIST.: & 2000 samples except 750, 10: < 1000 samples). Polarity transfer also allows the networks to learn with higher probability (second column) and fewer epochs (third column, curve 0). Such faster learning is true regardless of the performance threshold value (Supp Fig B.4). In sum, transferring and fixing polarity is almost always superior to weight transfer both in terms of statistical efficiency and computational efficiency. Such an observation suggests polarity configuration is an effective medium, if not superior to weight pattern, for transferring knowledge between networks.
When networks were transferred with polarities but not frozen (Fluid IN-Polarity pink), they almost always perform better than Fluid IN- Weight (orange, Fig & Supp B.3 to see variance), this is true except rare cases (e.g. CIFAR-10 10 2500&5000 samples) where the difference is not significant due to the wide performance variation of Fluid IN- Weigtt
Surprisingly, when we initialized the Frozen-Net with IN-weight (cyan), there is some gain in performance, but to limited extent; in fact, can sometimes be worse. When training data limited & 1000, CIFAR-10 1000), Frozen-Net IN-weight gained little performance (cyan==0) and could be detrimental at times (CIFAR- 10 500& 1000 samples). When training data is more abundant, there is more consistent accuracy gain by initializing Frozen-Net with WN-wht (Fashion-MNIST gain ~ 1%. CIFAR-10 gain ~ 4%). Such a gain is discounted by more training iterations (third column), and sometimes less likelihood of reaching high level of performance (second column).
Furthermore, similar to random initialization, weight transfer tend to have wider performance variations compared to polarity transfer, for both Frozen and Fluid networks (Fig[B.3). The exact reason behind this observation remains to be explored, our current hypothesis is the stochasticity of sample batching: by only transferring polarity while initializing the magnitudes randomly, the learning process is more robust against such stochasticity. The difference in initialized magnitude distribution between polarity transfer vs. weight transfer could also be potential contributing factor (supp Fig B.6).
In sum, polarity transfer is superior to weight transfer in most of the scenarios we tested and such a superiority further secured by fixing the polarities throughout training. To a large extent, weight polarity alone, not weight per se, is enough to transfer knowledge between networks. Giving the additional magnitude information to Frozen-Net can give some performance gain, but only when data and time is abundant; in all other scenarios (i.e., data-limited or time-limited), initializing Frozen-Net with stereotypical weight magnitudes could be detrimental to the learning performance.
5 WHAT DO WE LOSE BY SETTING WEIGHT POLARITY a priori?
Intelligent agents have limited resources in 1) data, 2) time, 3) space (number of hidden units or other network size parameters), and 4) power (xxtime xspace, number of flops). In Sec 23 we showed that fixing weight polarity helps to save on two of these resources, time and data, but with condition - the polarity configuration must be adequately set such that the network can still represent the function. With fixed polarity, certain configurations will result in networks never being able to learn the task. What is the probability of such unfortunate events happening? We investigate this direction with our simulated task XOR-5D.
Assuming unlimited resources (i.e., perfect learning algorithm, unlimited data and time), we deduced the theoretical probability limit for a ingle-hidden-layer network to be able to represent XOR (and its high- nigh-dimensional variants with task-irrelevan higher dimensions) as function of the number of hidden units (Supp Theorem D.I). The theoretical results are plotted in Fig[paanl (top). In theory, it takes at least 3 units for Fluid-Net and Frozen-Net sufficient-PP Polarity to be able to learn XOR-5D on every single trial. Such a probability will never be 100% for Frozen-Net RAND-Polarity, no matter how large the networks are. Luckily, the probability grows exponentially with the network size: having 15 hidden units is already sufficient for randomly configured Frozen-Net to learn XOR with > 99% probability.
sufficient-Polariyy
This is exactly what we see in our simulation results (Fig 4 panel A bottom). For a generous amount of data (500 training samples) and learning time (100 epochs), the Frozen-Net RAND-Polarity curve nicely matches the theoretical curve- networks with more than 15 hidden units learn XOR-SD with very high probability, even though their polarities are fixed randomly.
7

---

Under review as a conference paper at ICLR 2023
A)
Space Efficiencyy Thoretical (unlimited and resourca)
B) Hidden units
triats reached valication accurarc
Pllsarning XOR)
Computational Efficiency
Statistical Efficiency
PXXOitcapable)
reach accuracy
80
Validat error %0
#epochs
25
30 Network ta)
100
500
100
500
100 samples
500
# samples
samples
*
Efficiency Simulation Result (500 training samples, 100 opochs)
C) Hidden units = 20
90% vollccttion eccuracy
Pllearning XOR)
Computational Efficiency
Statistical Efficiency
trais readhed
200
reached
100
reach sccaracy
8
Vallddation enror %9
50
on
#epochs
sufficiet//ella fby
40
25
Fluid
Free IAND-Polenity
100
500
100
500
100
Network width hidden
# samples
# samples
D) All
P(learning XOR)
Computational Efficiency
Statistical Efficiency
30
Freete-unintormed
neece-informed necce-unifformeed
Space
Space
Space
Preere-uninformer Fluid pecte-informed
Plad mnwoe-informed, Preeoearrinfoomee
2
Data 100
100
60
Data 100
Figure 4: network has to be sufficiently large for it to learn more quickly, with less data, its weight polarity is configured randomly. A) For randomly configured Frozen-Net learn XOR-5D, it has to be sufficiently large. In both the theoretical limit and simulation results, it takes at least 15 hidden units for a Frozen-Net RAND-Polarity (green) have > 99% chance learning XOR-5D; it takes three hidden units, in theory, for adequately configured Frozen-Ne (red) and Fluid-Net (blue) to learn XOR-SD. B) When Frozen-Net RAND-Polarity not large enough, has much lower chance of learning XOR-SD, and when it does learn, it uses more time and data. C) When Frozen-Net RAND-Polarity sufficiently large, it at least shows the same level performance standard Fluid-Nets. D) Frozen-Net nufficient-Polriii shows advantage over Fluid-Net (red + yellow) in almost training sample sizes (especially when data is limited) and across all network sizes. Randomly configured Frozen-Nets have an advantage when the network is sufficiently sized (> 15 hidden units). We ran 50 trials for all experiments. All curves represent medians with shaded regions denoting t5th percentiles.
Fluid-Nets (green + yellow), mostly
Both theory and simulation results show that we will lose advantages if weight polarities are fixed but not configured adequately; example of such is small, randomly configured Frozen-Net (e.g.. 10 hidden units, Fig 4 panel B). Notice that for the same network size, we ensure the configuration adequately set (red), then the network learns quickly and is data-efficient By allowing more space (network size), Frozen-Net
BAANDPPolarity starts to pick up the advantages in time and data efficiency (Fig 4 panel C). In summary, gain from setting weight polarity a priori only if the polarity is configured adequately (Fig 4 D); the adequacy can either be made more probable by having a larger network or can be guaranteed by an explicit configuration rule (c.g.. through development for NIs, or explicit rules in our simulations, or transferred polarities from previously trained networks).
6 DISCUSSION
We showed in this paper that 1) if the weight polarities adequately set a priori, then networks can learn with less data, time, space, and power; 2) polarity may be more effective and compressed medium of network knowledge transfer; and 3) while lose all advantages when the polarities
8

---

Under review as a conference paper at ICLR 2023
fixed but not set adequately, Below we discuss the novelty of our work and some future directions.
regain these advantages by increasing the size of the networks.
In the literature of bio plouusauebble artificial neural networks (ANNs), the most related work is on Dale's principle: a single unit's output weights are exclusively excitatory or inhibitory. An exciting attempt of applying such principle ANNs achieved performance comparable to multi- llati--lleerrrr perceptrons (gray-scale image classification tasks) and VGG16 (CIFAR-10) (Cornford et al.) (021). Our approach differs in several ways; the most fundamental one is ours does not require exclusivity of unit's weight polarity, we only ask the polarity configuration to stay fixed throughout training. Because we made fewer assumptions on the architecture and network properties. we were able to reveal the true power of weight polarities polarity- fixed networks can not only perform as well as the traditional approaches when the polarities are set adequately, but they can also learn more quickly with less samples. Additionally, we revealed that polarities, not weights, may be more effective and compressed form of knowledge transfer medium. Furthermore, our Freeze-SGD algorithm | is easily applicable to any existing network architecture and any learning mode, be it transfer learning or de novo learning, thus enjoy a wider application range.
Our hypothesis is also novel in the machine learning literature. In statistical learning framework, it is counter-intuitive to disassemble a single parameter into magnitude and sign. To the best of our knowledge, we are the first think in this way and prove that connection polarities themselves serve as an effective form of inductive bias. Previous work that vaguely took on a connectionist's view mostly focused on the existence range of connections (e.g., adding skip connections (He) iet al. 2016)), but the polarity of such connections were essentially left out of the discussion. Our work broke the stereotypical view and brought weight polarity into the playing field. lesson we learned from this research is that when designing network architectures, we should not only focus on the existence of connections, but also pay attention to the polarities of connections. In the transfer learning literature, to the best of our knowledge, there has been no indication that transferring polarity alone is sufficient for knowledge transfer ours is the first demonstration on the importance of polarity.
Our work is in syne with the framework of learning theory. Our way of explicitly setting the weight polarities provides strong prior for the networks. This is in line with the bias-variance trade-off theorem (Geman et al., 1992, Tibshirani & Friedman, 200T). Frozen-Net sufficient- PPolarity has a high inductive bias by trading representation capacity, they gain in time and data efficiency through discounting parameter variances. Indeed, the performance gain we observe from fixing polarities might be explained by preventing over-fitting thus better generalization to the validation data.
Our work is also in sync with the lottery ticket hypothesis (Frankle & Carbin 2019). Our observations on larger networks enjoying high probability of learning XOR-5D even when their polarities are randomly configured is a realization of the lottery ticket hypothesis.
We engineered our Freeze-SGD algorithm entirely based on SGD because we are interested in the pure effect of fixing polarity during learning. As discussed in See such an approach intrinsically put Frozen-Net in a disadvantageous position as while resetting weights to the correct polarity after each learning iteration, this procedure effectively fights against the gradient update direction. It is an interesting next step to adopt more polarity-freeze compatible learning algorithm, possibly allowing us to further improve learning performance. One possibility is adapting the primal-dual interior- method (Vogelstein et al.) 2010) or Hebbian learning (Amit) 2019), as well as a host of bio- bla-ssluee learning algorithms (Miconi 2017 Booppthy & Fiete 2022 Dellaferrera & Kreiman 2022).
From this work, we have strong experimental evidence (simulation and image classification) and some theoretical discussion suggesting that if the weight polarities are adequately set a priori, then networks can learn with less data, time, space, and power. We look forward to applying our Frozen- Net approach to more tasks to see if we can empirically extend our results to more diverse scenarios. We also look forward to extending our results theoretically (c.g., including SGD assumption in our Lemma 2.1
REFERENCES
Yali Amit. Deep learning with asymmetric connections and hebbian updates. Frontiers in Computa- tional Newroscience, 13:18, 2 2019. ISSN 16625188. doi: 10.3389/FNCOM.2011 9,00 8/BIBTEX.

---

URL Under review as a conference paper at ICLR 2023
Yassir Bendou, Yuqing Hu, Raphael Lafargue, Giulia Lioi, Bastien Pasdeloup, Stéphane Pateux, and Vincent Gripon. Easy: Ensemble augmented-shee y-shaped learning: State-of-the-art few-shot classification with simple ingredients. arXiv, 2022. doi: 1.488500amxiv.2221109999 URL
httt:://rxxv..or org/ abs 2201 0122200.6990 .09699
Akhilan Boopathy and Ila Fiete. How to train your wide neural network without backprop: An input-weight alignment perspective. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th International Confer- Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 2178-2205. PMLR, 17-23 Jul 2022. URL htt.hhttts:/ppp.. boopathy22a. html
ngs.mlg.m..........m............ wtt.s..mr..r.rs///.m www.srrsssvvv... 62.
Jonathan Comford, Damjan Kalajdzievski, Marco Leite, Amélie Lamarquette, Dimitri Michael Kullmann, and Blake Aaron Richards. Learning to live with dale's 's principle: [ANN|s with separate excitatory and inhibitory units. In International Conference on Learning Representations, 2021. http:tttt://opeeeeeeee
ew. net/ forumorrrm fasrummid=e??? 627xxppz)
Giorgia Dellaferrera and Gabriel Kreiman. Error-driven input modulation:
Solving the credit assignment problem without backward pass. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 4937-4955. PMIR, 17-23 Jul 2022. URL httpt::t///ppt //tpssee/ wgw.gm.......... m press/vsvvvv 62/
de 1aferrera22a. html
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126 1135. PMLR, 06-11 Aug 2017. URL
finn 1 7a htmll
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 7th International Conference on Learning Representations, ICLR 2019, 2019. URL
https: SttEp:://cperreviee net/ forummidd=2J1-33ea
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural Computation, 4:1-58, 1992. ISSN 0899-7667, doi: (tl62//eco..199.4..... URL
pettps://doi.orge 1tw.0...16/mmee 199.........9.2....
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016. doi: 10. N.1109/CVVR..006..99
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Aasa Feragen, Marcello Pelillo, and Mareo Loog (eds.), Similarity-Bon Based Pattern Recognition, pp. 84-92, Cham, 2015. Springer International Publishing. ISBN 777-3-311-24261-
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvumakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A.A. Kohl, Andrew J. Ballard, Andrew Cowie. Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Pushmeet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with al- phafold. Nature, 596:583-589, 2021. ISSN 14764687,
doi: 10.
http:://as 6oi org. 10. 1.....0335 1038/ 00/38//441500 /4155441866661 021 -
Alex Krizhevsky, Ilya Sutskever, and Geoffrey F. Hinton. Imagenet classification with deep con- volutional neural networks. Communications of the ACM, 60, 2017. ISSN 15577317. doi:
10.1145/3065386.
Michalis Lazarou, Tania Stathaki, and Yannis Avrithis. Iterative label cleaning for transductive and semi-supervised few-shot learning. ICCV, pp. 8731-8740, 2022. ISSN 15505499, doi: 10.1 109/cccv4992..2021.oom .00663............
10

---

Under review as a conference paper at ICLR 2023
Thomas Miconi. Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks. eLife, 6, 2017. ISSN 2050084X. doi: 10.7554/eLife. 20899.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-255 2015. ISSN 15731405. doi: 10. 10.10077/1126--05--8816
David Silver, Julian Schrittiieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George Van Den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nature, 550:354 359, 2017. ISSN 14764687. doi: no.1038/nature22270. URL http: doi org 038 na8/naauee222770
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL. / ngs neurips. cc/ wwt.riip..ccpppppeeeeeee httppcccpppppp////1/./// 7000017/fle.. 221
Nicholas C. Spitzer. Neurotransmitter switching in the developing and adult brain. Annual Review of Neuroscience, 40, 2017. ISSN 15454126. doi: 10.1 166 neuro- neuro-0721 072116-031204,
Robert Tibshirani and Jerome H Friedman. The elements of statistical learning (electronic resource): data mining, inference, and prediction: with 200 full-color illustrations. Springer, 2001.
Evie van der Spoel, Maarten P. Rozing, Jeanine J. Douwing-Duistermaal P. Eline Slagboom, Marian Beekman, Anton de Craen, Rudi G Westendorp. and Diana van Heemst. Siamese neural networks for one-shot image recognition. ICML Deep Learning Workshop, 7, 2015. ISSN 19454589.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Curran As- sociates, Inc., 2016. URL https: http:::s/pppoceeddingg nwwrips....... wwtpcccpppppeeeeee 6/file/ 90e
98361 2fw05e3ecc9e 4ww.e-19888c111 4 Paper p
Joshua T. Vogelstein, Adam M. Packer, Timothy A. Machado, Tanya Sippy. Baktash Babadi, Rafael Yuste, and Liam Paninski. Fast nonnegative deconvolution for spike train inference from population
calcium imaging. Journal of Neurophysiology, 104:3691-3704 12 2010. ISSN 00223077. doi:
URL https
7 //journals [/foaraaaalaaaaaaaaaooooroooooor
010.3.0000... 01073.2009
Qingyang Wang. Michael A. Powell, Ali Geisa, Eric Bridgeford, and Joshua T. Vogelstein. Why do networks have inhibitory/negative connections?, 2022.

---

Under review as a conference paper at ICLR 2023
A FREEZE-SGD ALGORITHM
Algorithm Freeze-SGD
for = 1.2..... do
Get weight polarity template (()) based on the configuration rules. Match W to 7
end for
for epoch 1,2, ........ do
for batch = 1, 1...... do
SGD updates all weights. for 1,2....... L do
Compare signs of the weights W to the template T())
get checker = 7 (0). do
for (i,j) where (i. <
Make (i in compliance with (i, by one of the following four ways:
Case posCon (i,)i 7 (i, j) where > 0
Case posRand (i.j) T (i,5) rand(10, e]) where >> 0
Case zero (i,j) =0
Case flip (i, -i,
end for
end for
end for
end for
12

---

Under review as a conference paper at ICLR 2023
B SUPPLEMENTARY FIGURES
A) Learning curve
VMl CaCLA AccunDc
102
180
592 580
570
222
520
88
posCon
posRand
180
472
350 472
58
esech
B) Probability learning XOR
% traals reached 90% validaaion accurryy
posRand
eccuracy
posCon
90% volidian Hiniitioon sccuumes
flip
90% valiciation ac accuracy
zero
% trials reae
% triats reached
90%
200
500
500
200
500
samples
samples
C) Computational Efficiency
posRand
posCon
tlip
zero
#apachs to reach 90% accuracy
reach accuracy
50
reach 90% accuracy
50
#epochs to reach 90% accurecy
S0
#epochs to
25
25
25
100
500
100
500
100
500
100
500
semples
samples
samples
# samples
D) Statistical Efficiency
posRand
posCon
flip
Volidation emor %
50
50
Validaatio eimrr %6
Volidation emror %
50
Volidtatio error
Hal
25
25
25
samples
samples
somples
samples
Figure B.I: Different reset methods in the Freeze-SGD algorithm yields similar results. The meaning of different reset methods is described algorithm [ A) Freeze-iffrrmed consistently reaches higher validation accuracy after the same amount of training time. This is especially evident when training sample scarce. B) Same curves as Fig 4 panels
first column, plotted for different reset methods. C) & D) Same curves as in Fig I panels B & C, plotted for different reset methods. run 20 trials for all experiments in this figure. All curves correspond to medians with shaded regions representing the 25th and 75th percentiles.
13

---

Under review as a conference paper at ICLR 2023
100
55% validation accuracy
100
85% validation accuracy
100
% trials reached
96 reached
reached
reach accuracy 60% validation
Freeze IN- Polollrity RAND-Polarity
Freeze RAND-Polarity.
IN- P-Polarity
Fashion-MNIST
9o0
1000 samples
6000
100
1000 # samples
6000
0
100
1000
6000
samples
#epachs to accurary
#epochs to accuracy
70
#epochs to
35
900
1000
6000
100
1000 # samples
6000
1oo
1000
6000
# samples
samples
trials reached 45% validation accuracy
100
55% daliinaiion accuracy
100
validation accuracy racy
100
trials rea ched
reached
60%
CIFAR-10
100
5000
50000
100
5000
50000
100
5000
50000
# samples
samples
samples
reach 45% accuracy
reach accuracy
reach 60% accuracy
#epochs to
#epochs to
#epochs to
35
20
100
5000
50000
900
50000
50000
# samples
samples
samples
Figure B.2: Related to Fig 2 Regardless of validation accuracy threshold, Frozen-Nef IN- Polarity always learn more quickly. Same curves in Fig Fightt two columns, plotted for different validation accuracy thresholds.
14

---

Under review as a conference paper at ICLR 2023
50% validation accuracy 60% validation accuraty
100
#epochs to reach 60% accuracy
% reached
20
Freeze IN-Polarity
200
1000
6000
900
Fluid IN-Polarity
samples
# samples
reach 80% accuracy
Freeze IN-Weight
% trials reached
50
Fluid IN-Weight
#epacits la
Fashion-MNIST
Validation error %
50
25
100
1000
6000
100
1000
6000
85% va ialiddaioon accuracy
samples
samples
900
1000
6000
trials reached
100
reach 85% accuracy
90
# samples
esochh to
45
100
1000
6000
100
1000
6000
samples
samples
50% validation accuracy
100
80
% trials reached
#epochs to
reach 50%
40
100
5000
50000
100
50000
100
55% validation accuracy
samples
samples
100
90
CIFAR-10 10 Validation error
% trials reached
#epochs to 55%
50
45
900
5000
50000
100
5000
50000
100
5000
50000
# samples
60% validation accuracy
samples
samples
100
60% accuracy
% trials reached
# #epochs to
45
100
5000
50000
100
5000
50000
samples
samples
Figure B.3: Related Fig All scenarios plotted separately instead of plotting only for the differences.
15

---

Under review as a conference paper at ICLR 2023
60% validation accuracy
100
15% 15% validation accuracy
85% validation accuracy
100
% trials reached
% triels reached
reached
Freeze IN- Polarity
Fluid RAND-Polantyy
Fluid IN-Polarity
Freeze IN Weight
Fluid IN-Weight
%
Fashion-MNNI5
9oo
0
1000 samples
6000
100
1000
6000
1000
6000
samples
samples
reach 60% accuracy
75% accuracy
reach 85% accuracy
20
10
A Wepochs to
#epochs
A #epochs
10
100
1000
6000
100
1000
6000
100
1000
6000
# samples
samples
samples
45% validation accuracy
100
validation accuracy
100
60% validation accuracy
100
% trials reached
% trials reached
% trials reached
CIFAR-10
100
5000
50000
100
5000
50000
100
5000
50000
samples
samples
# samples
reach 45% accuracy
20
reach 55% accuracy
reach 60% accuracy
70
à #epochs to
#epochs
Wepochs
10
35
100
5000
50000
100
5000
50000
100
5000
50000
samples
samples
# samples
Figure B.4: Related to Fig 3 Regardless of validation accuracy threshold, transferring and fixing polarities help networks learn faster than traditional weight transfer strategy. Same curves as in Fig right two columns, plotted for different validation accuracy thresholds.
16

---

Under review as a conference paper at ICLR 2023
Fashion-MNIST
CIFAR-10
Weight polarity flipped fraction Weight polarity flippdd fraction Weight polarity fllpped fraction Weight polarity flipped fraction
100 training samples
Weight polarity fllpped fraction Weight polarity flipped fraction Weight polarity Nlpped fraction Weight polarity flipped fraction
100 training samples
0.5
0.5
10-3
10-3
10->1
10-6
50
50
epochs
epochs
500 training samples
500 training samples
0.5
10-3
10-3
conv1
conv2
conv3
10
50
50
conv4
epochs
epochs
conv5
1000 training samples
1000 training samples
fc6
0.5
fc7
fc8
10-3
10-3
fc9
10-b1
10:
50
50
epochs
epochs
6000 training samples
10000 training samples
0.5
10
10-3
10
10-6
50
50
epochs
epochs
Figure B.5: Tracking number of polarity flips. Weight polarity flips were analyzed for Fluid RAND-Polarity (SGD with random initialization) and measured by the ratio of weight parameters (excluding bias terms) flipped sign between two consecutive epochs. The first 50 epochs were analyzed and plotted, separately across layers and training data size. Curves are median with shaded area representating the 25th and 75th percentiles out of the 20 trials.
17

---

Under review as a conference paper at ICLR 2023
0.8
Proportion
0.4
Freeze Polarity
Fluid RAND Polaritt
RAND MMDD-Pt
0.2
Freeze IN Weight
n-Weight
Fluid IN-Polartty
0.00
0.01
0.02
0.03
0.04
0.05
magnitude
Figure B.6: Weight magnitude distribution at initialization across all experimental conditions for image classification tasks. All scenarios follow the exact same magnitude initialization magnitude except WN-wt transfer.
C METHODS
XOR-5D Data were prepared by sampling from the XOR-5D distribution as described in the main text, the training data varied, validation is always 1000 samples across all scenarios. A single hidden layer network (64 hidden units for Fig [ various for Fig was trained for 100 epochs; in total 50 randomly seeded initializations (trials) were ran. To have the best controlled experiments, we fixed the magnitude distribution, architecture, training samples (n varies), learning rate, batch sequence of the training data, and the validation samples across all tested scenarios. Statistical efficiency was quantified by plotting validation error rate convergence across different training data scarcity levels. Computational efficiency was quantified by plotting the number of epochs to reach certain level of validation accuracy. We note not all trails could reach the same level of validation accuracy cut-off, to present the whole picture, we 1) plotted the percentage of trials that reached the validation accuracy cut-off (success rate); 2) showed results across a range of cut-off selections (supp Fig B.2 B.3 B.4
Image classification Experiments were done essentially the same as XOR-5D in tightly controlled fashion, same magnitude distribution, batch sequence (batch size = 1000), learning rate, and training data across all scenarios. 1r=0,001 was chosen from [0.03, 0.01, 0.005] based on the most accuracy gain after 50 epochs across all scenarios. Networks were trained for 100 epochs, 20 trials in total. We did not do any image augmentation. All performance quantification follow the above description.
All experiments were run on RTX-8000 GPUs.
18

---

Under review as a conference paper at ICLR 2023
D XOR THEOREMS RELATED TO SEC
Lemma D.1 (minimum XOR solution, 2D), For any single-hidden-layer network to be able to solve XOR, it is sufficient to have 3 different XOR-compatible hidden units. The weight pattern for each ())),, where j is the unit index. j E {l..... n.,, n2 the number of hidden units. An XOR compatible hidden unit is where the weight pattern triples have (1)
unit can given by a triplet (wl.) 1.3. 22,3 2,11),
their polarities satisfy sign(w), (2)
j.1) sign(w, (w] x sign(w2)).
See proof on page 20
Lemma D.2 (can learn XOR probability, 2D). For single hidden layer network with n. hidden units (n E 2 J. randomly initialize each weight with its polarity following P(polarity) ~ Bernoulli(0.5) then the probability that this randomly initialized network can learn XOR without changing any of the weight polarity is lower bounded by
S(P(n)) =1 Ex=o Q(k, 4, n)
Zt
8
k-1
m,,, M, = I(M + +k)
Σ
Q(p, M, n)]
k
p=
Q here is a helper function for counting, it is defined in more detail in the proof.
See proof on page 21
Theorem D.1 (can learn XOR probability, high dimensional). For a single hidden layer network with n hidden units, randomly initialize each weight with its polarity following P(polarity) ~ (0.5 then the probability that this randomly initialized network can learn high- dimensional XOR (first two dimensions are relevant, the rest (d 2) dimensions are irrelevant) without changing any of the weight polarities lower bounded by
S(P(n, d)) = 1 E2-0 Q[k, 2d, 22222) (2d+1)²
n., 222
Okkm,,M.n I(M + k3) EQle.kMM,n)] m
Q here is helper function for counting, it is defined in detail in the proof.
See proof on page 22
19

---

Under review as a conference paper at ICLR 2023
E PROOFS
Lemma 2.1 (capacity-specs speed trade-off). If the weight polarities are set a priori, such that the function is still representable, then the network can learn faster.
Proof of Lemma 2.7 A feedforward DNN can be described by as a graph
G = EV, w E - D..
Nodes of the graph correspond to neurons (units), where each neuron is function (x)( aW x + b( E [mr]. All the weights for the network take values from the set II f-d...., ............... O.....,, d} for some d N. The network is organized in layers. That is, the set of nodes can be decomposed into a union of (nonempty) disjoint subsets, = -f Vi. such that every edge in E connects some node in Vi-1 some node in Vi, for some 1 E [Z]. Assume we have fully connected layers. Then the number of incoming edges per node is V Let |Vol be the input space dimensionality. All nodes in Vo (input layer) and Vz. (output layer) are distinct.
Let G denote the total number of distinct weight patterns of the graph G. Then for a single hidden layer network where L = 2, have:
GG|
[TD |B| = |TD (16N1/1))))))))))))))))
(2)
Then
G|
2d +1 (1a/+ [1|+/11/1111
(3)
GG GooterityFrooe Frozen
d+1
Assume a different weight pattern represents a different function (trivially holds in linear + full rank weight case), then for every single representable function, there always exists set of weight polarity configurations Georrect such that Goorrect C polarity Frozens therefore
Georreot
Georreet
(4)
G FolarityFrozan
G|
This means setting weight polarity priori in an adequate way constraints the combinatorial search space to have much higher proportion of correct solutions, hence easier to learn under exhaustive search algorithm (Lemma 2.1.
O
Lemma D.1 (minimum XOR solution, 2D). For any single-hidden-layer network to be able to solve XOR, it is sufficient to have 3 different XOR-compatible hidden units. The weight pattern for each
unit can be given by a triplet (w(") 1. W2.17 wx where j is the unit index, j E {1..... n), n is the number of hidden units. An XOR compatible hidden unit is where the weight pattern triplet have (1))
(1)
(2) their polarities satisfy sign(u j,1 = sign (1) x sign(w?)))
Proof Lemma D.I] For each hidden unit, we can enumerate all of its 8 possible weight polarity configurations, with the index set A Polarities = {1, 2,, 4,5, 6, 7,8):
#
W 1
2
+
+
3
+
4
+
5
6
7
8
The set EXOR-Polariie follow XOR. To prove Lemma D.I we first consider the case of 3 hidden unit single layer network. To prove a network of certain polarity pattern is capable of solving XOR, it suffices to give working solution. Below, we exhaust all possible 3-umit network that satisfy the statement of Lemma D.I i.e. having different XOR-compatible units; and list out their corresponding working network solutions.
3,, 5, C Apotorities contains indexes of polarity patterns that
20

---

Under review as a conference paper at ICLR 2023
(
#
wi
W2
2.3
(T)
w
working network F(x,1) y)=
2
+1
+1
[
3
0
0
sigmoid(-o(x+ +
0
2
+1
0
0
11
(
+1
+1
sigmoid(-o + y) - o(-1))
8 0
-1
0
2 0
+1
-1
(2,5,8)
+1
+1
+ a) - o(-z))
8
-1
0
0
-1
0
-1
+1
(3,5,8)
5
-1
0
0
+1
sigmoid(o(-9) + + o(-z y))
8
For the case that single-hidden-layer network has more than 3 hidden units, if satisfies the rule in Lemma D.I we can always construct XoR-solvable network solution by setting all other weights to zero except for 3 different XOR-compatibl units, then we arrive at one of the four situations in the above table and we just proved they are XOR solutions.
Therefore, for any single hidden layer network to solve XOR, it sufficient to have at least 3 of the XOR polarity patterned units.
Lemma D.2 (can learn XOR probability, 2D). For single hidden layer network with n hidden units (n E Z+), ), randomly initialize each weight with its polarity following P(polarity) ~ Bernoulli(O... then the probability that this randomly initialized network can learn XOR without changing any of the weight polarity is lower bounded by
((P(n)) = Ei=o Qkk, 4,,,
nE Z+
8
m
Ok,m,,M,r) = 1(M k)
Σ
Q(p k, M, M,
Q here is a helper function for counting, it is defined in more detail in the proof.
Proof of Lemma D.2 For any network randomly initialized, its weight polarity pattern a set of draws with replacement from the polarities indexed by the set Aooaritiess | Define H = ........--......
8. h,.., Am. E Aoloritica H n. to be the tuple of the indices of the observed weight patterns for the hidden layer. hm is the index of the weight polarity pattern for unit m. For any network to be able to solve XOR, it needs to have at least 3 units whose weight patterns are distinct members of
Polarttees (Lemma D.I). That is, let J {hm hm E H, hm SxonnPalarittie J. Then we need that [J] 2 3. We can define the probability of having exactly k of the XOR compatible weight patterns present within the hidden layer as following (for brevity,
A = Apolarities: B = B BXoR- Polorities):
None the members in set B appears is given by:
P(J 0)
=
A
Only one of the member in set B appeared in H, and that member can appear more than once in H is given by:
P(J) =) (1) ([BB + -B
(5 - 4n)
1
8
This is explained by choosing one of members of B (L)), then multiply by the chosen member appears at least once (( + 1)" |B(")
21

---

Under review as a conference paper at ICLR 2023
Only two of the members in set B appeared in H. and both appear more than once in H:
P(J] = 2) = (2) 2) (() + + 1B [
A
( (6" ( (5 4n)
4T)
8
To count two members appear at least once in H. we have (122) ways of choosing the members from B, and there (IB) + 2)" ways of choosing with replacement populate the tuple H, where each hidden unit can choose from in total (IB) + 2) possible patterns, and subtract the situation
where only one of the two members appeared ( (1 + 1)" [B and the situation where neither appeared | The above equations can be put into compact form
Q(k: |1. |A| - [B),
P([JI k-)
|A|"
where Q(ki m. M,
[(M k)* Σ Q(v, k M, n)] (2), E {O....,m).I
1.... B). M I...., ......|A|- |A|- |B) e zt is helper function that gives exactly of the different XOR- com-compatible polarity patterns appeared in units, and in total m - M options are considered for each unit.
Then, P(n) is counting at least 3 of the 4 set B patterns appear:
((((m)) = P([] 2 3) = - P([J = - P(J]]=1) - P([J] = 2)
1 4kk,4,4,1)
8"
(2)m (1) - 47)
(6) (2) (5" - 4T) 4")
8
Theorem D.1 (can learn XOR probability, high dimensional). For a single hidden layer network with n hidden wnits, randomly initialize each weight with its polarity following P(polarity) ~ (er5oulli(0.5), then the probability that this randomly initialized network can learn high- dimensional XOR (first two dimensions are relevant, the rest (d 2) dimensions are irrelevant) without changing any of the weight polarities is lower bounded by
S(P(n, d)) = =1 -2 Q(k, 2d, 2222 (2d+1)²
n, E ,
Q(k, M, IIM k)) Qle, M, m
Q here a helper function for counting, is defined in more detail in the proof.
Proof of Theorem D.7 We solve Theorem D.I with the exact same counting algorithm as in Lemma D.2 the only difference is now |A = =2+11 and ExxoR-Polarinn = 2d We prove these two equalities below.
[A = 2d+1 because we have input weights and output weight for each unit.
The weight pattern of single hidden unit in this case is given by a tuple (w (w j........ w (1)
d.j. w j.1 ).j
(2)
{1.....72 n and n the number of hidden units. Our conclusion Lemma D.I trivially extends to the high dimensional case. This is because nigh-dimensional-Xe XOR-solvabll network solutions can be
trivially constructed from Lemma by setting the irrelevant imput dimension weights to 0. We can
restate Lemma D.I for high-dimensional XOR following:
22

---

Under review as a conference paper at ICLR 2023
For any Lingl-hidden-iee network to be able to solve high dimensional XOR (only first two dimensions are relevant), it is sufficient to have different coRmcaiaaa hidden units. A
high dimensional XOR compatible unit polarity pattern is ruled by sign(w}.1 (2) sign(wi.)) (1) X
sign(w2.))
Therefore the irrelevant dimension input weights can be of either polarities and there are 2d-2 different combinations of them. Therefore [Bxon-Polerities] = 4 x 2d = 2d.
Follow the steps in Lemma
D.2 have (for brevity, A Apolarities: B BxoR-Polaritae
P(J] = 0) || td
2d
2d+1 )
|
P(J - 1) (11) ((I
+ 1 ) (2)) + +1)a - (24)*)
|||"
(2d+1)n
P(J] = 2) (12) ((IB) +2) (2) (([ B + 1)ah - [B (T)
|A|"
(2) + 2) - ((24 + 1)" (24)m) (24)m)
(20+1)n
Therefore, we have
((P(n, =1 Ex=0 Q 24, 2k,,2,,2',n) 2
(2
=1 =1- 12)00 (12 + 13"h - (24)m)
(22 +2) - (22 + 1)
(24)a) - (2d)²)
(2d+1)"
(20+1)n
n. fodezt,cm 222
23


Reviews:
[{"review_id": "kScxilFLoRb", "reviewer": "Reviewer_R5EG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Connections between neurons in the brain are almost universally exclusively excitatory or inhibitory, which seemingly reduces the representational capacity of biological networks. Inspired by this observation, and the assumption that the brain has been optimized throughout evolution, the authors propose that by freezing the sign of connections between neurons may facilitate faster and more data-efficient learning. ", "summary_of_the_review": "While this is interesting and important work the main contribution is unclear to me - is freezing important or just choosing signs apriori. The connection to the biological networks seems lacking and should be developed in order to make the claim that this is the strategy the brain is employing. \n", "strength_and_weaknesses": "Strengths\n- Interesting and important direction of research with potential improvements for AI and insights for both AI and neuroscience.\n- If the brain is following this strategy it is a very interesting theory for why the brain has the restrictions it does. \n\nWeaknesses\n- The main contribution of this work is unclear to me. There seems to be two separate issues that are not adequately separated in the text and experiments: (1) the initialisation of a network with a task-specific pattern of signs that results in better learning (2) the freezing of signs during learning. The experiments lack the controls of “Fluid sufficient-Polarity”, and “Fluid IN-polarirty” which would be critical to understanding this. My suspicion is that (1) mainly underlies the results. \n- The networks do not reflect the type of polarity restrictions as found in the brain (all of the outgoing connections from a neuron are of one polarity). Though the author’s acknowledge this in the discussion, it is not clear to me how this is work is a bridge. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X3Bx3AbZwy", "reviewer": "Reviewer_JBKK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores the computational features that result from a neural network with weights of fixed polarity. The authors demonstrate through proofs in constrained settings, and experiments in both constrained and more general settings, that networks with fixed and appropriately initialized weight polarity learn faster than randomly initialized networks. They also demonstrate that for randomly initialized networks of sufficient size, constraining the polarity of the weights in the update function does not compromise learning.\n\nThe paper proposes  a learning algorithm which is a variant on stochastic gradient descent but which does not permit polarity changes in the weights (Freeze-SGD). The paper compares networks trained under this algorithm (frozen nets) with networks trained under regular SGD, including networks that have previously had some pretraining. They present a theoretical trade-off between network representational capacity and learning speed, as a function of the network’s polarity being fixed.\n\nIn experiment 1 they consider single-layer networks on the 5D-XOR problem (the first 2 dims are classic XOR and the final 3 dims are noise). They compare learning rates of 3 networks to consider the influence of two factors: fixed weight polarities and well-chosen polarity patterns. The first net is initialised with the weights set to have the correct polarity combination for the task (a priori knowledge) and which uses a polarity-frozen update rule the authors propose. The second network uses this same update rule but is initialised with random small weights (presumably of the same scale), and the final network uses random weights and vanilla SGD. The network which had both the correct % polarity and used the polarity-preserving update rule learned the fastest.\n\nIn experiment 2 they consider image classification with AlexNet on the datasets Fashion-MNIST and CIFAR-10, and consider the same three initialisation and update conditions.  \n\nIn experiment 3 they compare initialising the network with ImageNet-trained weights vs Imagenet-trained polarity %s, to assess whether transfer between tasks is more effective when transferring the weights magnitudes and polarities or simply the  polarity alone (both under the polarity-preserving update rule). This yielded a very surprising result that polarity alone resulted in faster learning on the new tasks. \n\nThe paper then calculates the probability that a randomly initialised network with a fixed polarity update rule will never be able to learn the XOR task. The experimental results closely match the theoretical results.\n", "summary_of_the_review": "This paper explores the computational features that result from a neural network with weights of fixed polarity. The authors demonstrate through proofs in constrained settings, and experiments in both constrained and more general settings, that networks with fixed and appropriately initialized weight polarity learn faster than randomly initialized networks, and that for randomly initialized networks of sufficient size, constraining the polarity of the weights in the update function does not compromise learning. In my opinion the findings are novel and very interesting, but subject to a few caveats that need confirming.", "strength_and_weaknesses": "*Strengths:*\n- A very interesting idea of broad and general interest to both the computational neuroscience community and the ML community.\n- The authors explore the idea of fixed weight polarity is some depth, discussing the theoretical benefits and costs of fixed-polarity networks (costs: limits on representational capacity, pros: learning speed).\n- Nicely designed experiments in both toy and more realistic settings (however see comment below on an important way they can be improved).\n- Beautifully written.\n\n*Weaknesses:*\n- In all experiments, but most crucially in experiments 1 and 2: The authors should do the final control experiment in which they initialize the network with the same weights as Freeze-sufficient polarity (set to be the correct % polarity) but without the frozen learning algorithm. This will truly isolate the two factors of (1) initially setting the weights % with the correct polarity, and (2) keeping that weight polarity throughout training. This yields a fully factorized experimental design which is tidier for isolating the two contributing factors and their interactions. It may be that initializing the network with the correct polarities but without constraining these polarities via the update rule, is sufficient for the network to gain the same benefits. This experiment would tell us these sufficiency conditions.\n\n- I couldnt see whether in each experiment the initial weights were set with the same generative process for the different experimental conditions (and so have the same magnitude distribution). To ensure that different weight magnitude distributions (which are known to affect learning rates) are not playing a role in these results, the Freeze- in polarity vs Freeze Rand, vs Fluid weights should all start with the same magnitude distribution, despite the differences in polarity. For these results to be interpretable this must be true - and so I assume this is the case - but I could not see this mentioned in the text (perhaps I missed it) and it’s quite an important feature to highlight.\n\n- Could the authors comment on whether randomly initialised networks trained with SGD learn polarity early on in training? I am aware that weight magnitudes are typically learned very early on in training but am not sure whether polarity is also a commonly learned early feature of weights with SGD. If this is known, it would help to complete the picture and could be added to the discussion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LLP-2yGtGJm", "reviewer": "Reviewer_8Q8m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Update after rebuttal**\nThe authors added an important control experiment (fluid in-polarity) which shows that the main improvements result from transferring a favorable initial set of weights, whereas actually freezing the polarity (which is *the* main NI inspiration) has a a marginal effect (and it is conceivable that this effect mainly results from very early layers). This makes the empirical results in the paper stronger, but significantly weakens the link to the NI inspiration (after all, as the authors point out, polarity change in biological brains is rare). The authors chose to largely keep the emphasis on NI and reformulate one, two sentences (where explicit claims were made) rather than shortening the discussion to a single paragraph in the discussion - this is a rhetoric trick to save work with re-writing, but essentially leaves most of my criticism unaddressed. Overall, the authors rebuttal was quite deflective and many of the issues I raised were not addressed or only minimally addressed (see detailed comments to author feedback). I therefore remain with my final verdict, that at its core the paper explores an interesting phenomenon in transfer learning (a combination of good initialization and a regularizing effect of fixing polarity); the connections to NI are misleading and contradictory (if freezing polarity is so important, how can the fluid-in-polarity results be explained; in fact it is currently unclear what fraction of weights changes polarity in that setting). From a purely transfer learning perspective it is unclear whether the observed effects also hold in a regime where models are practically useful (currently results where the proposed scheme performs well are in the low accuracy regime); limiting the impact of the work to practitioners aiming to improve classifiers' performance beyond the very low data regime. Given the additional control experiments and some of the small fixes I would increase my score by one point (from 3 to 4), but the current scoring system only allows a 5 as the next score and I personally think that the paper in its current form is not borderline. I remain open to updating my final verdict based on the reviewer discussion of course.\n\n\nThe paper investigates the effect of fixing the signs of weights (polarity) in an artificial neural network during training by using an oracle (expert knowledge or pre-trained network). This idea is inspired by the fact that “neuronal connections in the brain rarely see polarity switch”. As the paper shows on some simple datasets, if signs of weights are set correctly (and not randomly) and held fixed during training, faster / more sample-efficient learning is possible compared to a network that does not use the oracle knowledge (i.e. a standard neural network). Some theoretical insight is provided via the argument that fixed polarity leads to a reduction over the functions representable by the network, and thus a reduced search space over functions via exhaustive search. The second set of experiments shows that transfer learning in the very low data regime with pre-trained ImageNet weights to FashionMNIST and CIFAR-10 works slightly (but statistically significantly) better when only transferring weight polarity but not magnitude. \n\nDisclaimer: I have reviewed a previous version of this manuscript, and am happy to see that the authors have taken into account some suggestions raised by the reviewers, but have chosen to ignore some other suggestions. My review might thus be somewhat repetitive in some places (I have not recycled anything from my previous review though).\n", "summary_of_the_review": "Overall I think the paper has an interesting technical core (investigating the effect of frozen weight polarity, provided by an oracle / pre-trained net). There are some alternative hypotheses that need to be ruled out via control experiments (see improvements) and the results are currently a bit limited in terms of practical relevance. Particularly, the transfer learning results would be very interesting if observed in more challenging settings and in a regime where the classifier actually performs reasonably well. The theory does not add much to the paper unless it is extended to include SGD, which currently does not seem trivial. In my opinion, the neuroscientific/neuroevolutionary speculation currently lowers the overall quality of the paper - I suggest removing it completely from the current work, since it is not necessary and does not add to the technical part on artificial neural networks trained via SGD. As the paper currently stands I recommend rejection - even if the neuroscientific motivation is removed, the results (though promising) are not quite above the threshold for an ICLR publication (but would make an interesting workshop contribution). Having said that, I think compared to the version I previously reviewed, the paper has made good progress in the right direction.", "strength_and_weaknesses": "**Main contributions, Impact**\n1) Empirical analysis of the gains of freezing weight polarity during training using oracle knowledge. To the best of my knowledge this is novel. Results confirm the expectation that additional knowledge helps reduce the required number of training examples / increases sample efficiency by providing additional biases. While the result is interesting, its practicality is very limited since this oracle knowledge is often unavailable.\n\n2) Empirical analysis of using pre-trained signs instead of pre-trained weights for adapting from ImageNet to FashionMNIST/CIFAR-10. In the very low data-regime, where the model still has very large validation errors, there is some small, but consistent improvement (Fig. A.3 gives a better sense of the scale). This could potentially be interesting, but is currently of very limited impact due to the following two issues: (i) effect currently only holds in a regime where the model performs very poorly and disappears for large enough sample sizes from the transfer dataset; (ii) current experiments cannot rule out that we simply observe a regularizer that prevents overfitting to the small dataset instead of transferring useful knowledge (the missing control experiment for this is Freeze RAND-polarity).\n\n3) Simple theoretical analysis that fixing the polarity of weights reduces the cardinality of the space of possible representable functions. This trivially leads to a reduced search time under exhaustive search. Impact: marginal - the result is very simple to construct, and its relevance for training a network via SGD is unclear (SGD is different from exhaustive search and a reduction in cardinality of the hypothesis space does not trivially translate into faster and more sample efficient SGD training).\n\n**Strengths**\n * Investigation of an effect not studied before (to the best of my knowledge)\n * Most of the paper is written clearly\n * Statistical significance analysis of some results\n\n**Weaknesses**\n * Practical relevance currently seems minor: in many cases oracle knowledge around the correct polarity is not available, and the current transfer learning results are not very convincing (only small improvements in a regime where the performance of the classifier is quite bad).\n * Theoretical analysis has no link to training neural networks via SGD.\n * Connections and claims w.r.t. natural intelligence, learning in biological brains, and neuroscience are overstated, vague, and of insufficient scholarly standard. What’s needed as a minimum is a technical discussion of learning in dynamic models of neural processing (spiking neurons or other models involving short- and long-term neuronal **dynamics**); talking about the role of excitatory and inhibitory synapses by referring to the sign of weights in an extremely simplified rate-based model only allows for extremely simple statements - the paper makes claims far beyond these. This would be tolerable if toned down and limited to a small paragraph in the discussion, but currently occupies roughly a third of the paper throughout all sections.\n\n**Improvements**\n\n1) Remove the claims and analogies w.r.t. biological brains and how this paper uncovers the main reason why evolutionary optimization has led to fixed polarity of “neuronal connections”. It is ok to use this fact about biological neural networks as an inspiration (one, two sentences in the intro) and then have one paragraph of discussion, but none of the current discussion is actually necessary for the technical part concerning artificial neural networks and SGD. If the amount of space devoted to this is kept, then the scholarly standard needs to be raised significantly. As a starting point I strongly suggest reading up on *spiking neuron models* and various types of *short-term plasticity*, as well as neuro-computational models using circuits of spiking neurons and population dynamics. Many basic textbooks in computational neuroscience cover the topic - I can recommend “Spiking Neuron Models: Single Neurons, Populations, Plasticity” by Gerstner and Kistler 2002. To defend against this criticism raised during the last round of reviews the paper says “As our goal is to see the pure effect of fixing weight polarity, we did not adopt any bio-plausible learning algorithms as they may introduce confounding factors.” - this is fine and reasonable, as long as **NO** claims are made w.r.t. biological learning or brains. If such claims are made they need to be discussed in the context of biologically plausible neuron models and learning algorithms.\n\n\n\n2) Important questions w.r.t. the neuroscientific connections currently not answered in the paper:\n  * Biological neural networks are much more dynamic than artificial neural networks (= simple rate based models). Neurotransmitters, short-term plasticity, electrochemical depletion, all the way to the dynamic recruitment of whole neuronal circuits and populations can play a huge role in the function of the brain for rapid adaptation (few show learning) and transfer learning - and different excitatory and inhibitory mechanisms play a significant role, and have been well studied, in each of these. The paper currently only says “neuronal connections in the brain” - what exactly does that refer to (synapses, neurons, populations?), and what exact excitatory and inhibitory mechanisms is the paper trying to explain?\n  * The experiments in the paper use artificial neural networks (= simple rate based models) trained via SGD. Most neuroscientists would claim that results obtained from these have very limited bearing on explaining the function of biological neural networks and biological learning. Why do the experiments in this paper have biological relevance, and what exactly is that relevance?\n  * The one neuroscientific reference provided in the paper (Spitzer 2017), except for the discussion, seems quite irrelevant - it is about the (rare?) phenomenon of ‘neurotransmitter switching’ where “Neurotransmitter receptors on postsynaptic cells change to match the identity of the newly expressed neurotransmitter.” [from the paper’s abstract]. While the phenomenon is potentially associated with certain neural diseases and addiction, it is not hypothesized to have a connection with few-shot learning or transfer learning. Why choose this as the one reference?\n  * “post-development, neuronal connections in the brain rarely see polarity switch”. What developmental stage is exactly meant here? What “connections” are meant? Be precise, and add citations.\n* “What then makes it so that our brains may have willingly chosen to give up on a vast portion of their representation capacity? Our answer is: to learn more quickly.” I disagree with this answer. The widely agreed upon reason (to the best of my knowledge) is that biological (spiking!) neurons communicate activations via increases in firing rate; without inhibitory connections stable dynamics in circuits of such neurons are impossible (activations, and thus firing rates would reinforce each other without bounds, which quickly drives neurons beyond the biologically feasible operating regime; there are hard physical limitations for maximum firing rates of neurons and neural populations - inhibitory mechanisms are necessary to allow any computation at all). Please clarify.\n\n3) Figure 2 and 3 also need to show Liquid (IN-)polarity, where the correct polarity (but not the weight magnitudes for Fig 3) are used to initialize the network, but then the polarity is free to change during training. This answers whether it is important to freeze the polarity (as proposed in the paper), or whether it’s sufficient to only initialize it correctly.\n\n4) Fig. 3 is missing the Freeze RAND-polarity experiment, where the correct proportion of polarities is frozen but not necessarily for the correct connections. This allows ruling out that  freezing polarity simply provides additional regularization but does not lead to any transfer of information from the pre-trained network. \n\n5) Perform the transfer learning analysis in Fig. 3 layer-wise (i.e. transfer polarity for the first layer only, then the first two layers, etc). One hypothesis to explain the results in the paper is that the transfer of Gabor-like filters in very early layers explains a large fraction of the observed performance gains. Sign-information might indeed be virtually all that’s needed for Gabor-like filters. Transferring whole weights for the final classification layer might cause most of the observed decrease in performance for Liquid IN-weight.\n\n6) Ideally repeat Fig. 3 with a validation set and early stopping instead of a fixed set of epochs, to rule out effects of overfitting on the very small datasets.\n\n\n**Minor comments**\n\nA) Change in notation from ‘Fluid’ (Fig 2) to ‘Liquid’ (Fig 3).\n\nB) “In the literature of bio-plausible artificial neural networks (ANNs), the most related work is on Dale’s principle: a single unit’s output weights are exclusively excitatory or inhibitory (Dale, 1935).” With all due respect, but the literature on biologically plausible neural networks was at best at its infancy in 1935 - if this is the closest reference to this work, I would see this as a strong sign of concern in terms of biological plausibility (not a sign of groundbreaking novelty of the current work). \n\nC) “Our angle is completely novel in the neuroscience literature.” This is too strong; there is a vast body of literature on the relationship between excitatory and exhibitory circuits/neurons/synapses/neurotransmitters and their role e.g. in decision-making and short-term adaptation and motor control. While the core idea of fixed polarity might be an interesting hypothesis to explore seriously from a modern neural dynamics perspective, the current work fails to bridge this gap.\n\nD) “As discussed in Sec 2, such an approach intrinsically put Frozen-Net in a disadvantageous position”. This is not entirely correct: the net is in a disadvantageous position in terms of representing any possible function - if the polarity is set correctly the net is in a highly advantageous position since the search-problem has been greatly reduced (under exhaustive search).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8tyrk9aGLV", "reviewer": "Reviewer_afAt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper argues that in biological systems the weights’ polarities (signs) remain fixed, and only magnitudes (synaptic plasticity) change over time, leading to faster learning. Inspired by this argument, the authors propose to fix the polarity of weights in the DNNs to learn and transfer faster. Towards this the authors propose a learning algorithm, called Freeze-SGD, whereby given a predefined polarity of all the weights in the network, the weights’ signs are kept fixed throughout training and only magnitude is updated. The authors argue that proper task-based configuration of the polarity before training is important for faster learning. The authors further argue that transferring polarity instead of the weight patterns (magnitude + sign) is a more effective and compressed form of knowledge transfer. The experiments are conducted on classification tasks using a synthetic 5-XOR and image CIFAR-10 dataset.", "summary_of_the_review": "Please refer to the strengths and weaknesses section above for details.", "strength_and_weaknesses": "**Strengths**\n\n1. While I am not an expert in biological neural networks literature, it seems the fact that the weight signs don’t change throughout learning is an important observation and should be exploited more in the artificial neural networks.  \n\n\n**Weaknesses** \n\n1. **Experiments**: I feel that the claims made by the authors, especially on ImageNet $\\mapsto$ CIFAR10 transfer learning setup, are not well justified by the experiments. \n\n* **Hyper-parameter setup**: The authors use a fixed learning rate 0.001 and number of epochs 100 for all the setups? How are these hyper-parameters determined? Could it be the case that this particular learning rate setting favors the fixed polarity baseline more?\n\n\n* **Sec 3, ImageNet pre-training for polarity determination**: Obtaining weight polarities from the ImageNet trained model, and then training it on CIFAR10, and comparing it with randomly initialized model on CIFAR-10 seems unsatisfying. It is well-known in the over-parameterized network’s theory that the SGD trained network remains close to the initialization, so already by transferring the polarity from ImageNet to CIFAR-10, you start with a model with a good knowledge of the task making the overall comparison unfair. \n\n* **Sec 4: Transferring polarity is better than transferring weights is not well-justified from the experiments**: It can be seen from Fig 3, that in fact transferring the weights and freezing them works better than transferring polarity in terms of statistical efficiency. Similarly, from A.3 it can be seen that for different thresholds of the validation accuracy different methods work better making it difficult to see whether there is a clear pattern where one method is consistently better. In fact, the authors themselves say that, given their experiments, “... polarity configuration is an effective medium, if not superior to weights pattern”. If it is not consistently superior I am not sure then what’s the point of the experiments and how can the authors conclude that polarity is a more effective and faster transfer mechanism. By the way, it could still very well be the case that in all these experiments the hyper-parameter settings are sub-optimal for the weight transfer baselines (as mentioned above). \n\n\n* Overall, I feel that the authors only reported their experiments on two rather small settings, one of which was in fact a toyish synthetic setting. Even on CIFAR10, the experiments were not convincing enough to establish the points made by the authors. \n\n\n2. **Writing**: It seems to me that the paper was written with a neuro-science audience in mind whereas most of the ICLR audience are ML researchers. Since the arguments made in the paper are not very difficult to follow so it may not make much of a difference. Nevertheless, it would be good to mold the paper along ML lines. For example, define the setup properly – metrics, train/ eval protocol etc, define what is a “Success rate”. Maybe even define what is the meaning of “polarity”. Overall, make paper more self-contained.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject