Dictionary Contrastive Learning for Efficient Local Supervision without Auxiliary Networks
Authors: Suhwan Choi
Myeongho Jeon
Yeonjung Hwang
Jeonglyul Oh
Sungjun Lim
Joonseok Lee
Myungjoo Kang
Venue: ICLR 2024 spotlight (2024)
Keywords: Contrastive learning
Forward learning
Local learning
Image classification
Efficient learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a simple and efficient local contrastive learning objective that directly compares local features with label embeddings.

Abstract: While backpropagation (BP) has achieved widespread success in deep learning, it
faces two prominent challenges: computational inefficiency and biological implausibility.
In response to these challenges, local supervision, encompassing Local
Learning (LL) and Forward Learning (FL), has emerged as a promising research
direction. LL employs module-wise BP to achieve competitive results yet relies on
module-wise auxiliary networks, which increase memory and parameter demands.
Conversely, FL updates layer weights without BP and auxiliary networks but falls
short of BP’s performance. This paper proposes a simple yet effective objective
within a contrastive learning framework for local supervision without auxiliary
networks. Given the insight that the existing contrastive learning framework for
local supervision is susceptible to task-irrelevant information without auxiliary
networks, we present DICTIONARY CONTRASTIVE LEARNING (DCL) that optimizes
the similarity between local features and label embeddings. Our method
using static label embeddings yields substantial performance improvements in the
FL scenario, outperforming state-of-the-art FL approaches. Moreover, our method
using adaptive label embeddings closely approaches the performance achieved by
LL while achieving superior memory and parameter efficiency.

Published as a conference paper at ICLR 2024
DICTIONARY CONTRASTIVE LEARNING FOR EFFICIENT LOCAL SUPERVISION WITHOUT AUXILIARY NETWORKS
Suhwan Choi Myeongho Jeon'
Yeonjung Hwang'
Jeonglyul Oh
Sungjun Lim
Joonseok Lee 1,3 Myungjoo Kangln lef Scoul National University, aC,, *Google Research (schoi828, andyjeon, hyjok, jamesch0813, 1sjung567, joonseok, mkang) @snu.ac. ac.kr
ABSTRACT
While backpropagation (BP) has achieved widespread success in deep learning, it faces two prominent challenges: computational inefficiency and biological implau- sibility. In response to these challenges, local supervision, encompassing Local Learning (LL) and Forward Learning (FL), has emerged promising research direction. LL employs module-wise BP to achieve competitive results yet relies on module-wise auxiliary networks, which increase memory and parameter demands. Conversely, FL updates layer weights without BP and auxiliary networks but falls short of BP's performance. This paper proposes a simple yet effective objective within a contrastive learning framework for local supervision without auxiliary networks. Given the insight that the existing contrastive learning framework for local supervision is susceptible to task-irrelevant information without auxiliary networks, we present DICTIONARY CONTRASTIVE LEARNING (DCL) that opti- mizes the similarity between local features and label embeddings. Our method using static label embeddings yields substantial performance improvements in the FL scenario, outperforming state-of-the-art FL approaches. Moreover, our method using adaptive label embeddings closely approaches the performance achieved by LL while achieving superior memory and parameter efficiency.
INTRODUCTION
Backpropagation (BP) (Rumelhart et al. 1986) has been a fundamental tool in deep learning. However, BP exhibits two inherent limitations. Firstly, the requirement for weight symmetry during forward and backward passes renders BP biologically implausible (Liao et al., 2016). While the causal relationship between biological fidelity and the effectiveness of learning algorithms has yet to be established clearly, numerous deep learning studies have been focused on emulating human biological and cognitive processes (Fei et al.) 2022 Taniguchi et al., 2022). Secondly, forward passes can initiate only when backward passes are fully completed (backward locking), and the same applies in reverse (forward locking), which results in computational inefficiencies due to limited parallelization. Furthermore, because weight gradient computation requires storing local activations of each layer, memory usage is also inefficient.
In response, several alternatives to BP have been presented. Feedback alignment (FA) (Lilicrra et al. 2014 2016) substitutes symmetric feedback weights with fixed random weights. However, remains constrained by forward/backward locking. Its successor, Direct Feedback Alignment (DFA)
(Nokland 2016). directly propagates error signals to each layer to alleviate backward locking. Yet, DFA does not resolve the forward locking problem. For this, local supervision leverages local weight updates by minimizing local losses. Specifically, local learning (LL) (Nøkland & Eidnes, 2019 Belilovsky et al. 2020) employs local BP with module-wise auxiliary networks, which process local outputs to align with the targets for local loss computation. Auxiliary networks allow existing LL to achieve performance comparable to BP, but using them at every module substantially increases model parameters.
IPAI (Interdisciplinarry Program in Artificial Intelligence, Seoul National University) Corresponding authors

---

Published as a conference paper at ICLR 2024
A newer local supervision approach takes bold leap by entirely eliminating BP and auxiliary networks from LL. In this paper, we refer to this method as forward learning (FL). In FL, weight updates for each layer guided by layer-specific local losses, avoiding forward/backward locking issues and leading to substantial improvements in computational efficiency. In the absence of BP and auxiliary networks, the essential aspect of implementing FL. lies in formulating local targets for loss computation. For instance, the
algorithm (FF) (Hinton 2022) defines local targets by overlaying one-hot encoded labels onto images, treating them as individual pixels within the image. Since the local outputs include the target information, FF-based methods (Ororbia & Mali 2023) Lee & Song 2023) optimize the self-dot product of local outputs for contrastive learning objectives. However, this makes contrastive learning in the FL scenario susceptible to task-irrelevant information in the local outputs, resulting in subpar performance compared to BP and LL.
Our investigation suggests that auxiliary networks play crucial role in mitigating the impact of task-irrelevant information. In response the challenges posed by the absence of auxiliary networks, we propose straightforward yet effective local contrastive learning objective, DICTIONARY CON- TRASTIVE LEARNING (DCL), which effectively aligns local outputs with label embedding vectors. We evaluate two versions of DCL: one employing static label embedding vectors tailored for the FL scenario and another featuring adaptive label embedding vectors. Remarkably, our static method significantly outperforms state-of-the-art FL. baselines in the FL scenario by discarding task task-irreevvant information more effectively. Moreover, our adaptive method showcases performance on par with BP and LL while markedly surpassing LL in terms of parameter and memory efficiency. Further extensive analyses not only support our intuitions and the effectiveness of our method but also unveil intriguing properties of label embedding vectors.
2
RELATED WORK
2.1 LOCAL LEARNING
The current research trend encompasses two crucial directions in the field of LL. Nokland & Eidnes (2019) stands surpass the performance of BP. They utilize two distinet local losses, each originating from a separate auxiliary network pathway. The reconstruction loss measures the I.2 distance between the self- similarity matrix of one-hot encoded labels and the local features, while the label prediction loss employs cross-entropy, commonly used in both BP and LL's local layers (Belilovsky et al. 2019 2020 Pathak et al. (022). Nøkland & Eidnes (2019) also introduce biologically plausible, BP-ffre versions of these losses, i.e., bpf The reconstruction loss uses a standard deviation operation instead of a convolutional auxiliary network, and the label prediction loss employs feedback alignment for weight updates.
of the pioneering studies to illustrate that non-BP learning algorithm can
In a different vein, WWnn et al.] 2020) conduct an information theory-based analysis of LI.. The authors investigate the influence of the local loss objective on mutual information metries: I(h, x) (between local features and inputs) and / [hh, v) (between local features and labels). Compared to BP, [h, x) and I(h, y) exhibit more pronounced decreases as the LL's layers deepen. To mitigate the loss of I(h, x) during the forward passes, the authors introduce module- wise reconstruction loss, which encourages local features to preserve input information. Furthermore, they prove that minimizing the supervised contrastive loss in Eq. (T) (Khosla et al.) 2020) maximizes the lower bound of T(h,y) y)- Utilizing both contrastive and reconstruction losses leads to significant performance improvements, although it comes the cost of increased computational demands stemming from the module-wise reconstruction loss.
2.2 FORWARD LEARNING
As FL. aims to avoid reliance on BP, it limits the use of auxiliary networks, which guide features toward their targets with local updates by BP. In the absence of these auxiliary networks, the formulation of local targets becomes crucial. To address this challenge, the Forward Forward Algorithm (FF) Hinton 2022 defines local targets by overlaying one-hot encoded labels onto images, treating them as individual pixels within the image. Employing Noise Contrastive Estimation (NCE) (Gutmann & Hyvärinen 2010), FF optimizes the self-dot product of local features to be above a certain threshold if the overlaid labels match the images (positive pair). Otherwise (negative pair), local features
2

---

Published as a conference paper at ICLR 2024
optimized to be under certain threshold. Other FF-based approaches follow the same NCE objective and label overlay, with specialized architecture (PFF) (Ororbia Mali 2023) or refined label overlay (Lee & Song 2023).
technique (SymBa)
Taking a distinct path apart from FF, the cascaded forward algorithm (CaFo) Zhao et al. 2023 linearly projects flattened local features to local label prediction. To achieve this without BP, Caro freezes the weights of feedforward layers, reserving weight updates solely for the local linear projection layers. On the other hand, the direct random target projection (DRPT) (Frenkel al.) 202T) treats one-hot encoded labels as error signals themselves and uses fixed random weights to propagate the error signals each layer. As one-hot encoded labels are locally accessible, parameter updates can occur during each forward pass.
3 BACKGROUND
Contrastive learning stands as a power- ful tool for representation learning, and its efficacy has also been demonstrated the context of LL and FL. InfoPro (Wang) et al. 2020), an LL method, compares lo- cal features derived from module-wiss aux- iliary networks. In contrast, FF-based ap- proaches leverage self-dot products of local features, as the features contain the label information. In Wang et al. (2020), for a batch of local outputs h R x from a forward pass layer, the local contrastive loss is defined as follows:
RosNee:3 VGGB
1:72
contrastt
20.44
Lfett
16,76
10
15
20
25
30
35
40
Test Ervor (%)
Figure I: Comparison of Lreat with Leontast on CIFAR- 10. Both losses employ Eq. {T). but Lieat utilizes no auxiliary network, such that (h) ) = h. Training de-
tails are available Appendix J.2.4
exp (ap(aaa a(/Ta/T)) E lip (a: as//T)
Ceomtrast
0 log
fo (h:), (1)
Eijs 1
where is temperature hyperparameter, y E {I,..., 2} is the ground truth label, and fo is an auxiliary network. Eq. ( a; and aj are positive features, such that y/s = m/. This function aims to maximize the similarity between positive features while minimizing that between negative features. When the auxiliary network fo is an identity function, this objective represents the FL scenario. For convenience, primary objective of this article to enhance the performance with contrastive learning in the absence of auxiliary networks. To that end, Lreat can be regarded as the foundational framework that will be further elaborated upon in the following sections.
Lieat denote Lcomtrast with (h) h. Please note that the
Although both based approaches and InfoPro exploit concepts of contrastive learning to formulate local objectives, the performance of FF-based approaches falls short compared to InfoPro (LL). Additionally, with precisely the same setup. we compare the performance of Lconiast and Leea and report the significant performance gap in Figure [ These findings underscore the significance of auxiliary networks local contrastive learning, setting the stage for our goal to develop a local contrastive learning framework that excels without auxiliary networks.
4
METHODOLOGY
4.1 MOTIVATION
To enhance the model's performance using local contrastive learning without auxiliary networks, we commence our method design by examining the role of auxiliary networks. We suggest that the notable disparity in performance between Leontrast and Lient can be attributed to the presence of the mutual information rh, where r, referred as nuisance, denotes task-irrelevant variable in
x. Then, given a task-relevant variable y. it follows that [(r, y) = O because mutual information signifies the amount of information obtained about one random variable by observing another (Achille) Soatto, 2018). Lreat maximizes similarity between local features (h +T h²), rather than similarity between h and labels y. Accordingly, maximizing the similarity between local features could also

---

Published as a conference paper at ICLR 2024
Detached Forward
Label Embedding Vector
Forward Pass
Error Signal
Local Layer
L cross
Dictionary D
dict
Dog
Bind
to
t t2
O
t
Figure 2: Overview of DCL. Every input is detached before forward pass, ensuring that no layer propagates error signals backward. The final layer ft receives its error signal from the cross-entropy loss Leross. Every other layer receives its error signal from Ldiat. which optimizes the similarity between wise local features and label embedding vectors.
lead to an increase I(r (P)), misleading the model to consider task-irrelevant information as meaningful features.
In this respect, auxiliary networks have the capacity to filter out r, reducing the impact in LL (see Appendix A more details). However, in FL where auxiliary networks unavailable, the influence of r becomes more detrimental and noticeable. This likely explains the subpar performance of existing contrastive learning in the FL scenario.
4.2 DICTIONARY CONTRASTIVE LOSS
To address the problem with in FL, we propose novel objective that directly maximizes the similarity between h and embedding vectors corresponding to target labels.
Mapping labels to embedding vectors. To obtain label embedding t, from each target label ye- define embedding mapping function fn- The embedding mapping function fm N -> CD one-to-one mapping from a label to a Cp-dimensional label embedding vector, which can be directly compared with dense local features. Every label embedding vector t is initialized as the standard normal random vector, each element of which is i.i.d. random variable sampled from the standard normal distribution. For Z label classes, have label embedding dictionary D = (fm( Us E {1...., }}}. where fm (Uz) = t..
Local features. We aim to optimize the similarity between label embedding vectors t and local features h. First of all, because the shapes of local features may vary across different architectures, we standardize the representation of h. We represent local features at the I-th layer as h E IRC, x where K, is the number of C₁2 dimensional feature vectors. Because Cj, can differ for each layer 1, define the label embedding vector dimension Cp as max CY For fully connected layers (FC), we reshape flat output vector haat E RRC, Ki into by E CRC, x (See Appendix E for experiments concerning the optimal selection of K). For convolutional layers, local outputs feature maps h CRC HixW; where C₁, signifies the channel dimension, whereas H, and W, denote the feature maps' height and width, respectively. We maintain the integrity the C₁2 di-deiensionnl vectors within the feature maps by setting K₁ W,, To prevent BP across layers, employ the stop gradient operator ag|-], such that h; =
Training objective. The weights the final prediction layer fr. are updated through the standard cross-entropy loss. All other layers fi for = 1,.... . L
leverage the dictionary contrastive loss
4

---

Published as a conference paper at ICLR 2024
Laiet to update their weights. For batch of local features { 1 minimize the loss:
N
exp( (ep(((nntt.))) exp( (te {2))
Ldict
log n=1
t' E (t=)) E D
N
where define h, vector t+ corresponds to the label of h.. The dimension of local feature vectors may vary across different layers 1. To align the vector dimension of tz E RDCD to that of h R,C, we employ the one-dimensional average pooling pool: RCD R differently for each layer I.
h
denotes the dot product, and the label embedding
In this paper, we apply static label embedding vectors solely in the FL. scenario to address the con- straint that layer weights and label embedding weights cannot be updated simultaneously. Technically, this limitation makes our adaptive method partially forward/backward locked, although this locking limited in scope and negligible in practice. Nonetheless, for a fair evaluation against other FL unlocked, our static method maintains initial label embedding weights constant throughout the training process, such that to tztatic sg sg]ts] contrast, in scenarios not bound by the FL-specific constraint, label embedding vectors are adaptive, updating their weights every layer via error signals from Lasa- Figure illussrates the training workflow using Ldiet. We establish that the minimization of Lditt maximizes lower bound of / 1(h, y) in
approaches, which are fully forward/backwasd
Appendix B
Comparison with other contrastive objectives. Contrastive objectives based on InfoNCE (Oord
al. (2018) are known for their sensitivity the size of negative samples (Khosla et al.) 2020
Radford et al. 2021] Chen et al., 2020). These contrastive objectives, including Lreat, often utilize
in-batches negative samples and tend to show improved performance with larger batch sizes N (Wang
et al. 2020 Lee et al. 2018). In contrast, the number of negative samples in Laia corresponds to
Hence, the efficacy of Laia depends on the number classes. Empirical results confirm that
higher number of label classes Z tends to yield more pronounced performance when compared to using static label embedding vectors. Nevertheless, competitive performance still achieved even with fewer classes.
Layer-wise prediction. Minimizing Laiar maximizes the similarity between local features h. and their corresponding label embedding vectors t4. while concurrently minimizing the similarity to non-corresponding label embedding vectors. Leveraging this property of Ediet, D2 can be employed for inference without the final linear classifier fL. Predictions can be generated by selecting the target label with the highest similarity to the feature vectors:
arg max(h, (,))), t' (t. D
Accordingly, prediction is possible at every layer. Furthermore, this allows for weighted sum of layer-wise predictions to serve as the global prediction as in [Belilovsky et al. (2019); Zhao et al. (2023). This approach surpasses predictions made solely by fz. Experiments layer-wise prediction are available in Appendix C
EXPERIMENTS
5.1 EXPERIMENTAL SETUPS
Due to the substantial performance disparities between FL and LL, we conduct separate comparisons. In the FL scenario, our static method DCL-S, featuring static label embeddings, is evaluated using simple fully connected (FC) and convolutional (Conv) architectures line with the FL baselines. For our adaptive method DCL, we employ the VGG8B (Simonyan & Zisserman 2015) architecture as utilized by Nokland & Eidnes 2019).
In practice, we for R Σk_ (h,, t) over (hn, t). While both are mathematically equivalent given the linearity of the dot product, averaging feature vectors first leads performance degradation due a greater loss floating- point precision.

---

Published as a conference paper at ICLR 2024
Table 1: Test errors and the number of parameters with convolutional networks. We highlight the top-performing results in bold and the second- best results by underlining them. Results marked with an asterisk (+) indicate replicated results.
Table Test errors and the number parameters with fully connected networks. The same archi-
tecture is used except for PFF and DRTP.
Appreach
MNIST
CIEAR-10
CIFAR-100
Punons
Poruns.
Approach
MNIST
CIFAR-10
CIFAR-100
BP
L.7M
1.29
18.9M
34.73
19.2M
65.94
Punans.
Ert.
Paramt
Exc
Panana
Erz
187M
1.36
18.9M
41.00
19.2M
'95.20
22.84
PFF
23.0M
1.34
32.4M
150.14
32.7M
*81.31
CaPo
243K
1.20
243K
32.57
2.4M
99.24
FL
DRTP
6.3M
4,00
16.5M
51.27
46.4M
388.32
FL
DRCP
L8M
1.48
4.1M
31.04
19.2M
765.02
SyпBа
5.6M
1.42
18.9M
40.91
19.2M
70.72
43.67
Cest
1.77M
3.13
18.9M
43.29
19.2M
70.67
DOL-8
152K
3.21
153K
25.86
143M
54.21
DCC
1.87M
1.46
18.9M
35.12
19.2M
66.48
Table 3: Test errors across different datasets using the VGG8B architecture employed by Nokland & Eidnes (2019). LL-predsim denotes LL model trained with both prediction (LL-pred) and similarity mantlaitt-matching matching loss (LL-sim). LL-bpf, the BP-free version of LL-predsim, also utilizes two local losses explained in Section 2.1 LL-contrec signifies
LL model trained with both Lcontrast in Eq. (T) (L.--cont) and image reconstruction loss used by Wang et al. 2020), as explained in Section 2.1
The top-performing results are highlighted in bold. The second- best results are underlined.
Loss Type
Method
MNIST
F-MNIST
CIFAR-10
CIFAR-100
SVHN
STL-10
Single Global Loss
0.26
4.53
5.99
26.20
2.29
33.08
LL-contrec
*066
*5.71
'9.02
*3.33
'2.34
*29.74
Two Local Losses
LL- LL-predsim
0.31
4.65
5.58
24.10
1.74
20.51
LL-bpl
*0335
*5.68
9.02
*37.6
22.31
266.12
LL-cont
*0.37
*5.92
*7.72
*31.19
*2.29
26..8
Single Local Loss
LL-pred
0.40
5.66
8.40
29.30
2.12
26.83
LL-sim
0.65
5.12
7.16
32.60
1.89
23.15
Single Local Loss
DCL
0.33
5.52
8.64
31.75
2.19
22.87
Experiments with FC and Conv architectures. We compare our method against FL approaches, including Lieat, as well as the standard BP. These comparisons are carried out MNIST (LeCun) 1998), CIFAR-10, and CIFAR-100 (Krizhessky et al. 2009). We benchmark our method against FL
models: PFF Ororbia & Mali 2023). DRTP (Frenkel al. 2021 (Hinton 2022), and SymBa (Lee & Song 2023). Our FC models share the same FC architecture as FF and SymBa for a fair comparison. We also evaluate against Conv models: DRIP and CaFo (Zhao et al. 2023). Since DRTP and CaFo use different architectures, we employ a Conv architecture with fewer parameters. The BP and freat baseline models are trained using the same architectures and hyperparameters as models. Details on architectures, datasets, and training setups provided in Appendix
Experiments with the VGGSB architecture. We employ the VGG8B architecture to compare against LL, BF-free LL (LL-bpf), and BP. Our evaluation encompasses the same datasets employed by Nokland Hidnes (2019): MNIST, CIFAR-10. CIFAR-100, Fashion-MNIST (Xiao et al. (017). SVHN (Netzer al.] 2011). I), and STL-10 (Coates al. 2011). We the batch size, learning rate, learning rate schedule, and training epochs used by Nøkland & Eidnes (2019). Please refer to Appendix for more details.
5.2 MAIN RESULTS
Comparison with FL approaches. We first compare our static method against BP and other FL approaches. Table report the test error and number of parameters on MNIST, CIFAR-I0, and CIFAR-100. On MNIST, CaFo continues exhibit the best performance among FL models. However, our objective outshines other FL approaches for more realistic datasets (CIFAR- 10 and CIFAR-100). Table [ that our Conv model outperforms other FL models significantly despite having fewer parameters. Our FC models also surpass other FC models on CIFAR- 10 and CIFAR- depicted in Table Furthermore, method consistently outperforms Lieat across all datasets.

---

Published as a conference paper at ICLR 2024
Table Comparison of GPU memory usage and model parameters for VGG8B models. Memory denotes the peak GPU memory consumption measured during single GPU training with a batch size of 128. A represents the increase in the number of parameters compared the BP baseline.
MNIST, F-MNIST CIAFR-I0. SVHN
CIAFR-100
STL-10
Method
A
Memory
A0
Memory
A0
Memory
A
Memory
BP
847 MiB
0
1086 MiB
0
1088 MiB
2315 MiB
LL-contrec
1.15M
811 MiB
2.07M
1049 MiB
2.07M
1050 MiB
2.07M
5954 MiB
LL- LL-predsim
9.53M
1038 MiB
9.60M
1291MiB
10.9M
1310 MiB
9.60M
2594 MiB
LL-bpf
3.28M
708 MiB
3.28M
895 MiB
3.28M
897 MiB
3.28M
1851 MiB
L.L-cont
918K
695 MiB
1.84M
894 MiB
1.84M
895 MiB
1.84M
1846 MiB
LL-pred
71.8K
682 MiB
143K
870 MiB
1.43M
890 MiB
143K
1826 MiB
LL-sim 9.46M
933 MiB
9.46M
1154 MiB
9.46M
1156 MiB
9.46M
2290 MiB
DCL
5.12K
580 MiB
5.12K
747 MiB
51.2K
751 MiB
5.12K
1589 MiB
Overall, for each architecture type, our method demonstrates superior performance and scalability compared to other FL methods.
Comparison with LL approaches. We then compare adaptive method against LL and LL-bpf. exbhet3 the test error across various datasets. Our methods outperform the BP baselines on the SVHN and STL-10 datasets. For each dataset, our models exhibit competitive performance against L.L. models trained with single local loss, as indicated in Table While our models generally perform well, L.L. L.L-prdssim, trained with two local loss functions, still outperforms our method. However, across all datasets, our models consistently achieve better results than LL-bpf, the BP-free version of LL-predsim. It also worth highlighting that auxiliary networks in LL entail a significant increase in the number of parameters. In contrast, our method introduces much fewer additional parameters: Z x CD. highhlitts that our methods achieve better memory efficiency over both LL and BP and need fewer parameters compared to LL.
5.3
FURTHER ANALYSIS AND DISCUSSION
0.46
0.95
0.58
of l(hx)
Eslimtes 10by)
Esiimiss I(h
0.42
0.9
0.5
0.66
0.8
0.42
4
Ediet
non static
Diaat
Fpontrant
0.3
0.75
Lay Index
5
Layer Index
Layer Index
(b)
(c)
Figure 3: Task-inrelvvan information captured by intermediate layers of VGG8B. (a) Estimates of I [h, æ). (b) Estimates of / (h, y), and (c) Estimates of [(h,r) r). We train networks to reconstruct input images from local features h, employing the train loss the estimate of I(h, x). Likewise, we train networks to classify labels from local features h and utilize the accuracy as the estimate of ((h,)) y). I(h, x) [h,,)) y) equals the upper bound of I(h,r) (Wang al. 2020). but the substraction yields negative value because Tese (h, x)) < Test (h, y). To rectify this, we adjust the scale of the estimated upper bound by adding 1. More details the mutual information experiments are available in Appendix J.2.7
Robustness task-irrelevant information. To recast motivation discussed in Section 4.1 we amalyze our objective in comparison to Crem and Lgvatest regarding the presence of task-irrelevant information, as depieted in Figure Table and Table 2 highhighh the performance gaps between
7

---

Published as a conference paper at ICLR 2024
Ldiet and Lieat, especially with convolutional networks. We interpret these differences through the lens of task-irrelevant information. As our assumption, Figure exhiibits that Lieat encounters the nuisance problem: that is, there is reduction of [(,, r) from the baseline. In contrast, Laiet effectively reduces r) starting from the fourth layer, ultimately matching the T(h, r) level achieved by auxiliary networks (Ccontrast), even when label embedding vectors static. Appendix J.2.7 presents the methods used for estimating mutual information in detail.
VGG8B
CIFAR-100
CIFAR-10
D D
D
Conv
FC
10
20
30
40
50
60
70
Test Enror
Figure 4: Performance with different types of dictionaries. D dictionary of adaptive label embedding vectors. D contains static, standard normal random vectors. D consists of static, orthogonal vectors. Vectors in both D and D have been scaled to match the norm D
The effectiveness of adaptive embeddings. Figure 4 depicts the effectiveness of adaptive embed- dings in contrast static embeddings across CIFAR-10 10 and CIFAR 100. Models trained with dictionary of adaptive label embeddings D consistently outperform models trained with a dictionary of static label embedding vectors, regardless of whether the compared static embeddings are random (D-) orthogonal (D ). In particular, the performance gaps for convolutional architectures are more pronounced on CIFAR-100 than on CIFAR-I0.
Ground Truth
Top-1 Prediction
Top-2 Prediction
Ground Truth
Top-1 Prediction
Top-2 Prediction
Trick
Truck
Automobile
Ship
Ship
Airplane
Horse
Deer
Dees
Figure 5: Saliency maps. For a label yi, saliency corresponds to the dot product between the k-th feature vector h and the embedding vector of yi- The color on the heatmaps represents the level of saliency, For visualization on the heatmaps, saliency is normalized between and 1.
Explainability with label embeddings. Figure 5 showcases visualizations of the saliency maps for the two labels with the highest confidence as predicted by the final layer VGG8B trained on CIFAR-IO. These saliency maps generated through the dot product between the label embedding vector and individual local feature vectors, with each local feature vector representing a distinct region within image. Regarding the top-1 labels, there clear alignment between the saliency maps and the regions within the input images that are relevant to those particular labels. For example, the saliency linked the top-1 label "Horse" precisely matches the horse's 's body.
The saliency maps can also provide a convincing explanation for the high confidence incorrect labels. In Figure the saliency of the top-2 label "Deer" in the "Horse" image suggests that the model
8

---

Published as a conference paper at ICLR 2024
is hallucinating antler, confused by the two trees near the horse's head. This misinterpretation occurs because antlers often generate elevated saliency for the "Deer" label, as demonstrated by the saliency associated with the top-1 label "Deer" in the corresponding "Deer" image. More examples are available in Appendix K.3
Semantic properties of adaptive embeddings. Figure the semantic relationship between adaptive label embedding vectors on CIFAR-100, which comprises 20 super-labels, each encompassing 5 sub-labels. The clustering of these vectors high- lights their capacity for semantic learning, akin to their alignment with label-specific salient features in Figure S
chair coudh table television had computer keyboard
plain
cloud mountain
warzmbe
lamp
forest trees
worm snake clock
dinasaug tur
procodile
spider
lizard labster
Embeddings from the same super-labels tend cluster together, while those from seman- tically similar but different super-labels also show proximity. For example, "forest" closer to "trees," and "chimpanzee" is closer to "people" than to other embeddings within their super-label groups. Please refer to Appendix D for additional experiments on super-labels.
people
people
famiure
chimpanzee
dlophant
shall
cattle
kangeroo
Figure 6: t-SNE of label embeddings. See Appendix K. for the complete plot.
Adaptive label embeddings and confusion rates. Figure capptures the correlation between the confu- sion rates and adaptive label embedding similarity on CIFAR-10. With the labels yi and Vy, the confusion rate for y4 and Y is defined by the average of confij and conf Here, confusion conf,; is the test error incurred when incorrectly predicting Yi as yn-
2.5
Confission Rette Rate (%)
1.5
Figure / illustrates that as the confusion rate between labels increases, the embedding representations of these labels become more distinct. This suggests that when the model struggles to discriminate between two labels, it compensates by dynamically adjusting the label embedding space to facilitate better label separation.
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Cosine Simiarlity Between Label Embeddings
Figure 7: Relationship between confusion rates and label embedding similarity. Each point represents a pair of labels yi and yj- Please refer to Appendix K.1 for exact values.
Comparision between average pooling and projection.
Table 5: Projection vs. average pooling.
We consider fully connected layer fo RCD RC as substitute for the one-dimensional average pool- ing poolt employed in Eq. (2). The linear projection layer f/P maps -p-dimensional label embedding vectors to Cj di.-iimesioonal label embedding vectors, such that ft,) = E.. Table illustrates that poot' outperforms for in performance and memory/parameter efficiency.
CIAFR-10
CIAFR-100
Ere.
Memory
Ere.
Memory
12.47
753 MiB
36.08
770 MiB
pool
8.64
747 MiB
31.75
751 MiB
CONCLUSION
In this paper, we find that the limited efficacy of conventional contrastive learning objectives without auxiliary networks primarily arises from the inclusion of fask-irrelevant information. To address this challenge, we present novel objective, DCL, which directly aligns local features with label-specific embedding vectors. Even without auxiliary networks, our approach effectively discards iask-irreevvant information, outperforming other FL approaches by a large margin. In addition, our method using adaptive label embedding vectors achieves performance levels comparable to those of BP and LL while maintaining superior purameter/memooy efficiency compared to LL. Further extensive analyses support our model design and its effectiveness. We aspire for this work to pave a valuable path forward, positioning DCL as formidable alternative to BP.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
This work was supported by 1) the NRF grants [2021RIA2C3010887, 202HHID3A2200008807, 0D22ICIII101622 RS-2023-0022663], 2) Artificial intelligence industrial convergence clus- ter development project funded by the Ministry of Science and ICT (MSIT, Korea) & Gwangju Metropolitan City, 3) Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) [2022-0-00264, 2021-0-01343-004, Artificial Intelligence Graduate School Program (Seoul National U and 4) Youlchon Foundation (Nongshim Corporation and affiliated companies). We also extend our sincere thanks to CRABs for their generous financial support and the provision of GPU resources.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19()):1977-198,, 2018.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Greedy layerwise learning can scale to imagenet. In International conference machine learning, pp. 583-593. PMLR, 2019.
Eugene Belilovsky, Michael Eickenberg, and Edouard Oyallon. Decoupled greedy learning of cnns. In International Conference on Machine Learning, pp. 736-745. PMLR, 2020.
Ido Ben-Shaul and Shai Dekel. Nearest class-center simplification through intermediate layers. In Alexander Cloninger, Timothy Doster, Tegan Emerson, Manohar Kaul, Ira Ktena, Henry Kvinge, Nina Miolane, Bastian Rieck, Sarah Tymochko, and Guy Wolf (eds.). Proceedings of Topological. Algebraic, and Geometric Learning Workshops 2022, volume 196 of Proceedings of Machine Learning Research, pp. 37-47. PMLR, 25 Feb- 22 Jul 2022.
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineering self-supervised learning. In Thirty-seventhh Conference Neural Information Processing Systems, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiv:181..01718, 2018.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 2253.2232222 JMLR Workshop and Conference Proceedings, 2011.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representutions, 2020.
Gamaleldin F. Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In Neural Information Processing Systems, 2018.
Nanyi Fei, Zhiwu Lu, Yizhao Gao, Guoxing Yang, Yuqi Huo, Jingyuan Wen, Haoyu Lu, Ruihua Song, Xin Gao, Tao Xiang, et al. Towards artificial general intelligence via multimodal foundation model. Nature Communications, 13(1):3094, 2022.
Charlotte Frenkel, Martin Lefebvre, and David Bol. Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks. Frontiers in neuroscience, 15:629892, 2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton. The forward-forwart algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Saraa, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, Advances in neural information processing systems, 33:1 18166-66117775 2020.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/nmist// 1998.
Heung-Chann Lee and Jeonggeun Song. Symba: Symmetric backpropagation-fie free contrastive learn- ing with worwadd algorithm for optimizing convergence, arXiv preprint arXiv:2303.08416 2023.
Joonseok Lee, Sami Abu-EI-Haija, Balakrishnan Varadarajan, and Apostol Natsev. Collaborative deep metric learning for video understanding. In Proc. of the ACM SIGKDD International conference on knowledge discovery & data mining, 2018.
Qianli Liao, Joel Leibo, and Tomaso Poggio. How important is weight symmetry in backpropagation? In Proceedings of the AAAI Conference Artificial Intelligence, volume 30, 2016.
Timothy P Lillicrap. Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random feedback weights support learning in deep neural networks. arXiv preprint #rXiv:141.00247 2014.
Timothy P Lillicrap, Daniel Cownden, Douglas B Tweed, and Colin J Akerman. Random synaptic feedback weights support eиror backpropagation for deep learning. Nature communications, 7(1): 13276, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Reprexentations, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Arild Nøkland and Lars Hiller Eidnes. Training neural networks with local error signals. In International conference on machine learning, pp. 4839-4850, PMLR, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /007.0087.33744 2018.
Alexander Ororbia and Ankur Mali. The predictive forward-forward algorithm. arXiv preprint arXiv:2301.01452, 2023.
Vardan Papyan, X. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 14652-24663, 2020. doi: 10.1073/pmas.20155099
Priyank Pathak, Jingwei Zhang, and Dimitris Samaras. Local learning on transformers via feature reconstruction. arXiv preprint arXiv:2212.142155 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
11

---

Published as a conference paper at ICLR 2024
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28729-28745. PMLR, 23-29 Jul 2023.
David Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back proppaatpggg errors. nature, 323(6688) 1986.
Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 7703.008000 2017.
K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. International conference on machine learning, pp. 1139-1147, PMLR, 2013.
Tadahiro Taniguchi, Hiroshi Yamakawa, Takayuki Nagai, Kenji Doya, Masamichi Sakagami, Masahiro Suzuki, Tomoaki Nakamura, and Akira Taniguchi. whole brain probabilistic generative model: Toward realizing cognitive architectures for developmental robots. Neural Networks, 150: 293-312, 2022.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer. Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mlp architecture for vision. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp. 24261-24272. Curran Associates, Inc., 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative end-to-end training. In International Conference on Learning Representations, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking 2017.
machine learning algorithms. arXiv preprint
Gongpei Zhao, Tao Wang, Yidong Li, Yi Jin, Congyan Lang, and Haibin Ling. The cascaded forward algorithm for neural network training, arXiv preprint WxXhv22333009223 2023.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A THE EFFECTIVENESS OF AUXILIARY NETWORKS IN DISCARDING IASK-IRRELEVANT INFORMATION
0.16
0.5
Fsemmtts of l(ho
fest
l(han)
contrast
6.0
0.23
0.48
0.30
0.51
0.3
0.75
04
5
Lay Index
Layer Index
Layer Index
(b)
(c)
Figure I: Information contents of local features from VGG8B models trained with Ccontrast and
Creas. (a) Estimates of mutual information between local features and inputs 1(h, æ). (b) Estimates of mutual information between local features and labels (h, y). (c) Estimates of task-irrelevant information in local features I [hh,r) r). I [hh x)) can estimated by reconstructing from h. I(h, y) can be estimated by classifying from h. We obtain the estimated upper bound of I(h, r) by subtracting Test y) from Test (h, æ). The subtraction results in negative values because the values of Test (h, x)) are smaller. Thus, we adjust the scale by adding 1. Implementation details are available Appendix 2.7
To demonstrate the effectiveness of auxiliary networks in reducing rh, compare local features of VGGSB models trained with Ccontrast and its auxiliary network-free counterpart, Creat Figure [nderlines the pronounced decrease of r in local features when auxiliary networks are employed, even from early layers.
B DEMONSTRATING THAT MINIMIZATION OF Ldiet MAXIMIZES THE LOWER BOUND OF TASK-RELEW INFORMATION
This section demonstrates that minimizing the dictionary contrastive loss,
exp(hn t+) Z n=1 Σ exp em tu)
N
K hn = Σ
Ldiet
log
h* t D
(4)
maximizes a lower bound of I(h, v). We follow the progression of the proof by Wang et al. (2020)
showing that minimizing Ccontrast in Eq. ( maximizes a lower bound of mutual information p(h.y)
(h, = En.y log p(h))p())
Suppose that sample local feature f. from set of N local features X (h1 hx J. Given that h2 corresponds to label yz. ft also corresponds to label embedding vector t because we have (mx))) = t., where the embedding mapping function fm is the one-to-one mapping. We use t. to denote such positive label embedding vector. For label classes, we have label embedding dictionary D t, D neg: where D {t [t.,...,tz)
Then, we have the expectation of Ldiet.
exp(h, E.) 2 (h t.))
E[Laict] = Ex
log
(5)
Minimizing Ldiet can be regarded as minimizing a categorical cross-entropy loss of identifying the positive label embedding vector correctly, given the local feature h. For examples, in Eq. 3 (h, t,) serves as the confidence of classifying h as the label yz- Thus, we can define the optimal probability ppos (t.) X) representing the true probability of t, being the positive label embedding vector. Because

---

Published as a conference paper at ICLR 2024
the label of h equals to the label of of by definition, it can be said that the positive label embedding
vectors are sampled from the true distribution p(t ) and the negative label embedding vectors are from the true distribution p(t). Accordingly, obtain the optimal probability follows:
plt, (b) p(t.)
pt,, h) hh)l.
P tt. (x)
(6)
ple, 16) p(ts]
Dj=1
Then, we derive Coptimal by using ppos (t+ X) the optimal value for Thus, assuming uniform label distribution of h, we obtain the inequality
exp(h,e+) in Eq. (5)
diet
Σラ exp((s,,))
plc : (h) p(e,)
E
[Leict]
2 E
diet
coptimal
Ex
log
ptt, (A) m(t,)
p(c, (h) p(f+)
Ex
log
(8)
p(t- 1h) + EijEDi p(t
ple, h) p(t,)
p(t+)
p(t h)
Ex
log
1 + (t+ h1) EjEDicu
(9)
(t,))
(t_)
((, h2)
2 Ex
log
(t+ (h) (Z - 1)BB, rpple)))())
(10)
(t,))
+ p ((+)
1/ (2-
= Ex
log
(11)
pt+ h1)
(t+ h) (Z (2-1)
(t++)
2 Ex,4
log
(12)
II + log (Z - -1 2 I(hh y) + log (Z - 1).
(13)
Eq. 10) is derived from Oord et al. (2018). according to which the approximation becomes more accu- rate Z increases. By the definition of mutual information, we have / [(h, t+) Ex +. log p(t [h)
p(
Due to the data processing inequality (Shwartz-Ze) Ziv Tishby, 2017). we have I [hh, y) 2 I(h, t.).
and thereby Eq. (13). Given the derived inequality [Caic log(Z 1) I(h, y), minimization of Ediet maximizes a lower bound of [h,, y).
C EXPERIMENTS WITH LAYER-WISE PREDICTION
Table I: Layer-wise test errors for FC and Conv models on CIFAR-10. Layer 3 correspond to the final classifier layer.
Table II: Layer-wise test errors for FC and Conv models on CIFAR-100. We report average test errors across 5 runs.
Layer
Layer
Layer
Layer
Layer
Layer
FC
70.95 + 0.25%
34.22
0.24%
34.62 + 0.30%
FC
72.82 0.31% 66.36
0.15%
65.85 + 0.19%
Conv
48.14 +: 0.22%
25.42 + 0.25%
25.63 +: 0.31%
Conv
71.11
0.21%
50.37
0.28%
49.78 +: 0.30%
This section evaluates the performance of layer-wise predictions in models trained with Laiet and adaptive label embeddings. Initially, we scrutinize the individual layer-wise predictions for the FC, Conv, and VGG8B architectures. Given the shallow nature of FC and Conv architectures, we further assess the VGG8B architecture using weighted sum of layer-wise predictions the global prediction.
C. METHOD
Models trained with Laiet can generate layer-wise predictions by leveraging Eq. 3 namely
arg max t.)), t' E (pool, (t=)lt= D²
(14)
ii

---

Published as a conference paper at ICLR 2024
Suppose that we have model consisting of layers (fi....,fii where fL is the final linear classifier
layer. Then, for I < L. we can obtain a layer-wise wise embedding vector t' E {pool,(t2)te E D By leveraging the similarity (h, Ez) as the confidence of classifying h as the label Vs. obtain layer-wise prediction by choosing the label with the highest confidence.
C.2 SINGLE PREDICTION
80
Since both FC and Conv architectures consist of three layers, we present their performance together.
74.78
CIFAR-100
70
Test Error (%) (%)
CIFAR-10
60
52.28
53.36
50
43.68
Results for the FC and Conv architectures. Table
36.37
Jand Table M summarize the performance of layer-
40
32.72
31.98
31.75
wise predictions. On CIFAR-10, Layer 2 predictions outperform the predictions from the final classifier layer for both FC and Conv models.
30
25,00
20
15.96
11.08
9.51
868
8.66
&66
10
Results for the VGGSB architecture. Figure displays the prediction performance for all layers of VGG8B. In both datasets, while the final layers provide the most accurate predictions, those after Layer are only slightly less accurate than the final layers.
2
6
7
8
Layer Index
Figure II: Test errors across layers of the VGG8B models. The 8-th layer corresponds to the final linear classifier layer.
C.3 WEIGHTED SUM OF LAYER- WISE PREDICTION AS THE GLOBAL PREDICTION
Weighted sum. For each intermediate layer
31,9
31,89
1, we can obtain a confidence vector ÿt (hh f (h, 122) (h, tzz))". Then, we obtain the global prediction y as the weighted sum of ÿI across
31.6
Test Entr (%)
31.3
31.25
31.30
31
10
31.47
31.65
30.00
I, such that y Di Argr. where a is the coeffi- cient controlling the influence of ÿI on g. We eval-
10
9.61
CIFAR
fired layer
9.6
CIFAR-10
uate y by setting \ IP, such that < p 10.
812
The trained VGGSB models from Section 5.2 are employed for these global predictions.
261
8.39
057
8.5
861 862
6
7
8
9
10
Figure details the global prediction re- sults for different p values. When p = all layers contribute equally, leading to a performance drop compared to the final classifier layer. As p increases, deeper layers exert greater influence on y. Higher values generally enhance predictions above the base- line, notably on CIFAR-100. Yet, an overwhelm- ing influence from deeper layers can degrade perfor- mance, regressing toward the baseline.
Results.
Figure III: different values. Here, adjusts the confi- dence level of each layer, such that x = IP. Baselines are derived from the average test errors of the final classifier layer. The under- lined values correspond to the baselines.
Test errors for global prediction at
D MULTIPLE LABEL EMBEDDINGS PER LABEL
This section delves into a situation where each class has multiple label embeddings. Specifically, we examine an ideal scenario where the variation within each class label is accurately and specifically identified. CIFAR-100, composed of 20 super-labels with 5 sub-labels each, serves as our testbed. Training on CIFAR-10K 100 and performing inference on CIFAR-20 essentially equates to employing 5 unique label embeddings for each super-label. This approach enables us to leverage the exact knowledge of distinct variations for each label.
Setup. We employ two distinct methods for inference to leverage the hicrarchical relationship between CIFAR-100 and CIFAR-20 labels. The Mean method averages the sub-label embeddings
iii

---

Published as a conference paper at ICLR 2024
within each super-label, creating unified super-label embedding for example, averaging embed- dings of "man," "woman," "boy." "girl." and "baby" to represent "pcople." During inference, the model predicts the super-label by identifying which averaged embedding most similar to the local features from the last layer h utilizing a similarity metric as outlined in Eq. (3. On the other hand, Supe r method predicts the super-label based on the closest proximity of h 1--11 any sub-label embedding within a super-label.
For comparative analysis, introduce two baseline approaches that utilize conventional feedforward processes for inference. Naive approach uses the super-label of the predicted sub-label as the final prediction. We utilize sub-label predictions by the models trained on CIFAR-100 1-00 from Section 5.2 For instance, if the model classifies an image as "baby." the associated super-label "people" is used as the final prediction. Meanwhile, E2E represents an end-to-end training method, where models are trained using Ldict on CIFAR-20 super-labels.
Results and analysis. Table M indicates that Super achieves the best performance overall. However, Mean does not exhibit a significant advan- tage over Na i ve, even showing performance decline for the VGG8B model. This could be attributed to the presence of outlier sub-labels in CIFAR-100, which can negatively skew the computed mean. For instance, as depicted in Figure VII the "forest" sub-label embedding is closer to the "trees" embedding than to other sub-labels in the "large natural out- door scenes" category. Consequently, an averaged embedding for "large natural outdoor scenes" may not effectively represent the nuances of this super-label.
Table III: Comparison of test errors on CIFAR-20
FC
Conv
VGG
E2E
56.65 40.72 23.42
Na ve
65.85 49.78 31.75
Supez:
60.95
37,56 21.84
60.98 48.61 32.26
It is also important note that the semantic relationships learned by Ldiet do not always align with the dataset's 's predefined hierarchy. For example, CIFAR-100 categorizes "chimpanzee" under "large omnivores and herbivores" alongside "elephant, cattle, camel, and kangaroo." " Yet, semantically, "chimpanzee" aligns more with "people," as reflected in the learned label embeddings in Figure VII This observation highlights that models trained with Laiet intuitively grasp the semantic relationships between labels, while learning less inherently semantic structure, such as taxonomy, requires explicit supervision.
E FEATURE SEGMENTS K
Figure IV iillustrates how performance varies across different values for CIFAR-IO and CIFAR-100, In our FC layers, we divide a local feature heat E CCK
73
Test Error (%))
into K segments of C-dimensional feature vectors mentioned in Section 4.2 For our FC models,
44
CK = 3072 across all intermediate layers. This transformation effectively converts the flat feature into grid-like feature h CCCK Then, Ldict optimizes the average similarity between label em- bedding vectors and feature vectors across K seg- ments. While the optimal number for K may vary across datasets, it is evident that the segmentation of features consistently leads to improved performance compared to the baseline K =1
CIEVR 100
40
36
32
4
8
12
21
32
48
61
96
Figure IV: Performance of FC models with varying numbers of feature segments K.
F DCL VARIANTS FOR PARALLEL TRAINING
Setups. DCL, our default approach, updates label embeddings following every intermediate layer's forward pass. While our experiments exclusively employ sequential training, we also explore parallel training scenarios where each layer operates on a distinct GPU in parallel. With DCL, label embeddings be updated via layer-wiss gradients averaged across all intermediate layers. However, this averaging, which integrates concurrent error signals from all layers, might negatively impact the weight updates. Thus, propose versions better suited for parallel training. DCL-O updates
iv

---

Published as a conference paper at ICLR 2024
label embeddings exclusively using the error signals from the last intermediate layer. In contrast, DCL-LD employs layer-wise dictionaries D {tz RCD, {l,... 2}}, allowing for parallel updates of wyserwwiee label embeddings.
Table IV: Comparison of DCL variants trained on the VGG&B architecture. Memory denotes the peak GPU memory consumption measured during single GPU training with a batch size of 128. Ae represents the increase the number parameters compared the BP baseline.
MNIST
CIAFR-10
CIAFR-100
STL.-10
Method
Memory
Err
Memory
Memory
Err.
Memory
DCL
5.12K
0.33
580
5.12K
8.64
747
51.2K
31,75
751
5.12K
22.87
MiB
DCL-0
5.12K
5.12K
747
51.2K
5.12K
MiB
DCL-LD
2188
0.34
581 MiB
21.8K
8.45
749 MiB
218K
31.64
766 MiB
21.8K
22.59
1593 MiB
Results. Table IV compares performance and parameter/memory costs for DCL, DCL-O, and DCL-LD across four datasets: MNIST, CIAFR-I0, and CIAFR-100, and STL-10. DCL-LD's layer- wise dictionaries might provide advantages in certain contexts, for CIAFR-10 and STL-10, albeit with higher parameter/memoory cost. DCL and DCL-O maintain a balance between efficiency and performance, with DCL-O slightly edging out in error rate reduction in less complex datasets. The increased parameter count for DCL-LD suggests a trade-off between representational capacity and resource efficiency, which may justified by its performance in more complex tasks.
in its competitive error rates
G EXPERIMENTS WITH OTHER ARCHITECTURES.
Table V: Comparison between Ldiet and Lcontrast across various architectures on CIFAR-10.
ResNet-32
ResNet-32--W
VIT
MLP-Mixer
A8
Ere.
Memory
A9
Err.
Memory
A9
Err.
Memory
A
Err.
Memory
BP
6.701 3179 MiB
0 4.781
13433MiB
16.25
5300 MiB
O 17.23 6361 MiB
Coontrast
73.6K 32.71 1617 MiB
266K 20.02
6378 MiB
394K 32.95
1354 MiB
394K 31.67
Ldiet
0.64K 25.19 1617 MiB
2.56K 14.36 5350 MiB 5.12K 32.63 1348 MiB
5.12K 23.51 1445 MiB
We evaluate Laiet against Lcontrast on CIFAR-IO using a range of architectures: ResNet-32 (He et 2016), its wider-channel variant ResNet-32-W. W, VIT (Dosovitskiy et al.) 2020), and MLP-Mixer (Tolstikhin et al. (202T). These architectures consist of multiple modules, each embodying functional unit and containing multiple fully connected convolutional layers. For instance, single module of MLP-Mixer is combination of channel-mixing and token-mixing layers, so that each module effectively blends information both channel-wise and token-wise. Our experimental setup applies Lditt at a module level, incorporating module-wise BP. However, unlike in Lcontrast. we do not employ module-wise auxiliary networks. Please refer Appendix J.2.3 for additional information.
Results. Table V demonstrates that Laier outperforms Leuntrast across various architectures while requiring fewer parameters and less memory. Although the performance difference for ViT is marginal, a considerable gap is observed in ResNet-32, ResNet-32- W. and MLP-Mixer.
H THE CHOICE OF SIMILARITY MEASURE
compare dot product and cosine similarity across different temperature parameter Although cosine similarity is a commonly employed similarity measure in contrastive learning, consistently delivers inferior performance when compared dot product, irrespective of
In Figure V

---

Published as a conference paper at ICLR 2024
80
95
70
(CIPAB-130)
Test Eloor (%G
Test Elaor (%G)
Test Entor(%)
detigns (C:PA 8-101
(CITAR
30
50
40
35
30
20
0.1
0.06
0.01
0.005
0.001
0,0005
0.0G
0.01
0.005
0.00E
0.0055
011
0.05
0.01
0,005 0.001 0,0036
T
(a)
(b)
(c)
Figure V: Cosine similarity vs. dot product across different architectures. (a) FC, (b) Conv, and (c) VGG8B. t denotes the temperature hyperparameter for cosine similarity. Across all tested architectures and datasets, dot product consistently outperforms cosine similarity.
I IN RELATION TO NEURAL COLLAPSE
Neural Collapse. During the late stages of training, various phenomena collectively known as neural collapse appear at the layer preceding the final classifier. The most relevant to our study the simplification to nearest class center (NCC): the network classifier tends select the class whose train class mean is nearest (Papyan et al., 2020). Previous research have identified the gradual emergence of neural collapse at local layers (Ben-Shaul
Dekel 2022 Ben-Shaul et al. 2023 Rangamanii et al. 2023). Moreover, it has been shown that inducing NCC in these layers through specialized local losses can be advantagcous (Ben- Shaul & Dekel 2022 Elsayed et al. 2018).
Regarding our work. By minimizing EiCaic Dz)], our approach can be seen as encouraging
NCC at local layers by aligning the local class center N. ΣN. h with label embedding vectors t.. akin to the motivation of using local losses by Ben-Shau & Dekel (2022); Elsayed et al. (2018). However, Ben-Shhul & Dekel (2022); Elsayed al. 2018) focus local features within end- to-end BP framework, whereas our work investigates locally decoupled layers, treating each as the final layer before the classifier. Our results indicate that global feedback, typical of BP, may not be crucial for neural collapse. In Figure we observe the improvement of layer-wise prediction through layers, consistent with Ben-Shaut al. (2023). Moreover, in Appendix [ clustering by semantic similarity, as opposed to predefined taxonomy, mirrors the clustering to semantic labels observed by encouraging NCC in self-supervised learning (Ben-Shaul et al. 2023
J IMPLEMENTATION DETAILS
J.1 DATASETS
FC and Conv experiments. For the Conv and FC architectures. we test our method on MNIST [LeCun 1998), CIFAR-10, and CIFAR- 100 (Krizhevsky et al.) 2009) datasets, in Table 2 and Table [] The MNIST dataset consists 60000 training and 10000 test samples, with 10 label classes. Fach sample is a 28 x 28 grayscale image. CIFAR-10 10 and CIFAR-100 provide 50000 training and 10000 testing RGB images of size 32 x 32, but with 10 and 100 label classes respectively. All three datasets maintain uniform label distributions across training and test sets.
VGGSB experiments. In Table 3 employing the VGG8B (Simonyan & Zisserman, 2015 Nøkland Eidnes 2019) architecture, our method was evaluated on datasets including MNIST, Fashion- MNIST (Xiao et al., 2017). CIFAR-10, CIFAR- 100, SVHN (Netzer et al. 2011). and STL-10 (Coates) ct al. 2011) datasets. Rushion-MNIST (F-MNIST) has 60,000 training and 10,000 test grayscale images with 10 label classes. SVHN, with 32 x 32 RGB images, comprises 73257 training, 531131 extra training, and 26032 test images. We adopt both training sets, following Nøkland & Eidnes (2019). STL-10 provides 5,000 training and 8,000 test RGB images (96 x 96). Labels of Fashion- MNIST, CIFAR-IM -00, CIFAR- 100, and STL-10 are uniform distributions for training and test samples. Nøkland & Fidnes (019)) also tested their models Kuzushij-MNNIST (Clanuwat al. 2018).
vi

---

Published as a conference paper at ICLR 2024
Because the newer, available version of Kuzushiji-MNIST is different, exclude Kuzushiji-MNIST from the experiment.
J.2 TRAINING DETAILS
Table VI: FC and Conv architectures. DCL and BP share the same architectures, except for D Likewise, DCL, FF (Hinton 2022), and SymBa (Lee Song 2023) share the same FC architectures.
K refers to the feature segments explained in Section
(MNIST)
FC (CIFAR-10)
FC (CIFAR-100)
Conv
Conv (DCL-S CIFAR-
D
x
64
100 256
10 256 (100 256)
x
K
48
Input Size
784
3072
3072
3x32x32
3x32x32
Unit
fe 1024
fe 3072
fe 3072
conv(3 x 3 x4, 1,
conv(3 x x 1,
Unit 2
ReLU
ReLU
ReLU
Batchnorm
Batchnorm
Unit 3
Layernorm
Layernomm
Dropout 0.3
ReLU
ReLU
Unit 4
fe 1024
fe 3072
LayerNorm
conv(3 x 3 x 256, 2.
conv(3 x 3 x 256, 2,
Unit 5
ReLU
ReLU
fe 1072
Batchnorm
Batchnorm
Unit
Layemorm
Layeroom
ReLU
ReLU
ReLU
Unit
fo 10
fe 10
Dropout 0.3
Global Avg Pooling
conv(3 512,
Unit
Layernorm
(100)
Batchnorm
Unit
100
ReLU
Unit 10
Global Avg Pooling
Unit 11
100
Table VII: Hyperparameters for training the FC, Conv, VGG&B architectures.
FC
Dataset
Epoch
Learning Rate
Decay Milestones
MNIST
150
0.0005
50 100 125
CIFAR-10
400
0.0002
50 150 200 350
CIFAR-100
200
0.0001
50 100 150
Cony
Dataset
Epoch
Learning Rate
Decay Milestones
MNIST
150
0.0075
50 75 100 125
CIFAR-10
500
0.0075
100 200 300 400 450
CIFAR-100
400
0.0075
200 250 350
VGG8B
Dataset
Epoch
Learning Rate
Decay Milestones
MNIST
100
0.0005
50 75 89 94
F-MNIST
200
0.0005
100 150 175 188
CIFAR-10
400
0.0005
200 300 350 375
CIFAR-100
400
0.0005
300 350
SVHN
100
0.0003
50 75
STL-10
400
0.0005
200 300 350
Our experiments conducted using Pytorch. Across all architectures. optimizer (Loshchilov & Hutter 2018) with the default Pytorch setting: 3 = 0.9, 32 = 0.999, and weight_deacy 0.011. Table VII details the training hyperparameters for every architecture.
the AdamW
vii

---

Published as a conference paper at ICLR 2024
Table VIII: Backbone architecture VGG8B. All convolution layers utilize 3 kernels, with stride and zero-padding set to 1.
Layer
Layer
Layer
Layer
Layer
Layer
Unit 1
conv 128
conv 256
conv 256
conv 512
conv 512
conv 512
Unit 2
Batchnorm
Batchnorm
Batchnorm
Batchnorm
Batchnorm
Batchnorm
Unit
ReLU
ReLU
ReLU
ReLU
ReLU
ReLU
Unit 4
Dropout
Dropout
Dropout
Dropout
Dropout
Dropout
Unit 5
Maxpool
Maxpool
Maxpool 2
Maxpool
Dropout Rates across Different Datasets
MNIST
F-MNIST
CIFAR-10
CIFAR-100
SVHN
STL-10
Dropout rate
0.1
0.1
0.05
0.05
0.05
0.1
Table IX: Fully connected layers succeeding the VGG8B backbone. Feature maps from the final convolutional layers are transformed into feature vectors. The initial dimension of the "fe" layers represents the feature vectors' dimension. Dropout rates are identical to those used the backbone.
MNIST, F-MNIST
CIFIAR-10 (100), SVHN
STL-10
Layer 7
Layer 8
Layer 7
Layer 8
Layer
Layer 8
Unit
512x
Ie 1024x
Ic 2048x1024
1024x 10 (100)
1024x
Unit 2
Batchnorm
Batchnorm
Batchnorm
Unit 3
Dropout
Dropout
Dropout
J.2.1 FC AND CONV EXPERIMENTS.
For Table Tand 2 our models and their BP baselines are trained using the same training hyperparam- eters. The architecture details are listed in Table VI All FC and Conv experiments employ learning rate decay rate 0.5 and batch size of 512.
J.2.2 VGGSB EXPERIMENTS.
For Table VGG8B architectures come in three variants, each tailored to specific input image size, but they share the backbone layers listed Table Tall Table summarizes the variations in the fully connected heads. For the label embedding dictionary D 2 where Z equals the number of label classes, we use Cp = 512 across all datasets. For the "Layer 7" in Table [X] we segment the output di24-simn vector into 2 segments of 112-dimensional vectors for Ldiet in Eq. (2). batch size of 128 is used for all experiments.
J.2.3 EXPERIMENTS WITH RESNET, VIT, AND MLPMIXER.
ResNet-32 and ResNet-32-W. ResNet-32 is structured with 15 residual blocks, each containing convolutional layers. These blocks are treated as individual modules. ResNet-32-W is structurally identical ResNet-32, with the key distinction being that its channel dimension is four times larger than that of ResNet-32, Training details of ResNet-32 and ResNet-32-W are in Appendix J.2.6
VIT. We employ a vision transformer (ViT) containing 6 transformer encoder modules. Each module includes multi-hea attention (MHA) and multilayer perceptron (MLP), with both MHA and MLP having two fully connected (FC) layers. ViT's configuration involves attention heads, 4 x image patch size, 51 512-dimensional tokens, and single class token.
viii

---

Published as a conference paper at ICLR 2024
In line with the original design of the class token in ViT, we optimize the similarity between label embedding vectors t- and the class token vector. We use the same CIFAR-10 training hyperparameters utilized for VGG8B in Table
MLP-Mixer. In our experiments, MLP-Mixer employs 6 modules, each featuring channel- and a token-mixing layer. With 4x4 image patch size on CIFAR-I0, each local module processes 64 tokens of 512-dimensional vectors. Training hyperparameters for CIFAR-10 are identical those used for VGG8B, as shown in VII
J.2.4 IMPLEMENTING Ccontrast AND Lieat.
Training with Ccontrast We train VGG8B and ResNet-32 models with auxiliary networks for and Appendix/AAJecatuee Lcontrast entails the use of auxiliary networks.
Figure Figure
Table
For VGGSB, we used the same auxiliary network architecture and training hyperparameters used by Nøkland & Eidnes (211)). The model incorporates layer-wise fully connected auxiliary network, transforming A --dimensionnll vectors into 128 22--iimensional ones. A equals 2048 except for MNIST and FNSSI which use A 1024. The local feature maps h undergo adaptive average- pooling, ensuring the flattened local outputs are standardized to 1024-dimensional vectors. Training hyperparameters are available in Appendix [.2.2]
For ResNet-32, we adopt the same auxiliary network architecture as used by [aan et al. (2020), comprising three convolutional layers and two fully connected layers. The training settings detailed in Appendix J.2.6
Training with LL-contrec. For LL-contrec in Section 5.2 we combine Ccontrast with an image reconstruction loss free used in estimating / (hh, .)) (see Appendix J.2.7 for more details). Then, we have Lcontree = feontrust + X2Zeree where A = a,3 and A = a<1 33 (Wang et al. 2020). B
progressively increases as layers deepen, such that / L-1 1 f1. --....1-11,
Training with Liest. For Figures 3 Table Table 2 and we train the models using Lieat, with the training hyperparameters designated for each architecture. We can derive Lieat by using fo(h) = h Eq. [T. For the fully connected layers, we directly use the output vector as For CNN, flattening feature map hmap E Cx xxW leads to an unwieldy vector size. To address this, we employ the average local feature h * Σk ht, where K HW.
J.2.5 REPLICATED FINDINGS.
For benchmarks not available in published papers, reproduce baseline models with their open- source codes.
For FC and Cony architectures. We train FF (Hinton 2022). PFF (Orobiia & Mali 2023). and DRTP (Frenkel et al.) (2021) on CIFAR-100 for Table 1 and Table We apply the same hyperparameters and architectures prescribed for CIFAR-10. To modify the architectures for CIFAR- 100, upscale the output dimension of the final classifier from 10 to 100. This explains the increase in the number of parameters in Table aad Table 2 Likewise, for PFF, which originally takes 28 x 28 grayscale images, we upscale the input dimension to accommodate 32 32 RGB images.
LL- experiments. Because LL--pff the BP-free version of LL.-predsim, we use the same training hyperparameters designated for LL-predsim to train LL-bpf models for Table LL--bpf models utilize two separate layer-wise projections. The first projection maps one-hot encoded labels to 128-dimensional vectors using fixed random weights to prevent BP. The second projection requires adaptive pooling, such that flatten local outputs become (096-dimensional vectors. Then, a fully connected layer projects the 1096-dimensional vectors 228-dimensional vectors. Layers receive error signals from the binary cross entropy loss between the projected di-ddimnnii vectors. The enror signals are propagated by feedhack alignment (Lillierap al. 2016) prevent BP.
ix

---

Published as a conference paper at ICLR 2024
J.2.6 TRAINING SETTINGS FOR RESNET-32 EXPERIMENTS
We train ResNet-32 (He al. 2016 for five separate experiments: Figure [ Figure 3 and 3) Table V The training configurations remain the same across these tasks, based on the specifications from Wang et (2020). Namely, using an SGD optimizer with a Nesterov momentum (Sutskever et al. 2013) of 0.9, the ResNet-32 models are trained for 160 epochs, with the L2 weight decay ratio le-4, batch size of 128, and the cosine annealing with an initial learning rate of 0.8. In all our experiments, we compute local losses based on the output from each residual block. ResNet-32 comprises 16 residual blocks, resulting local losses.
J.2.7 DETAILS ON IMPLEMENTING MUTUAL INFORMATION EXPERIMENTS.
For Figure and Figure I we extract local features from the VGG&B models trained CIFAR-10 estimate I [h, 2) and I y). We follow the same architectures and hyperparameters used by Wang al. (2020) for these experiments. Each estimation experiment employs bilinear interpolation to align the local feature map's heightx width to 32 x 32.
Estimating [(h,y) y). We train ResNet-32 to predict using h as input. We employ the best test accuracy estimate of [h, y). Appendix J.2.6 summarizes the training setup for ResNet-32. Given the varying dimensions of these layer-wise outputs, we adjust the input channel dimension to align with the channel size of the local feature maps.
Estimating T(h, 2.). We train reconstruction networks to reconstruct x using h as input, such that free LBCE(fe (h), æ)), where LBCe is the binary cross-entropy loss, and f is a reconstruction network. We employ 1 LBCE between the output RGB image and æ an estimate of I I(h, æ). fa consists two convolutional layers. The initial layer aligns the channel dimension of the local feature maps to 12, and the subsequent layer transforms these maps into RGB images, ending with sigmoid activation function. The networks are trained for 10 epochs using the default Pytorch Adam optimizer (Kingma & Ba 2015), with learning rate set at 0.001.

---

Published as a conference paper at ICLR 2024
K VISUALIZATION
K.1
LABEL EMBEDDINGS AND CONFUSION RATES
airplane
0.94
0.005
0.015
0.006
0.003
0.001
0.003
0.002
0.025
0.004
automobile
0.005
0.95
0.001
0.004
0.001
0.001
0.001
0.002
0.005
0.025
0.8
bird
0.016
0.89
0.024
0.024
0.019
0.019
0.005
0.004
0.003
0.009
0.023
0.062
0.008
0.006
0.006
0.016
0.066
0.02
0.87
0.008
0.008
0.002
frog
0.005
0.000
0.016
0.018
0.006
0.004
0.95
0.001
horse
0.006
0.001
0.009
0.014
0.018
0.019
0.003
0.93
0.001
0.2
ship
0.023
0.005
0.003
0.005
0.001
0.003
0.001
0.95
0.004
truck
0.004
0.031
0.001
0.002
0.001
0.002
0.012
0.95
sirplane automobile
bird
deer
horse
truck
(b)
airplane
0.013
automobile
0.8
bird
0.013
0.095
0.094
0.086
0.0047
0.13
0.14
0.31
cat
0.21
0.2
0.095
0.099
0.11
0.044
0.2
0.17
0.21
0.6
deer
0.16
0.31
0.094
0.099
1
0.13
0.029
0.11
0.062
0.16
dog
0.25
0.24
0.086
0.11
0.13
0.077
0.0068
0.27
0.19
0.0047
0.029
0.062
0.041
0.2
0.012
0.016
0.14
0.17
0.082
0.27
0.043
0.19
0.023
truck
0.12
0.16
0.31
0.21
0.16
0.19
0.16
0.041
0.023
1
airplane automobile
bird
dog
frog
ship
truck
(a)
Figure VI: Label embedding similarity and confusion matrix of the VGG8B model trained using Ldiet on CIFAR-10. (a) Cosine similarity matrix of label embedding vectors. Each number cell denotes the cosine similarity between label embedding vectors corresponding to each label pair. (b) Confusion Matrix. Labels the x-axis are the ground truth, and labels y-axis are predicted labels. Each cell value signifies the ratio of predicting Byy-
xi

---

Published as a conference paper at ICLR 2024
K.2 SEMANTIC RELATIONSHIP BETWEEN ADAPTIVE LABEL EMBEDDING VECTORS
plain
sea
doud
mountain
castle
chair
house road
wardrobe couch
television bed
table
streetcar
train bridge skyscraper
tank
büs
bottle
pickup_truch crrpppttcckkkkk
lamp
tractor
cup
telephone computer keyboard komputer_keyboard
lawn mower
forest
bowi dock
motorcycle bicycle
willow tree oak tree
plate
rocket
maple thee
pine tre_tree
worm
snake
palm_tred tree
pear
apple
orange
archid rose
dinosaur
lizard
crocodile
turtle
sweet pepper
tulip poppy
aquarium fish
crab caterpillar
lobster
spider
mushroom sunflower trout
butterfly
bee
beetle
shark ray
cockroach
flatfish
frut and vegetables
man boy
whale
people
baby
large carrivores
woman girl
dolphin
shrew
mammals
mouse
Insects
otter
hamster
vehicles
chimpanzee seal
beaver
possum
skunk
bear
rabbit
raccoon
large man made outdoor things
elephant snail
large omniveres and herbivores
porcupine
household electrical device
squirell
large
non insect invertebrates reptiles
cattle
camel
wolf
kangaroo
medium-sized mammals
in tiger
leopard
mammals
vehicles 2
trees
Figure VII: The t-SNE Van der Maaten Hinton 2008) visualization of CIFAR-100 label embedding vectors. CIFAR-100 100 consists of 20 super-classes, each containing 5 sub-classes. The legend lists these slaee-classes.
xii

---

Published as a conference paper at ICLR 2024
K.3
SALIENCY MAPS
Ground Truth
Top 1 Prodiction
Top-2 Prodiction
Ground Trath
Top- 1 Prodiction
Top-2 Prodiction
Bird
Dog
Bird
Frog
Door
Cat
Dog
Cat
Dog
Cal
Dog
Horse
Horse
Aippian
Ship
Aimpane
Figure VIII: Saliency maps for incorrect top-1 prediction results.
Ground Truth
Top-1 Prediction
Top-2 Prediction
Ground Truth
Top-1 Prediction
Top-2 Prediction
Frog
Frog
Frog
Dog
Airplane
Airplane
Bird
Dog
Dog
Cat
Automobile
Automobile
Truck
Bird
Bird
Horse
Dog
Dog
Cat
Ship
Truck
Figure IX: Saliency maps for correct top-1 prediction results.
xiii

---

Published as a conference paper at ICLR 2024
Jup Prediction
Layer 1
Layer 2
Layer
Layer
Top-1 Truck
Ground Truth
Top2 Almiaze
Truck
3op-3 Automobii
Ibp Prediction
Layer
Layer
Layer
Layer 4
Top: Home
Ground Tralh
Tup-2
Tug-3 Truck
Figure X: Evolution of saliency maps across layers. The layer index denotes the source of the extracted local features.
xiv


Reviews:
[{"review_id": "L3c4NAj3zc", "reviewer": "Reviewer_UJZ4", "summary": "This work studies local training, that is training with local error signals so as to eliminate backprop, which is not plausible biologically. The proposed method is dictionary contrastive learning, built upon a prior work that proposed using (supervised) contrastive learning for local learning. The main difference from that prior work is using a embedding dictionary $t_1, \\cdots, t_Z$ for each of the $Z$ classes, and aligning intermediate features with the average pooling of these embeddings (for dimension agreement).", "questions": "Minor points:\n- The authors wrote $K = H \\times W$ in several places, which makes little sense to me. Why does an image need to have as many feature maps as the number of its pixels? And why does it need to have multiple feature maps in the first place?\n- Can you explicitly write out the formula of $pool_l$? For example, what does it do when $C_h^l > C_D$? Does it do zero padding? And can you tell me what is the motivation of pooling, other than adjusting the dimensions?\n\n\n**Summary:** Overall, though I do think that the idea of introducing a dictionary could be interesting, I don't think this work presents this idea very well and makes enough justification. The paper is really confusing at times. Moreover, I feel that even the authors themselves are confused sometimes, when they call their method FL which to my understanding really isn't, and use $K=HW$ features in Eqn. (2) which is equivalent to using a single feature. Thus, I recommend rejecting this submission.\n\n**Rebuttal note:** Score changed from 3 to 6.\n\nMy suggestion to the authors would be to read Wang et al. [1] very carefully, which I did before reviewing this submission. Their motivation was to improve over end-to-end training, and they did not mention removing BP or forward learning at all. That's why it makes sense for their method to use a BP-updated neural network, as long as it is modularized. Also, given the great similarity between this work  and [1], InfoPro should be the most important baseline in your experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. I think the subject matter of this work is very interesting, and I can see why contrastive learning is relevant in this context, though I should point out that neither the problem nor contrastive learning for local learning is first proposed by this submission.\n2. The authors conduct a number of experiments to show that the proposed method is better than previous local learning methods, which is good, though I would say that there is one very important baseline that the authors do not compare to (see below).", "weaknesses": "I am not an expert of local learning or forward learning, though I am very familiar with representation learning and contrastive learning. So I read the most relevant cited papers in this work, and from my understanding, this work is a follow-up work of Wang et al. [1]. And my understanding is that the main difference between this work and [1] is the introduction of $\\lbrace t_1, \\cdots, t_Z \\rbrace$, the dictionary. So while conventional contrastive learning maximizes the similarity between a positive pair of samples' features, this work maximizes the similarity between a sample's feature and its corresponding $t_i$, and $t_i$ is also being updated during training. My following review is mostly based on the above understanding, and the comparison between this work and [1]. Please let me know if there is anything that I misunderstand.\n\nThe following are my major questions and concerns:\n### 1. Regarding Sections 3 and 4\nSections 3 and 4, arguably the most important two sections of this work, are really confusing. \n- Figure 1 compares \"$L_{feat}$\" with \"$L_{contrast}$\", and at first glance seems to suggest that contrastive learning is better than some other method called \"feat\", but this is not the case. Both methods minimize the contrastive loss, and the difference is that \"feat\" uses the intermediate outputs $h$ directly while \"contrast\" fits a network $\\phi$ on top of it. So this figure does not show that contrastive learning is better, or \"the performance of FF-based approaches falls short in comparison to InfoPro\". Instead, it only shows that using a neural network $\\phi$ is important. Thus, I am not quite sure what message the authors want to convey with Figure 1, as well as Figure 3.\n- An important motivation of this work and LL/FL as a whole is the goal of removing backprop (BP), because BP is not biologically plausible and is not memory efficient. The abstract and intro of this submission make this point very clear. My question is: If you use a neural network $\\phi$ in Eqn. (1), then how can you remove BP? Don't you use BP to train this $\\phi$? It might be true that using a smaller network $\\phi$ can save memory, but my point is BP is still there if you are using this $\\phi$, unless you are updating this $\\phi$ with local learning too (which I don't think is the case). Thus, I don't think the proposed method can be called forward learning (FL), which by definition should have no BP at all. And it is not very fair to compare the proposed method with FL methods.\n- In Section 4, \"Mapping labels to embedding vectors\", the authors wrote \"the label embeddings in the dictionary are updated at each step, making them a dynamic concept\". But how are these $t_i$ updated? Are they updated by minimizing the loss $L_{dict}$? If this is the case, then this update is not local, because remember that the same $t_i$ are shared across all layers (which then go through pooling), so the update signals of $t_i$ come from all layers. Consequently, the layers cannot be learned in parallel as FL methods, because all layers need to jointly update $t_i$. I would say that it makes more sense to me if these $t_i$ are fixed, unless there are some imperative reasons that $t_i$ must be updated.\n- In Eqn. (2), why there is an average over $K$? And why does there need to be $K$ feature maps for one class in the first place? Eqn. (2) can be simplified as $L_{dict} = -\\frac{1}{N} \\sum [ \\text{log} \\frac{\\text{exp}( \\langle h_n, t_+' \\rangle )  }{ \\sum \\text{exp}( \\langle h_n, t_i' \\rangle ) } ]$, where $h_n$ is the average of $h_n^k$. So why not just use one $h_n$, but use K $h_n^k$ if they are equivalent?\n\n### 2. Regarding the experiments\n- My biggest concern with the experiments is that this work is built upon InfoPro [1], yet in the experiments it is not compared to InfoPro [1] at all. I feel that it is very likely that the method proposed in this work has a very similar performance to InfoPro.\n- The experiments compare the proposed method with FL methods, which as I said is not very fair, because I don't think the proposed method is FL since it still uses backprop when updating $\\phi$. InfoPro [1] did not claim their method to be FL. In fact, their paper did not mention FL at all.\n\n[1] Wang et al., Revisiting locally supervised learning: an alternative to end-to-end training, ICLR 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iKjOCelSZU", "reviewer": "Reviewer_3KwZ", "summary": "The paper offers a new algorithm for Forward Learning that is based on intermediate layer features being close to adaptive label embeddings. This work shows comparison to various FL techniques, and shows that the proposed technique is beneficial in terms of performance, and memory usage.", "questions": "**Questions**:\n1. It is interesting that the authors chose to have one embedding per class for all layers. One could think that different layers can achieve different levels of clustering and therefore both the embeddings and the loss weight on them can be different (see for example [2]). Another alternative to this would be to learn an MLP from the features to the embeddings, rather than using average pooling. I wonder if the authors tried this direction. \n2. Another question is wether the authors considered having more than one embedding per class, given that there may be variability within each class. - this could then be some hyperparameter: number of embs per class\n \n\n**Possible missing literature**:\n1. I think the paper relates very much to the literature of neural collapse in intermediate layers. It would be great to see the authors mention these works and make the connection. [1-4, 6]\n\n2. There have been previous works that use intermediate layer losses to encourage class label clustering in the BP setting [2, 5].\n\n\n[1] Papyan, V., Han, X. Y., & Donoho, D. L. (2020). Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40), 24652-24663. https://doi.org/10.1073/pnas.2015509117\n\n[2] Ben-Shaul, I. &amp; Dekel, S.. (2022). Nearest Class-Center Simplification through Intermediate Layers. <i>Proceedings of Topological, Algebraic, and Geometric Learning Workshops 2022</i>, in <i>Proceedings of Machine Learning Research</i> 196:37-47 Available from https://proceedings.mlr.press/v196/ben-shaul22a.html.\n\n[3] Ben-Shaul, I., Shwartz-Ziv, R., Galanti, T., Dekel, S., & LeCun, Y.  Reverse Engineering Self-Supervised Learning. In Proceedings of the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023).\n\n[4]  Rangamani, A., Lindegaard, M., Galanti, T. &amp; Poggio, T.A.. (2023). Feature learning in deep classifiers through Intermediate Neural Collapse. <i>Proceedings of the 40th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 202:28729-28745 Available from https://proceedings.mlr.press/v202/rangamani23a.html.\n\n[5] Gamaleldin F. Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. In NeurIPS, 2018.\n\n[6] Galanti, T., Galanti, L., & Ben-Shaul, I. (2023). Comparative Generalization Bounds for Deep Neural Networks. Transactions on Machine Learning Research, (ISSN 2835-8856).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper builds on intuition existing in many previous works (see Questions section for potential missing literature) to create a new FL algorithm. The work is presented well, and the experiments look convincing. Further, the method is simplistic, intuitive, and in my view has great potential in future works. The authors also provided code for reproducibility, which is great.", "weaknesses": "I REALLY enjoyed reading the paper, and I think other than some missing literature and visual comments, the paper is very good. \nOne possible comment is that the architectures used are a not exactly SOTA - I would love to see this done with ResNets, ViT, RegNets and so forth - this could enhance the paper greatly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yr2EOQFaSA", "reviewer": "Reviewer_SF1W", "summary": "This paper proposes a back-propagation free training method for neural networks based on a novel contrastive learning loss. This contrastive learning loss is defined in terms of a dictionary of label embeddings. The main motivation for using this loss is reduction of the mutual information between feature representations and a nuisance term, which model task-irrelevant information present in the inputs. The authors demonstrate theoretically that minimizing the proposed loss is equivalent to maximizing the mutual information of representations and labels. Furthermore, they demonstrate empirically that their algorithm reduces the mutual information of representations and the nuisance.", "questions": "__Q1.__ While the authors show in Appendix B that minimizing $\\mathcal{L}\\_{dict}$ is equivalent to maximizing a lower bound of $I(\\mathbf{h}, y)$, could a similar statement be derived for $I(\\mathbf{h}, r)$ or $I(r, y)$?\n\n__Q2.__ Do the learned label embeddings have any semantic meaning? The authors could verify this through dimensionality reduction on $\\mathbf{t} \\in \\mathbf{D}^{Z}$, in the context of a dataset with a large number of classes (i.e., CIFAR-100).\n\n__Post-Discussion__\n\nThe authors added a visualization of label embeddings, in which  they show that the embeddings learned by their method carry semantic meaning.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors tackle an important direction in the training of neural nets, i.e., back-propagation free training. They do so in a novel way, by proposing a new training objective based on a dictionary of label embeddings. Their contribution is also interesting through the angle of the label embeddings themselves. The resulting algorithm works well in practice, being comparable or even better than existing state-of-the-art, and while being relatively light-weight in comparison with existing approaches.", "weaknesses": "__W1: Presentation issues.__ Below I highlight a few presentation issues,\n\n- The authors mention the mutual information $I$, but never formaly define it. Furthermore, even though the authors given details about how they estimate mutual information, I think an explicit pointer (e.g., Appendix D describes how we estimate the mutual information) in the text could make it the paper easier to read.\n\n- In my view, the result the authors prove in Appendix B is part of the paper's contribution. I think authors should consider moving it to the main paper and properly make a statement about the claim, as it is used throughout the author's analysis.\n\n__W2: Inconsistent notation.__ While in eqn. $i$ and $j$ are used to refer sample indices, in eqn. 2 the authors use $n=1,\\cdots,N$. Index $i$ in eqn. 2 then means the class $i=1,\\cdots,Z$ and elements of the dictionary are referred to as $\\mathbf{t}\\_{z} \\in \\mathbf{D}^{Z}$, which is confusing as well, as $z$ can be easily mistaken by $\\mathbf{z} = f_{\\phi}(\\mathbf{h})$. Furthermore, in page 3, $y \\in \\mathbb{N}$ would imply an infinite amount of classes. Authors should refer to $y \\in \\{1,\\cdots, Z\\}$, as the authors do in table 4. The fact that one has a finite number of classes should be explicit.\n\n__Minor.__ The paper contains some reference errors, especially w.r.t. arxiv references to conference papers. Here is a non-exhaustive list of wrong references,\n\n- Yulin Wang, Zanlin Ni, Shiji Song, Le Yang, and Gao Huang. Revisiting locally supervised learning: an alternative to end-to-end training. arXiv preprint arXiv:2101.10832, 2021. (Published in ICLR 2021)\n- Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. (Published in ICLR 2015)\n\n__Note.__ the minor point __did not influenced my final score__.\n\n__Post-Discussion__\n\nThe authors have corrected the presentation issues and inconsistent notation. As a result I raised my score towards 8: Accept", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)