Dictionary Contrastive Learning for Efficient Local Supervision without Auxiliary Networks
Authors: Suhwan Choi,Myeongho Jeon,Yeonjung Hwang,Jeonglyul Oh,Sungjun Lim,Joonseok Lee,Myungjoo Kang
Presentation: ICLR 2024,Spotlight

Abstract: While backpropagation (BP) has achieved widespread success in deep learning, it
faces two prominent challenges: computational inefficiency and biological implausibility.
In response to these challenges, local supervision, encompassing Local
Learning (LL) and Forward Learning (FL), has emerged as a promising research
direction. LL employs module-wise BP to achieve competitive results yet relies on
module-wise auxiliary networks, which increase memory and parameter demands.
Conversely, FL updates layer weights without BP and auxiliary networks but falls
short of BP’s performance. This paper proposes a simple yet effective objective
within a contrastive learning framework for local supervision without auxiliary
networks. Given the insight that the existing contrastive learning framework for
local supervision is susceptible to task-irrelevant information without auxiliary
networks, we present DICTIONARY CONTRASTIVE LEARNING (DCL) that optimizes
the similarity between local features and label embeddings. Our method
using static label embeddings yields substantial performance improvements in the
FL scenario, outperforming state-of-the-art FL approaches. Moreover, our method
using adaptive label embeddings closely approaches the performance achieved by
LL while achieving superior memory and parameter efficiency.