Rethinking the shape convention of an MLP
Authors: name: Meng-Hsi Chen, name: Yu-Ang Lee, name: Feng-Ting Liao, name: Da-shan Shiu
Organization: MediaTek Research

Multi-layer perceptrons (MLPs) conventionally follow a narrow-wide-narrow
design where skip connections operate at the input/output dimensions while
processing occurs in expanded hidden spaces. We challenge this convention by
proposing wide-narrow-wide (Hourglass) MLP blocks where skip connections
operate at expanded dimensions while residual computation flows through narrow
bottlenecks. This inversion leverages higher-dimensional spaces for incremental
refinement while maintaining computational efficiency through parameter-matched
designs. Implementing Hourglass MLPs requires an initial projection to lift
input signals to expanded dimensions. We propose that this projection can
remain fixed at random initialization throughout training, enabling efficient
training and inference implementations. We evaluate both architectures on
generative tasks over popular image datasets, characterizing
performance-parameter Pareto frontiers through systematic architectural search.
Results show that Hourglass architectures consistently achieve superior Pareto
frontiers compared to conventional designs. As parameter budgets increase,
optimal Hourglass configurations favor deeper networks with wider skip
connections and narrower bottlenecks-a scaling pattern distinct from
conventional MLPs. Our findings suggest reconsidering skip connection placement
in modern architectures, with potential applications extending to Transformers
and other residual networks.