PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
Differentiable architecture search provided a fast solution in finding effective network architectures, but suffered from large memory and computing overheads in jointly training a super-net and searching for an optimal architecture.In this paper, we present a novel approach, namely Partially-Connected DARTS, by sampling a small part of super-net to reduce the redundancy in exploring the network space, thereby performing a more efficient search without comprising the performance.In particular, we perform operation search in a subset of channels while bypassing the held out part in a shortcut.This strategy may suffer from an undesired inconsistency on selecting the edges of super-net caused by sampling different channels.We solve it by introducing edge normalization, which adds a new set of edge-level hyper-parameters to reduce uncertainty in search.Thanks to the reduced memory cost, PC-DARTS can be trained with a larger batch size and, consequently, enjoy both faster speed and higher training stability.Experiment results demonstrate the effectiveness of the proposed method.Specifically, we achieve an error rate of 2.57% on CIFAR10 within merely 0.1 GPU-days for architecture search, and a state-of-the-art top-1 error rate of 24.2% on ImageNet within 3.8 GPU-days for search.Our code has been made available at https://www.dropbox.com/sh/on9lg3rpx1r6dkf/AABG5mt0sMHjnEJyoRnLEYW4a?dl=0.
Allowing partial channel connection in super-networks to regularize and accelerate differentiable architecture searchAn extension of the neural architecture search method DARTS that addresses its shortcoming of immense memory cost by using a random subset of channels and a method to normalize edges.This paper proposes to improve DARTS in terms of training efficiency, from large memory and computing overheads, and proposes a partially-connected DARTS with partial channel connection and edge normalization.