To Adapt or Not to Adapt? Real-Time Adaptation for Semantic Segmentation
Authors: name: Marc Botet Colomer; name: Pier Luigi Dovesi; name: Theodoros Panagiotakopoulos; name: Joao Frederico Carvalho; name: Linus Härenstam-Nielsen; name: Hossein Azizpour; name: Hedvig Kjellström; name: Daniel Cremers; name: Matteo Poggi
Published: 2023-07-27 17:59:59

Abstract: The goal of Online Domain Adaptation for semantic segmentation is to handle
unforeseeable domain changes that occur during deployment, like sudden weather
events. However, the high computational costs associated with brute-force
adaptation make this paradigm unfeasible for real-world applications. In this
paper we propose HAMLET, a Hardware-Aware Modular Least Expensive Training
framework for real-time domain adaptation. Our approach includes a
hardware-aware back-propagation orchestration agent (HAMT) and a dedicated
domain-shift detector that enables active control over when and how the model
is adapted (LT). Thanks to these advancements, our approach is capable of
performing semantic segmentation while simultaneously adapting at more than
29FPS on a single consumer-grade GPU. Our framework's encouraging accuracy and
speed trade-off is demonstrated on OnDA and SHIFT benchmarks through
experimental results.