Only apply padding when needed