Only add padding if we need to