Stop zeroing out paddings if not needed

@aduggin