Fix padding to work when layers are rearranged.