Rework calculation of padding and size

With this change it should be easier to understand what is done.