src: make buffer size errors more explicit

Fixes #6490
buffer: convert values to uint, not int

In many cases values expected to be unsigned were converted to a signed
integer.

Also include some small code cleanup.