Simplify buffer logic to avoid one memory allocation per buffer