Don't reallocate on buffer amount change that's the same

Signed-off-by: Carsten Munk <carsten.munk@jollamobile.com>