Don't release buffer when we shoudn't

Fixes #2171