Don't hold onto the buffer unless there is buffered content