Fix an assertion: we must have a buffer if we aren't skipping.