removed special-casing of batch_size == 1 in various places, though not all

A quick fix, since this has caused problems before. I didn't add the special-casing, so not sure yet if this will break tests.

Left one special-case intact, since I wasn't sure what it did. See comment on line 1647.