> Does LC_COLLATE affect both alphanumeric and numeric sorting? Would setting LC_ALL instead make sense?

Yes, it does change both, and kind of.  You probably want this:

# I hate POSIX locale variables
env LANG=C LC_ALL=C LC_COLLATE=C sort blah blah

(...possibly even better is `env -i LC_ALL=C sort` which will discard
the entire current process environment, passing only what you give to
the sort program.)