> Is that a problem? That's what mp_type_ValueError is for, no?

Yes, so that's what you should throw, not return error code which will be translated into OSError.

> what to do with tests for parts that are specific for MICROPY_CPYTHON_COMPAT?

It's on by default, and is there to help people build smaller/faster interpreter. When there will be enough such people, they hopefully will come up with testing support for such configuration, hopefully reusing ideas like quoted by @lurch.