Open files with 'rb' and catch msgpack deserialization errors

Fixes #12086