Explicitly set mode when no string is given

This makes sure we don't depend in nil's frozen status. I really don't
have a good idea how to spec this, since that would require freezing
nil, which we then can't undo for the remainder of the spec run. This
could cause all kinds of other side effects.

We also can't use undefined here, since this is not kernel/ code. MRI
can support this API since it's all in C land, but there's no way to do
this properly with just Ruby.

Fixes #2463