change return type of MVM_file_set_oshandle_encoding to void

My mistake.