Use an enum for the open-routine return value, as per Evan Huus's suggestion.

Clean up some things we ran across while making those changes.

Change-Id: Ic0d8943d36e6e120d7af0a6148fad98015d1e83e
Reviewed-on: https://code.wireshark.org/review/4581
Reviewed-by: Guy Harris <guy@alum.mit.edu>