Don't treat int return values as valid wtap_open_return_val values.

They happen to be, at least now, but that's not valid in C++, and it's
probably unwise in any case.

Change-Id: Ifd49920cfaa376e5e7788329ee83db3956a7cdff
Reviewed-on: https://code.wireshark.org/review/4585
Reviewed-by: Guy Harris <guy@alum.mit.edu>