Fix wslua_filehandler_open() to work as wiretap open routines should.

They should return 1 on success, -1 on error, and 0 if there was no
error but the file isn't one of the type for the routine.  They must not
return any other value.  If they return -1, they must set *err.

Change-Id: I0f1e1675b4cc8f0214ad67a23de0e4ecb09cabea
Reviewed-on: https://code.wireshark.org/review/4221
Reviewed-by: Guy Harris <guy@alum.mit.edu>