Report WTAP_ERR_UNSUPPORTED correctly.

WTAP_ERR_FILE_UNKNOWN_FORMAT is reported if the file is in a format that
libwiretap doesn't know about (either because it's not a capture file at
all or because it's a capture file in a format it doesn't support).
WTAP_ERR_UNSUPPORTED is for files in a *known* format that are using
features or file format elements (record type, link-layer header type,
etc.) that libwireshark doesn't support.  Fix some copy-and-pasteos
causing WTAP_ERR_UNSUPPORTED to be reported with a message appropriate
for WTAP_ERR_FILE_UNKNOWN_FORMAT.

Change-Id: Ic675ffd501c52838d8944a6c61e1b01041b73098
Reviewed-on: https://code.wireshark.org/review/5799
Reviewed-by: Guy Harris <guy@alum.mit.edu>
(cherry picked from commit d2eddde5bd84ceb39bea77052848930c024d356b)
Reviewed-on: https://code.wireshark.org/review/5801