Clean up the "unknown per-packet encapsulation" message.

Report the file name, report the frame number as unsigned (which it is),
and make it look more like other error messages.

Change-Id: Ia5281da1c94d55c745da8a8b20dafdb74dd13f92
Reviewed-on: https://code.wireshark.org/review/5793
Reviewed-by: Guy Harris <guy@alum.mit.edu>