Improve error messages for invalid HDF5 files

This will trigger a NeXusError message window if a file can’t be opened
with the Open File dialog because it is not a valid HDF5 file.