In case of fragments, don't consider ports. Bug #846.