Cleanup the ipv4 sorting hack, remove redundant field

Should have the *proper* fix which is indeed an unsigned long compare
(and had been proposed as a fix earlier by Kevin, and yanked by me
and now added back by me)