Drop priveleges only after listening on possibly priveleged port