Drop privileges before accepting any packets

This increases security as we don't allow the callback to fire before
dropping privileges.