Fix a leak of an address, if the address is scheduled for removal
and the stack is torn down.
Thanks to Peter Bostroem and Jiayang Liu from Google for reporting the
issue.

MFC after: 1 week


git-svn-id: svn+ssh://svn.freebsd.org/base/head@271209 ccf9f872-aa2e-dd11-9fc8-001c23d0bc1f