Fix a subtle deadlock

If you try to remove a handler from within a handler, you'll deadlock the udp
layer.
Udp cleanup