Removed unnessissary listener removal and re-add

THis made the set to free functuion in the datagram to take tons of CPU.
It is more processor light now.