Fix listeners not being removed

Listeners will now be removed from the array when they disconnect rather than making an ever growing user list.