Remove stopped listener from server sub thread