Fix an bug of not initializing channel listener set.