KRADA: Known-region-aware Domain Alignment for Open-set Domain Adaptation in Semantic Segmentation
Authors: Chenhong Zhou
Feng Liu
Chen Gong
Rongfei Zeng
Tongliang Liu
William Cheung
Bo Han
Venue: Accepted by TMLR ()

Abstract: In semantic segmentation, we aim to train a pixel-level classifier to assign category labels to all pixels in an image, where labeled training images and unlabeled test images are from the same distribution and share the same label set. However, in an open world, the unlabeled test images probably contain unknown categories and have different distributions from the labeled images. Hence, in this paper, we consider a new, more realistic, and more challenging problem setting where the pixel-level classifier has to be trained with labeled images and unlabeled open-world images—we name it open world semantic segmentation (OSS). In OSS, the trained classifier is expected to identify unknown-class pixels and classify known-class pixels well. To solve OSS, we first investigate which distribution that unknown-class pixels obey. Then, motivated by the goodness-of-fit test, we use statistical measurements to show how a pixel fits the distribution of an unknown class and select highly-fitted pixels to form the unknown region in each test image. Eventually, we propose an end-to-end learning framework, known-region-aware domain alignment (KRADA), to distinguish unknown classes while aligning the distributions of known classes in labeled and unlabeled open-world images. The effectiveness of KRADA has been verified on two synthetic tasks and one COVID-19 segmentation task.

Published in Transactions on Machine Learning Research (02/2023)
KRADA: Known-region-aware Domain Alignment for Open- set Domain Adaptation in Semantic Segmentation
Chenhong Zhou"
204.87795@lkfehhhbbe.ek
Deportment of Computer Science, Hong Kong Baptist University
Feng Liu'
feng.liul @undmellb.co edw.lbbeedd..o..
The University of Melbourne
Chen Gong
chen.gongEnjust.ede edu.cc
The Key Taboratory of Intelligent Perception and Systems for Digh-DDimessonal Information of Ministry of Educotion, School of Computer Science and Engineering, Nanjing University of Science and Technology
Rongfei Zeng Northeastern University
nengrf@eer.neuu.cdu.cc edu.cc
Tongliang Liu
tongliang, liu @sudney.......m edu
Sydney Al Centre, The University of Sydney
William K. Cheung
williamm&coopp.bieeee edu.
Department of Computer Science, Hong Kong Baptist University
Bo Han Department of Computer Science, Hong Kong Baptist University
bhanem@@@comp.hhtoll edu.
Reviewed on OpenReview: https: // openreview. net/ pot/forum? id= SII12ypVVo
Abstract
In semantic segmentation, we aim to train a pixel-level classifier to assign category labels to all pixels in an image, where labeled training images and unlabeled test images are from the same distribution and share the same label set. However, in an open world, the unlabeled test images probably contain unknown calegories and have differrnnt distribulions from the labeled images. Hence, in this paper, consider new, more realistic, and more challenging problem setting where the pixel-level classifier has to be trained with labeled images and unlabeled open-world images- we name it open-set domain adaptation segmentation (OSDAS). In OSDAS, the trained classifier is expected to identify unknown-class pixels and classify known- class pixels well. To solve OSDAS, we first investigate which distribution that unknown-class pixels obey. Then, motivated by the goodness-offfr test, we use stotistical measurements to show how a pixel fits the distribution of unknown class and select. highly-fitted pixels to form the unknown region in cach test image. Eventually, propose end-to-end lcarning framework, known--egion--uwnn domain aligrement (KRADA), to distinguish unknown classes while aligning the distributions of known classes in labeled and unlabeled open-world images. The effectivencss of KRADA has been verified on two synthetic tasks and one COVID-19 segmentation task.
"CHZ and FL. contributed equally to this paper.
EBH the corresponding author.
Our code is available https://ggtthub.comm cam www.w//oo/oogg/aaaeeeeeeemo caaigg--zhuuaaaa /RADA

---

Published in Transactions on Machine Learning Research (02/2023)
(b)
Figure 1: Illustration of the differences in two CT datasets. (a) Examples of normal CT scans. (b) Examples of COVID-19 CT scans, where the infected area is circled by red boxes. These two datascts vary at the visual level (domain shift) and are not consistent at the semantic level (calegory shift).
1 Introduction
Semantic segmentation aims to assign one category label to each pixel in an image, which has a large variety of applications from autonomous driving (Cordts et al., 2016; Siam et al., 2018), indoor navigation (Silberman et al., 2012) to medical image analysis (Tajbakhsh et al., 2020). In recent years, deep lcarning- -basn methods have been developing rapidly and achieved remarkable successes in semantic segmentation (Garcia-Garoia et al., 2017; Roth et al., 2022). These methods usually train a deep convolutional neural network (DCNN) using a training set that contains pairs of images and pixel-level labels (Long et al., 2015) to segment unlabeled test images. These works commonly assume that the training images and test images are taken from the same scenario and share the same calegory label set (Panareda Busto & Gall, 2017; Saito et al., 2018b).
However, in open world, test images might be taken from a different scenario and have additional category labels compared to training images. For example, in autonomous driving, to reduce the great demand for accurately annotated images, a synthetic dataset, such as SYNTHIA (Ros et al., 2016), is commonly used to train a network for the segmentation of urban scenes. Unfortunately, a realistic urban scenario is quite complex and different from the simulated one. Thus, roal-world urban images probably contain some additional category labels (i.e., unknown classes) that are not present in the synthetic images.
Another representative example is Coronavirus Dísease 2019 (COVID-19) infection segmentation task. Duc to scarce annotated images in COVID-19 dalasets, existing chest Compuled Tomography (CT) scans are expected to be utilized as training images to assist COVID-19 segmentation. However, the segmentation modols trained on a normal CT dataset usually show poor performance to segment the COVID-19 infected area, as a result of domain shift and calegory shift issues. Domain shift refers to a distributional discrepancy caused by the variations in light, conditions, and device types for the acquisition of training and test images, while category shift means the inconsistent label sets between training and test images, e.g., COVID-19, a new discase absent in training images. Figure 1 illustrates both issues in the COVID-19 segmentation task.
Regarding such realistic and challenging segmentation scenario, we name it open-set domain adaptation segmentation (OSDAS). Although closed-set domain adaptation segmentation (CSDAS) methods (Hoffman et al., 2016; Vu et al., 2019; Tsai et al., 2018; Luo et al., 2019; Wang et al., 2020; Saito et al., 2018a; Mei et al., 2020; Zou et al., 2019; Zhang et al., 2019; Kang et al., 2020) have been extensively studied to overcome the domain-shift issuc, they are not applicable to OSDAS because they probably mistakenly align unknown larget data (i.e., open-world images) with source data (i.e., training images), leading negative transfer (Bucci et al., 2020).
In this paper, to solve OSDAS, we first explore the inherent property of unknown classes and propose that the probability distribution of given input belonging to the unknown class outputted by a known-class classifier would conform to a prior probability distribution of the known classes. Motivated by the goodness-of-fe test, we propose to use statistical measurements to describe how likely a target pixel an unknown pixel, i.c., how well the output probability distribution of target pixel fits the distribution of an unknown class. The distribution disparity between these two distributions be measured by statistical measurements. Here we
2

---

Published in Transactions on Machine Learning Research (02/2023)
Training Stage
Inference Stage
Images
Pixel-level Labels
Unseen in training
Pixel-level outputs
Ps Lake
The property of anknown ploels
A test image
M
P
impul
M
output
P2
Figure 2: An illustration of the output distribution property of an unknown pixel. A test image contains a "lake" region whose semantic label (lake) is not present during training. The well-trained segmentation model M cannot recognize such "lake" pixel (Pi) and outputs a probability vector over known classes for it. Its probability distribution conforms to known-class prior probability distribution, different from that of known pixel (P2).
adopt two statistical metrics: 1) Ly-norm (Ahmad & Cerrito, 1993); 2) Kullback Leibler (KL) divergence (Song, 2002), which are general criteria for testing goodness of fit.
Based on these statistical measurements, we can identify the highly-fitted pixels as "unknown", while the unknown region in a target image can be determined. Hence, a segmentation model can be trained using source data and pseudo-labeled target data achieve better domain alignment by rejecting unknown target regions and aligning the distributions only for known-class data. We call this framework knoum--regonn-oven domain alignment (KRADA), which is independent of the network architecture and can be easily realized existing CSDAS methods to adapt them for OSDAS tasks.
We have realized KRADA on three CSDAS methods in our experiments and evaluated them on two synthetic- to-real street scene segmentation tasks and one COVID-19 segmentation lask. Experimental results show that KRADA enables CSDAS methods to identify unknown-class regions and achieve a better overall adaptation, verifying its effectiveness and good generalization ability.
2 Problem Setup: Open-set domain adaptation segmentation (OSDAS)
We address the problem of open-set domain adaptation segmentation (OSDAS) that is defined follows. Problem 1 (Open-set domain adaptation segmentation). Suppose that set of source images with anmotations are denoted as (x5, YS} where the source label space is I_HXW and L = (l....,, K} is the calegory label sel with K known classes. The target images X" are drawn from a different distribution and the target label set has an additional label + to denote the unseen classes that do not appear in Lt. We aim to train a segmentation model M to accurately classify each pizel in targel images X indo one class of the label set (1,..., K, K KK+1 1}.
Different from the open world setting in (Bendale & Boult, 2015; Joseph et al., 2021) which entirely concentrates on unknown classes, proposed setting further considers common phenomenon in known classes that a distributional discrepancy exists, apart from detecting unknown classes. Moreover, some existing studies about open set segmentation (Hwang et al., 2021; Cui et al., 2020; Cen et al., 2021) also do not consider the distribution shift problem. Instead, we expect the segmentation model in OSDAS to solve both domain shift and category shift issues simultaneously.
3 How Do We Determine Unknown Pixels?
To solve OSDAS, the critical issue is how separate unknown pixels from known pixels in target images. Due to lack of unknown-class supervision, classifier trained with source data will forcibly label an unknown
3

---

Published in Transactions on Machine Learning Research (02/2023)
pixel as one of the known (source) classes (Boult et al., 2019). To avoid this issue, consider introducing unknown-class pseudo-labels into a segmentation model so that the model can be trained to learn unknown- class information from pseudo-labels and gain the ability to identify unknown pixels. Before the generation of unknown-class pscudo-labels, we need to know what property an unknown class has. In other words, what can we use to represent an unknown class? If given this properly, we can exploit it to determine whether a pixel is unknown.
As unknown classes have never appeared in the training data and there is only prior knowledge of source classes, well-trained model hardly recognizes unknown pixel and tends to output prediction probability distribution conforming to the class prior probability of source classes (Figure 2). Hence, the distribution for an unknown pixel also conforms to such a known-class prior probability distribution. Those target pixels with this property would be considered unknown. In Figure 2, the output of a "lake" pixel (P1) in a test image conforms to known-class prior probability distribution, as this pixel is an unknown pixel for M.
4 A General Framework to Solve OSDAS
Generally, segmentation network M consists of a feature extractor F and pixel-level classifier C' to encode input images into the feature space and map the features to the label space, respectively. The network M can be formalized as M = Co F and it is usually trained with labeled source data:
L = E)((C(((())),
(1)
where [|] denotes the expectation over m.r.v.s, [.] is the empirical estimation of E|,, and E (---) is a cross-entropy (CE) loss function. In OSDAS, an open-set pixel-level classifier C' is expected to assign one category label from the label set {I..... K, K + 1} to cach pixel in target images. However, the known-class supervision Lggg is not sufficient to make M both identify unknown target pixels and classify known-class pixels well. Thus, we consider adding auxiliary supervision Lyeg by introducing unknown-clas pseudo-labela into the training stage:
Lseg- Lgeg + aaliggg
(2)
where
LT ((((C((((T)) seg
(3)
and 8T denotes the unknown-class pseudo-labels of larget data, and a is a hyperparameter to control the loss weight of unknown classes. Consequently, the open-set classifier C can be trained using the labeled source and pseudo-labeled target data. We introduce the generation of unknown-class pseudo-labels as follows.
4.1 Unknown-class Pseudo-label Generation
Based on the distribution property of unknown classes, we consider determining whether target pixel is unknown from a statistical point of view. Specifically, goodness-of-ff tests indicate the goodness of fit of a model by comparing the observed data with the data expected under the model (Kéry & Royle, 2015; Vasicek, 1976). Motivated by this, we utilize statistical measurements to mcasure how well the output probability distribution for target pixel fits a known-class prior distribution. Then, those highly-fitted target pixels would be labeled as unknown pixels.
Statistical measurements. A straigh way to determine unknown-class pseudo-labels is to compare the maximum softmax probability (MSP) with a threshold (Luo et al., 2020). If a pixel whose maximum softmax probability is smaller than predefined threshold, this pixel will be considered unknown and assigned to K ++
In this paper, we adopt Ta norm and Kullback Leibler (KT.) divergence to measure the distribution disparity between the output probability distribution of target pixel and known-class prior distribution. L2 norm, or Euclidean norm, is commonly used to evaluate the discrepancy between two distributions:
1/2
(x((z), = [l(()) -(x))12--2 =
Σ lp(x) q(x))) LEX
(4)
4

---

Published in Transactions on Machine Learning Research (02/2023)
In addition, KL divergence is non-symmetric mcasure that quantifies how much one distribution differs from another one, criterion used in goodness-of-fit tests (Song, 2002) to indicate the information lost when g(z) is used to approximate p(r):
((x) ((x))
p(z) log DKL(v()))l(())) Σp(z) ZEX
(5)
Remark 1. Typically, p(z) represents a "true" distribution or a theoretical distribution while q(z) represents the observed distribution. Here p(z) is a known-class prior distribution, and q(x) is the prediction probubility outputted by a model for a target pizel. Other statistical metrics can be chosen, and more avoiloble metrics are offered in (Gibbs & Su, 2002).
Known-class classifier and mark unknown-class pseudo-labels. To obtain the prediction probability over known classes (i.e., ((z) in Eq. (4) and Eq. (5)), we additionally introduce pixel-level known-class classifier C to output an image defined on the label space. C can be supervised by minimizing the segmentation loss on the source data:
Leg seg (F((C(((XX))) Y))
(6)
For a target image x² E xT, C* aims to produce probability map p of shape K x H x W:
I softmax(C* (F(x)))
T.c = 1. More specifically, Pij E R implics the probability vector for pixel [i,j} in xT, and thus Eeer. Pij Here parior E RK is used to denote a prior probability of known classes, which is a distribution of classes among all of the source data. Hence, we can calculate the distributional divergence between " and parior for each target pixel and compare with a threshold 5. Those target pixels whose distribution is close to known-class prior distribution are marked as unknown, which can be formalized as
T,K+1 -ft [L, 0, otherwise, Vij
if Le(perior, Pij 8,
(8)
where L2 norm can be replaced with KI. divergence or other probability metrics in (Gibbs Su, 2002).
Remark 2. Note that i" is initially a (K +1 H W shaped tensor with all elements equaling zero. of E gK+1 is a one-hot vector for a target pirel. If 0:3 TT equals one, it means that this pixel will be pseudo-labeled as unknown; otherwise, this pizel remains undabeled.
Adaptive threshold. We adaptive and smooth by considering the past historical information, denoted as Eq. (9). B is momentum factor to retain the past threshold informat ion. T y) indicates the update information from the current. source image 2, E S at time t, as shown in Eq. (10). | indicates the number of pixels in the source image 24 5 and y is a predefined proportion to represent the tolcrable ratio of pixels in x2 wrongly recognizod into the unknown class. As shown Eq. (11), we sort the L2 norm distance between persor and p." S in a descending order, where pr," S.+ softmax(C" (F (z). We acquired the threshold information from source images under the given tolerable proportion to adaptively adjust the threshold.
exponential moving average method to define the threshold 8 to make it
6 = 36.11 + +(1 (1- $ ,,))
(9)
1(x 8 7) = -P,,3 [,|0, $11
(10)
P2 = sort upiio 8: descending).
(11)
Eventually, we can produce pseudo-labels T for target images X". Particularly, pseudo-labels are constantly updated when target images are fed into the network once again. As a consequence, the network can be optimized using source data and previously pscudo-labeled target data while generating new psoudo-labels, which is different from the multi-round training mechanism in (Mei al., 2020; Zou al., 2019; 2018) that alternatively optimizes pseudo-label generation and network training.
5

---

Published in Transactions on Machine Learning Research (02/2023)
lsse
X
ps
ys
C
pr
Seg
9
Feature
Discriminator
Ladv
extractor
P
p7,
P
R
ps
ys
XT
Lseg
Figure 3: Overview of the proposed KRADA realized the existing CSDAS methods. It consists of a feature extractor (F), a open-set pixel-level classifier (C'), a known-class classifier (C*), and discriminator (D). The orange and green parts denote the source domain flow and target domain flow, respectively. Specifically, C is optimized under the supervision from both the source domain and pseudo-labeled target domain, where - denotes the pseudo-labels of target data previously produced at last time (t - 1). The aim of C* is to generate the pseudo-labels 8,7 at time (t) and known-class region Rhn, Then forward the features of source images and known regions in target images into D to perform known-class-aware domain alignment.
Remark 3. It should be emphasized that here L2 norm or KL divergence plays a different role from entropy since entropy measures the prediction uncertainty and only depicts the internal relationships of known classes for a target pizel. It does not exploit the distribution property of the unknoun class, so it does not measure the distance of a pizel to on unknown class in strict sense. This is conformed by the fact that those pivels with high entropy probably boundary pizels, not unknown pizels.
4.2 Known-region-awar Domain Alignment (KRADA)
Since there is a domain shift between source and target domains, we also consider reducing this domain gap when minimizing Eq. (2). Adversarial training (AT) based methods are a predominant stream of minimizing the domain gap; e.g., existing CSDAS works. These methods train M to learn domain-invariant features by confusing a domain discriminator D. An adversarial loss is minimized to align the distributions between source and target domains at input level (Hoffman et al., 2018; Gong et al., 2019), feature level (Hoffman et al., 2016; Vu et al., 2019; Chen et al., 2019; Wang et al., 2020; Hoffman et al., 2018), output level (Luo et al., 2019; Sailo et al., 2018a; Tsai el al., 2018). The main difference between these three kinds of domain alignment is the imput of the discriminator D. For a clear illustration, we take the feature-level aligmment as an example in the following. In this case, the inputs of the discriminator D are the source features and target features produced by the feature extractor F, and the adversarial loss is formulated as:
I AT
adv = - [[g(((((XX))]] È [log(1 - D(F(XT)))
(12)
Unfortunately, these CSDAS works cannot be directly applied to OSDAS since they would forcefully match the feature distributions of two domains, which makes unknown target data mistakenly aligned with source data, leading to negative transfer.
To avoid this issuc, we consider that cross-domain adaptation should be performed only on the known-class data. Therefore, a novel knoun-region-o aware domain aligument (KRADA) is proposed align the target image regions predicted as known with the source images, which is shown in Figure 3. The adversarial loss
6

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm 1 An implementation of KRADA existing CSDAS methods.
Input: source data (Xx, )²), larget data x", initial pseudo-labels Yg"
Parameter: network parameters: Or, fc. 0n, fc-, the number of iteration N, learning rate: 14, psendo-label loss weight: a, initial threshold: 6, tolerable proportion: " momentum factor:
Output: predicted target labels: Y
1: for - to N do
2: calculate Laeg using (X Y X ,,_,) according to Eq. (1), (2), and (3).
3: calculate Lieg using (X yd) according to Eq. (6).
4: generate Y Hthh using x" according to Eq. (7), (8), and (14).
calculate Ludo using (X _x=, _ according to Eq. (13).
6: OF 0p - Vep (Locy + Ling - Lade)
l Ic - 70c Leeg
8:
OD = 0D 1 Hoo Ladv
c. cc μ V76- Lieg
10:
update â according to Eq. (9), (10), and (11).
11: end
12: Prediction: Y c- C(F(X"))
can be developed from Eq. (12) to:
Ladu [lgg(D((XX)))) - [log(1 D((((*) R²"))]]
(13)
where Rhn binary mask denote the known-class region predicted for target images. Once pseudo-labels T.T are determined, Rkn can also be obtained by:
0,
if Dij TT =
Rkk
(14)
1, otherwise.
We multiply R and each channel of target feature maps and then forward the features of known-class regions in target images into D. Eventually, unknown-class regions rejected, and only the known regions of larget images are aligned with source images. It is worth mentioning that KRADA does not have any requirement for the discriminator, so the alignment process of KRADA has no difference from that of original AT methods, except for the target inputs of the discriminator. This indicates that KRADA is independent of the model or structure and has good extensibility.
5 Realizations of KRADA on Existing CSDAS Methods
this section, we realize the proposed KRADA on existing CSDAS methods, as illustrated in Figure 3. The original CSDAS network generally consists of a feature extractor (F), a pixel-level classifier (C), and a discriminator (D). To solve OSDAS, we additionally introduce known-class classifier (C*) which is arranged in parallel with C. The architecture of C* is similar to that of C' except for the last convolutional layer with K output channels. The training details are summarized in Algorithm 1. We use source data (X yS) and target data X with previously generated pseudo-labels 8,r, to calculate Laeg according to Eqs. (1), (2), and (3) (line 2), where the initial pseudo-labels YiT is tensor of (K+ 1) H x W with all elements being zero.
Then source data to calculate Lieg according to Eq. (6) (line 3). After generating the new pseudo-labed 8-T and the known-region map Rhn according to Eqs. (7), (8), and (14) (line 4), we calculate Ludo according to Eq. (13) (line 5). These network parameters and the threshold are updated until the network converges (lines 6-10). Finally, we obtain the segmentation results y of larget data (line 12). Overall, KRADA has no specific requirements for CSDAS architectures (F, C, and D) and can be easily integrated into CSDAS methods form unified segmentation models. Therefore, a series of CSDAS methods can be adapted solve OSDAS by minor modification. In our experiments, we realize KRADA using three representative CSDAS methods, and the results show that KRADA can help address the OSDAS problem well.
7

---

Published in Transactions on Machine Learning Research (02/2023)
Table 1: Results on SYNTHIA + Cityscapes. "B" denotes the best score during training, while "L" denotes the last score at the end of training. Source-only refers a base model only trained on source images without adaptation.
ruod
side.
build.
leace
pole
3ign
rider
motor
bike
unk.
Method
mioU
mioU
B
39.9
19.2
65.4
0.0
22.4
2.0
63.8
70.2
47.8
14.0
47.9
7.5
26.2
0.0
30.4
32.8
Soure-only
L
31.5
18.0
61.7
0.0
20.9
2.0
60.1
67.8
48.0
15.7
46.5
8.4
26.2
0.0
29.3
31.5
OSBP (Saito et al., 2018b)
31.7
17.4
67.5
0.0
20.3
63.0
71.0
33.3
11.7
60.8
7.7
26.2
1.7
29.5
31.6
L
28.7
16.3
66.6
19.5
0.6
59.0
72.7
28.9
7.0
50.2
3.2
23.9
2.0
27.0
29.0
AdaptSegNet (Tsai et al., 2018)
72.4
35.7
75.6
69.9
37.7
70.6
12.2
29.9
36.4
39.2
T. 66.6
35.9
74.0
0.0
14.2
1.1
68.4
72.8
35.2
13.6
62.4
10.4
27.8
0.0
34.5
37.1
CLAN (Luo et al., 2019)
B 83.4
37.6
76.8
0.0
21.
2.7
77.8
78.7
49.9
17.7
80.1
12.6
28.6
0.0
40.6
43.7
38.1
47.3
78.5
10.7
39.7
42.8
FADA (Wang al., 2020)
B 84.5
39.0
79.0
0.0
2
1.4
83.0
73.6
38.3
13.3
75.6
5.1
38.6
0.0
39.9
43.0
82.6
37.5
79.0
0.0
25.8
1.8
82.9
74.8
37.7
12.9
76.3
6.6
35.0
0.0
39.5
42.5
Not + KRADA
B 74.1
31.5
76.5
0.0
16.3
0.7
75.1
75.9
47.9
16.0
74.4
9.8
34.6
6.1
38.5
41.0
30.8
75.9
17.2
0.7
74.9
75.3
46.7
16.7
73.0
9.6
36.9
37.8
40.3
CLAN + KRADA
B 82.4
37.3
76.4
0.0
22.1
2.5
76.6
77.8
49.9
18.5
72.4
15.3
26.9
5.3
40.4
43.1
L
80.2
38.3
76.9
0.0
20.3
2.5
76.8
78.8
51.0
17.7
71.6
112
27.9
4.8
40.1
42.8
FADA + KRADA
85.3
41.6
80.3
0.0
28.9
0.9
82.7
78.6
42.6
12.8
80.1
8.5
42.7
6.5
42.2
45.0
L
81.6
40.0
79.9
0.0
27.6
1.5
82.4
77.1
42.6
12.7
78.5
8.5
40.5
6.4
41.6
44.3
6 Experiments
Synthetic OSDAS tasks. Since the rescarch about OSDAS has not been explored, there no public datasel for such a new selting. Based on two synthetic-lo-real benchmark tasks in CSDAS: SYNTHIA (Ros et al., 2016) + Cityscapes (Cordts et al., 2016) and GTA5 (Richter et al., 2016) + Cityscapes, adjust these two tasks to simulate the OSDAS scenario. For the task SYNTHIA classes (wall, light, and bus) to form the unknown class and discard those images containing either of the three classes. The remaining images in SYNTHIA are regarded as the source domain. For the task GTA5 + Cityscapes, we choose two classes (fence and sign) to form the unknown class. Those images which do not contain the unknown class in GTA5 are retained as the source domain. Following the common practice in CSDAS, we use the Cityscapes training set as the target domain and evaluate our models on the Cityscapes validation set with widely adopted evaluation metric: mcan Interscction over Union (mIoU) for all classos. Considering the evaluation protocol in unsupervised open set domain adaptation (UOSDA) (Panareda Busto & Gall, 2017: Saito et al., 2018b), we also report the mloU averaged over known classes only, denoted as mloU" in this paper. To justify the generalization ability of KRADA, we implement KRADA on three CSDAS methods: 1) AdaptSogNec (Tsai et al., 2018), 2) CLAN (Luo et al., 2019), and 3) a statc-of-tho-art adversarial training-bast based method FADA (Wang el al., 2020), denoted as AdaptSegNee + KRADA, CLAN + KRADA, and FADA + KRADA, respectively. Additionally, OSBP (Saito al., 2018b) is used as the baseline, which is of the few UOSDA methods that can be modified to segmentation tasks by directly convolutionalizing its network architecture. More data descriptions and implementation details are described in Appendix B.
Cityscapes, we select three
Results on synthetic OSDAS tasks. We present the results of SYNTHIA + Cityscapes in Table 1. Compared with Source-only, OSBP increases unknown-class loU from 0% to around 2%. But this achievement. is at the cost of the segmentation performance on known classes as both mloU and mIoU" of OSBP significantly decrease. Compared to Source-only, AdaptSegNet, CLAN, and FADA promote the adaptation for known classes by a large margin, but their unknown-class loU values are stable at 0%, which means that they cannot recognize the unknown class. After the realizations of KRADA, these three modified models consistently outperform OSBP with huge margins and achieve significant improvements in the segmentation of unknown regions. Compared to their corresponding original versions, these three models equipped with KRADA significantly improve the unknown-class loU from 0% to around 5% meanwhile obtaining higher mIoU and mloU* at the last epoch. This because KRADA mitigates negative transfer caused by unknown classes in the target domain wrongly matched to known classes the source domain. Thus, the segmentation of known classes also improves. Particularly, FADA + KRADA achieves the best performance at both best and last epochs.
8

---

Published in Transactions on Machine Learning Research (02/2023)
Table 2: Results on GTA5 + Cityscapes. "B" denotes the best score during training, while "L" denotes the last score the end of training.
road
side.
build
wall
pole
light
porson
bus
motor
bike
unk.
Mothod
mTnU
mTot
Sourc-only
80.0
6.8
74.4
16.2
25.6
27.9
77.2
15.7
72.3
52
19.5
70.5
16.2
17.1
0.9
11.0
0.3
0.0
32.5
34.4
77.6
4.3
70.9
14.9
22.0
26.5
75.6
10.2
71.3
51.8
17.0
69.6
14.9
16.3
0.1
12.4
0.3
0.0
30.9
32.7
OSBP (Saito al., 2018b)
B
84.9
36.8
77.9
18.4
24.1
23.0
80.6
26.3
74.5
51.3
13.2
74.8
20.1
23.0
0.0
16.4
0.0
0.5
35.9
38.0
85.7
35.7
78.0
15.6
23.8
20.6
79.6
72.6
19.1
16.6
0.0
35.0
AdaptSegNet (Tsai et al., 2018)
B 78.3
14.4
77.9
111
25.6
31.3
20.2
18.3
82.9
56.0
25.0
76.5
14.7
3.8
0.5
27.9
0.9
0.0
35
37.2
71.2
73.9
33.9
CLAN (Luo al., 2019)
87.8
17.3
76.8
22.4
23.0
26.6
82.0
30.0
80.0
54.8
16.6
83.6
35.7
43.2
0.0
26.7
0.2
0.0
39.3
41.6
87.2
16.7
76.0
19.9
21.7
27.3
82.5
28.4
79.2
55.0
9.3
83.1
32.2
38.0
0.0
26.8
0.2
0.0
38.0
40.2
FADA (Wang et al., 2020)
83.8
37.5
34.4
47.7
45.2
L 91.5
44.5
83.9
36.5
31.2
28.1
86.
41.0
87.1
62.0
33.
86.3
29.2
40.3
0.0
19.8
2.0
0.0
44.6
47.2
AdaptSegNet
70.5
25.8
35.9
38.0
KRADA
L 71.9
25.5
79.4
21.1
23.7
26.7
82
31.6
78.8
53.9
18.8
79.9
27.7
35.7
1.2
16.6
0.0
0.4
37.5
39.7
CLAN +
85.8
19.3
79.3
20.1
25.0
28.0
83
35.2
19.8
54.6
21.3
82.
32.1
37
0.0
23.7
0.2
0.9
39.4
11.7
KRADA
87.2
17.3
78.4
19.7
25.0
27.6
35.2
79.4
55.7
20.5
82.3
30.2
58
24.2
0.2
0.6
39.1
41.4
FADA +
B 91.9
42.9
84.2
35.6
31.8
32.0
86.3
36.8
87.3
61.9
33.6
86.1
34.5
43.9
0.0
34.6
2.7
0.9
45.9
48.8
KRADA
91.7
43.6
84.4
36.0
32.0
33.8
31.0
42.0
34.3
45.7
Target Image
Ground Truth
OSBP
FADA
FADA KRADA
Figure Qualitative results on SYNTHIA Cityscapes. From left to right, each row target image, ground truth, and segmentation results of OSBP, FADA, and FADA + KRADA, where the white area denotes the unknown-class region.
Table presents the results of GTA5 + Cityscapes. We can see that KRADA enables three CSDAS models to consistently outperform OSBP and improve the unknown-class IoU from 0% to nearly 1%, meanwhile obtaining margin gains in mloU and mloU* at the both last and best epochs. Overall, the results shown in Table and Table 2 demonstrate that KRADA can enable CSDAS methods to identify unknown-class regions and promote a better overall adaptation by rejecting unknown-class regions and conducting the known-region-aware domain alignment. We also provide some qualitative segmentation results in Figure 4.
We tend select the minority classes to construct the unknown class in the above experiments in order to retain as many source images as possible. Even though KRADA makes three CSDAS methods capable of detecting unknown regions, the unknown-class IoU values are not very large, especially for the GTA5 Cityscape task in Table 2. This is because the unknown class is challenging class to segment due to the imbalance issue. To explore the effect of different compositions of the unknown class, we conduct an extra ablation study. We choose car and light to form & new unknown class in GTA5 + Cityscape where the car is a majority class. The results are shown in Table 3. Compared to Table 2, there is a sharp overall decline in both mloU and mIoU* since an casy class (car) excluded from known classes. Comparatively, KRADA drastically increases the unknown-class IoU from 0% more than 22% under the CLAN architecture and also improves mIoU and mloU* by large margin.
Ablation studies: 1)
9

---

Published in Transactions on Machine Learning Research (02/2023)
Table 3: Ablation study of varying the unknown class on GTA5 + Cityscapes.
build.
pale
light
Veg.
sky
persou
rider
truck
bus
train
motor
bike
Method
mioU
m
CLAN (Luo et al., 2019)
B 58.1
18.4
61.0
19.2
17.8
19.3
11.0
818
29.5
78.5
620
7.2
20.1
8.0
0.0
18.6
5.0
0.0
28.1
29.7
59.9
17.9
16.2
82.2
29.
79.4
51.3
19.2
3.3
28.3
CLAN KRADA
B 61.1
12.9
71.3
19.9
19.2
16.9
10.6
812
31.0
77.1
50.3
7.0
21.4
10.9
0.0
20.3
8.1
26.3
30.4
30.7
37.2
11.0
73.6
18.0
18.3
15.3
8.9
81.0
27.5
77.1
49.3
6.0
18.6
8.3
0.0
14.8
4.1
22.3
28.4
28.8
Table 4: Ablation study of the effects of the known-region map on SYNTHIA Cityscapes. "w/o" indicates without the known-region map R
D10001
r00d
build
pole
sky
rider
car
Liker
unk.
Method
mlou
mloU
CLAN + KRADA w/o R
B
78.8
37.0
76.8
0.0
22.7
2.1
76.5
78.5
50.6
17.0
74.2
13.5
31.
4.5
40.3
43.0
72.8
37.4
76.3
0.0
21.7
2.2
79.2
50.7
16.3
72.7
13.4
29.2
3.9
39.4
42.1
CLAN KRADA
82.4
37.3
76.4
0.0
22.1
2.5
76.6
77.8
49.9
18.5
72.4
15.3
28.9
5.3
40.4
43.1
L. 80.2
38.3
76.9
0.0
20.3
2.5
76.8
78.8
51.0
17.7
71.6
14.2
27.9
4.8
40.1
42.8
Table 5: Results of CLAN + KRADA with different statistical metrics on SYNTHIA + Cityscapes.
road
side.
build.
Dole
sign
Derson
rider
GAT
motor
bike
unk.
Metrics
mloU
mloU"
81.9
37.6
77.8
0.0
19.4
2.2
77.8
79.6
50.3
18.2
61.4
14.7
34.3
5.1
40.0
42.7
MSP (Luo et al., 2020)
39.4
76.2
0.0
18.1
1.6
75.7
47.2
16.9
17.0
28.6
4.5
38.8
41.5
L2 norm
82.4
37.3
76.4
0.0
22.1
2.5
76.6
77.8
49.9
18.5
72.4
13.3
28.9
5.3
10.4
43.1
80.2
38.3
76.9
0.0
20.3
2.5
76.8
78.8
51.0
17.7
71.6
14.2
27.9
4.8
40.1
42.8
KL divergence
37.4
77.6
23.2
77.0
19.4
17.7
15.0
10.9
13.8
L 78.8
38.6
77.4
0.0
22.9
2.5
76.6
77.6
50.4
17.3
80.8
15.3
28.8
3.4
40.7
43.6
2) To investigate the impact of the known-region map R we compare the results of CLAN + KRADA with/withon withhutt 1bbn on SYNTHIA + Cityscapes in Table 4. The performance CLAN + KRADA w/o Rkn is worse than CLAN + KRADA neither at the last epoch nor at the best epoch, which confirms the effectiveness of R and the necessity of rejection of unknown-class regions in target images for better domain alignment.
3) We also compare the results of three statistical metrics (MSP, L2 norm, and KL divergence) used for generating unknown-class pseudo labels in Table 5. MSP achieves satisfactory and comparable results and L2 norm achieves the highest unknown-class IoU. Although KL divergence obtains relatively lower unknown-clas. IoU, it has superior performance in segmenting known classes (mIoU*) and the best segmentation overall performance (mloU). Both L2 norm and KL divergence leverage the property of the unknown class to compare the output probability distribution a pixel in the test image with the known-class prior probability distribution. This indicates that the statistical metric is not fixed and also proves the adjustability and flexibility of KRADA. However, MSP does not utilize the property of the unknown class derived from the source data, which may be the main reason why MSP achieves a slightly inferior performance.
4) To extend our proposed method in more segmentation settings, we have conducted experiments to prove that this method can be applied to the same domain, i.e., only with unknown class shifts without target domain shifts. Therefore, we use the abandoned SYNTHTA images (those with the unknown class) as the target domain and show the results in Table 6. The results show that the proposed method not only can be applied OSDAS (with unknown class shifts and target domain shifts) but also is applicable to the same-domain setting (with only unknown class shifts).
5) To evaluate the effectiveness of using the known-class prior distribution to depict the distribution characteristics of the unknown class, we also use the uniform distribution as a baseline and make a comparison in Table 7. Although the uniform distribution achieves similar mloU and mloU* as the known-class prior distribution the best epoch, its unknown-class IoU values are significantly lower the both last and best epochs. Table shows that the known-class prior distribution more suitable to portray the distribution characteristics of the unknown class while taking full advantage of the data attribute of source data.
10

---

Published in Transactions on Machine Learning Research (02/2023)
Table 6: Results of CLAN + KRADA applied to the same domain (SYNTHIA - SYNTHIA with the
unknown class).
road
side.
build.
lence
pole
person
rider
bilke
unk.
Setting
mToU
mio
Same domain
75.9
44.5
26,6
39.72
0.2
60.3
54.3
41.6
23.7
28.6
I 69.7
44.4
85.0
27.0
39.5
0.1
63.0
81.8
56.
36.7
57.5
17.6
24.8
7.7
43.6
46.4
OSDAS
B 82.4
37.3
76.4
0.0
22.1
2.5
76.6
17.8
49.9
18.5
72.4
15.3
28.9
5.3
40.4
43.1
L. 80.2
38.3
76.9
0.0
20.3
2.5
76.8
78.8
51.0
17.7
71.6
14.2
27.9
4.8
40.1
42.8
Table 7: Results of CLAN + KRADA with different porior for the unknown-class pseudo-label generation on SYNTHIA + Cityscapes.
road
build.
feace
pnle
sign
veg.
sky
person
rider
10otor
bikke
11nk.
Uknown-class
mloU
mloU"
Uniform distribution
80.6
36.8
77.8
0.0
24.6
2.4
76.7
77.1
49.3
18.1
81.7
14.3
30.7
2.1
40.9
43.9
77.6
37.4
77.0
0.0
22.4
2.0
76.2
48.5
17.3
13.8
29.8
1.6
39.9
42,9
Known-class prior distribution
80.2
37.4
77.6
0.1
23.2
2.8
77.0
78.5
19.4
17.7
80.4
15.0
30.1
3.9
10.9
43.8
78.8
38.6
77.4
0.0
22.9
2.5
76.6
77.6
50.4
17.3
80.8
15.3
28.8
3.4
40.7
43.6
Table 8: Results of CLAN + KRADA with single and two classification heads on SYNTHIA Cityscapes.
pe1300
rood
side.
build.
10n0e
pule
v8g.
rider
060
bike
uuk.
Classification heads
mloU
mIoU"
67.3
36.0
76.1
0.0
20.6
2.1
72.1
73.5
18.5
16.3
61.9
11.9
3.5
37.2
39.3 8
55.7
31.3
75.5
0.0
18.3
1.8
76.1
76.8
48.7
17.0
62.7
13.7
30.0
2.9
36.4
39.0
C and C*
82.1
37.3
76.4
0.0
22.1
2.5
76.6
77.8
49.9
18.5
72.4
15.3
28.9
5.3
40.4
43.1
80.2
38.3
76.9
0.0
20.3
2.5
76.8
78.8
51.0
17.7
71.6
14.2
27.9
4.8
40.1
42.8
Table 9: Results of CLAN + KRADA with varying Y on SYNTHIA + Cityscapes.
road
side.
build.
Foncp
pole
sign
person
rider
motoor
bike
unk
miou
mio U
0.003%
79.8
38.2
77.3
0.0
22.1
23
76.9
81.0
51.9
19.3
78.8
20.7
28
0.1
41.2
44.4
37.5
77.8
21.
78.0
46.5
18.8
13.1
40.6
B
80.3
39.4
78.2
0.1
24.8
2.8
78.0
79.3
49.4
18.0
80.8
12.9
31.0
1.6
41.2
44.2
0.01%
81.0
38.6
78.0
0.0
24.0
2.5
78.0
76.8
47.7
17.5
79.6
11.3
29.8
1.0
40.4
43.4
0.05%
37.3
76.4
76.6
49.9
15.3
5.3
40.4
43.1
80.2
38.3
76.9
0.0
20.3
2.5
76.8
78.8
51.0
17.7
71.6
14.2
27.9
4.8
40.1
42.8
0.1%
79.0
35.8
76.6
0.0
21.4
188
76.5
76.5
47.4
17.7
72.9
14.2
27.2
4.6
39.4
12.1
73.9
32.0
73.9
0.0
19.3
1.9
74.4
77.0
47.5
16.2
61.2
11.8
24.9
4.0
37.0
39.5
0.5%
B
70.3
31.9
73.7
0.1
19.7
0.9
69.8
71.9
48.9
18.3
64.4
16.5
33.6
3.9
37.4
40.0
49.9
11.3
48.0
0.0
10.2
34.6
45.4
27.1
7.5
28.7
6.0
8.5
2.1
20.0
21.3
6) To verify the necessity of two classification heads C and C*, we conduct the ablation study in Table 8. Experimental results demonstrate that two classification heads are necessary since these two classification heads have different functions and do not mix with each other.
7) To show the effects of different 7, we change the value of y and show the results in Table 9. Compared with 0.05%, reducing y (0.005% and 0.01%) leads to the decrease of identifying the unknown-class pixels (lower unknown-class ToU) and the increase segmenting known classes and overall performance (mIoU* and mloU). On the contrary, increasing y (0.1% and 0.5%) does not mean higher unknown-class ToU and instead, it diminishes the known-class segnentation performance and thus an inferior overall performance (lower mloU* and mloU). More specifically, we achieve unsatisfactory and unstable results when y is set 0.5%. To obtain satisfactory and competitive sogmentation performance, it is appropriate to set between 0.01% 0.1%. Therefore, we finally set 7 as 0.05%.
8) Comparison with other related works. Since there is no relevant work to solve the scenario that completely consistent with ours: open set, domain adaptation, and semantic segmentation, other methods are not casy and suitable directly be compared with our proposed method. Thus we adopt compromise way to find the comparable part that is unknown-class pseudo-label generation criteria. Therefore, compared the
11

---

Published in Transactions on Machine Learning Research (02/2023)
Table 10: Comparison with other related works.
pesson
2o0d
side.
build.
Seuce
pole
aigh
vog.
ske
rider
bikes
Metrics
mloU
mloU
MSP (Luo et al., 2020)
81.9
37.6
77.8
0.0
19.4
2.2
77.8
79.6
50.3
18.2
61.1
11.7
31.3
5.1
20.0
42.7
82.2
39.4
76.2
0.0
18.1
1.6
75.7
78.7
47.2
16.9
57.6
17.0
28.6
4.5
38.8
41.5
DML (Cen et sl., 2021)
B
77.3
37.1
77.3
0.0
23.9
1.6
75.9
68.6
18.4
16.5
76.7
12.8
30.7
0.9
39.1
22.3
72.6
22.0
71.4
62.7
18.5
37.3
MSP-baseline (Vaze al., 2021)
B 82.5
35.1
75.0
0.0
19.7
2.4
74.7
77.2
47.6
19.3
78.9
7.0
29.2
2.5
39.4
42.2
79.3
35.6
73.6
0.0
17.4
2.1
74.1
76.3
48.4
19.0
73.8
7.3
29.7
2.2
38.5
41.3
1.2 norm
82.4
22.1
77.8
72.4
28.9
L 80.2
38.3
76.9
0.0
20.3
2.5
76.8
78.8
51.0
17.7
71.6
14.2
27.9
4.8
40.1
42.8
KL divergence
80.2
37.4
77.6
0.1
23.2
2.8
77.0
78.5
49.4
17.7
80.4
15.0
30.1
3.9
40.9
43.8
78.8
38.6
77.4
0.0
22.9
2.5
76.6
77.6
50.4
17.3
80.8
15.3
28.8
3.4
40.7
43.6
Table 11: Results on COVID-19 infection segmentation.
Method
Lung IoU
Infection loU
Accuracy
Precision
Recall
Fl-score
OSBP (Saito al., 2018b)
69.9
0.6
66.7
66.7
100.0
80.0
AdaptSegNet + KRADA
80.7
0.4
66.7
100.0
80.0
CLAN + KRADA
81.7
0.8
66.7
66.7
100.0
80.0
FADA + KRADA
85.6
2.0
100.0
100.0
100.0
100.0
performance of different unknown-clen classs pscudo-label generation criteria: including the MSP in (Luo al., 2020) and DML in (Cen et al., 2021). We realized MSP and DML in CLAN KRADA framework and showed the results of SYNTHIA - Cityscapes in Table 10. In addition, Vaze et al. (Vaze et al., 2021) introduced baseline for open-set recognition (OSR). The OSR baseline refers to training a K-way classifier and identifying the unknown pixels directly during inference by assigning the unknown-class probability with minus the maximum value of softmax probabilities. We denoted this OSR baseline as MSP-baseline to distinguish it from MSP (Luo et al., 2020). Table 10 also shows the results MSP-baseline on CLAN for the SYNTHIA + Cityscapes task. Our proposed methods significantly outperform Dsselbaeline and DML with large margins in terms of unknown-class ToU, mIoU, and mloU*.
Real-world OSDAS task (COVID-19 infection segmentation in CT scans). To construct COVID- 19 task, exploit the public datasets summarized in (Ma et al., 2020). The source data consists of normal CT scans with lung annotations. Both target data and test data include COVID-19 cases and non-infeen non-infected CT scans. Detailed data descriptions and data processes are introduced in Appendix B. To give a comprehensive comparison, we evaluate the proposed method from both pixcl-level and instance-level aspects. The IoU values of lung and infection averaged among all test cases are reported in Table 11, denoted as Lung IoU and Infection IoU. We also provide four other metrics: Accuracy, Precision, Recall, and Fi-score, commonly used in medical fields for instance-level evaluation.
Results on real-world OSDAS task. In Table 11, all these models achieve 100.0% recall, mean- ing that all infected cases are detected, which is desirable medical image diagnosis. OSBP gains 0.6% ToU infection and the same instance- level performance AdaptSegNNtt
0
KRADA and CLAN + KRADA, but it heavily sacrifices the seg- mentation accuracy of the lung. Compared with OSBP, CLAN + KRADA shows superior perfor- mance in both pixel-level and instance-level eval- nations. Moreover, FADA + KRADA greatly out- performs other models and obtains the best results. Examples of segmentation results are provided in Figure 5.
Figure 5: From left to right: CT slice, Ground Truth, and results of FADA and FADA + KRADA, where the brighter area is the infected area.
12

---

Published in Transactions on Machine Learning Research (02/2023)
7 Conclusion
This paper considers the semantic segmentation task in an open world, where test images have a different distribution from training images and contain unknown categories/ peeggries/classes cllsssss. To address this new and challenging problem, we explore the inherent property of unknown classes and propose an end-to-end framework, KRADA, that performs known-region-awwar domaín alignment. KRADA a generalized framework with no particular structure dependency and can be easily implemented on existing CSDAS methods, as demonstrated by three realizations of KRADA in our experiments. Experimental results validate that KRADA enables CSDAS methods to distinguish unknown-class pixels from known-class pixels and classify known-class pixels well.
Acknowledgments
CHZ and BH were supported by NSFC Young Scientists Fund No. 62006202, Guangdong Basic and Applied Basic Research Foundation No. 2022A1515011652, RGC Early Career Scheme No. 22200720, RGC Research Matching Grant Scheme No. RMGS20221102, No. RMGS20221306 and No. RMGS20221309, BH was also supported by HKBU CSD Departmental Incentive Grant. CG is supported by NSF of China (No: 61973162), NSF of Jiangsu Province (No: B72021013), NSF for Distinguished Young Scholar of Jiangsu Province (No: BK20220080), and CAAl-Huawei MindSpore Open Fund. TLL was partially supported by Australian Research Council Projects IC-190100031, LP- P2220100527, DP-220102121, and FT-220100318. RFZ was partially supported by NSF of China No. 62172083. This research is partially supported by the Research Matching Grant Scheme RMGS2021 8 06 from the Hong Kong Government.
References
Hugo JWL Aerts, Emmanuel Rios Velazquez, Ralph TH Leijenaar, Chintan Parmar, Patrick Grossmann, Sara Carvalho, Johan Bussink, René Monshouwer, Benjamin Haibe-Kains, Derek Rietveld, et al. Decoding tumour phenotype by noninvasive imaging using quantitative radiomics approach. Nature communications, 5(1):1-9 9, 2014.
Ibrahim A Ahmad and Patricia B Cerrito. Goodness of fit tests based on the L2-norm of multivariate probability density functions. Journal of Nonparametric Statistics, (62):199-181, 1993.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), pp. 1992- 2015.
Terrance E Boult. Steve Cruz, Akshay Raj Dhamija, M Gunther, James Henrydoss, and Walter J Scheirer. Learning and the unknown: Surveying steps toward open world recognition. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), pp. 9801-9807, 2019.
Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 422-438. Springer, 2020.
Jun Cen, Peng Yun, Junhao Cai, Michael Yu Wang, and Ming Liu. Deep metric learning for open world semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. -5333, 2021.
Liang-Chis Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEFE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In Proceedings of the IEEE Conference on Computer Vision and Pollern Recognition (CVPR), pp. 1841 1850, 2019.
Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (TCIA): maintaining and operating public information repository. Journal of digital imaging, 26(6):1045-100 1057, 2013.
13

---

Published in Transactions on Machine Learning Research (02/2023)
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3213 3223, 2016.
Zhiying Cui, Wu Longshi, and Ruixuan Wang. Open set semantic segmentation with statistical test and adaptive threshold. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Lí, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-24882 255. Icce, 2009.
Zhen Fang, Jie Lu, Feng Liu, Junyu Xuan, and Guangquan Zhang, Open set domain adaptation: Theoretical bound and algorithm. IEEE Transactions on Neural Networks and Learning Systems, 2020.
Qianyu Feng, Guoliang Kang, Hehe Fan, and Yi Yang. Attract distract: Exploit the margin of open set. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 7990-7999, 2019.
Alberto Garcia, Sergio Orts-Escolanc Sergiu Oprea, Victor Villena-Martinez, and Jose Garcia- Rodriguez. A review on deep learning techniques applied to semantic segmentation. arXiv preprint arXiv:1704, 164.0774 2017.
Alison L. Gibbs and Francis Edward Su. On choosing and bounding probability metrics. International statistical review, (0(3):4194433) 2002.
Rui Gong, Wen Li, Yuhua Chen, and Lue Van Gool. Dlow: Domain flow for adaptation and generalization. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognation (CVPR), pp. 2477-2486, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 770 778, 2016.
Judy Hofiman, Dequan Wang, Fisher Yu, and Trevor Darrell. Fens in the wild: Pixel-level adversarial and constraint- based adaptation. arXie preprint 161iv 02649, 2016.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International Conference on Machine Learning (ICML), pp. 1989 1998, 2018.
Jaedong Hwang, Seoung Wug Oh, Joon- Young Lee, and Bohyung Han. Exemplar-based open-set panoptic segmentation network. In Proceedings of the IEFE Conference on Computer Vision and Pallern Recognilion (CVPR), 2021.
K J Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Ma Jun, Ge Cheng, Wang Yixin, An Xingle, Gao Jiantao, Yu Ziqi, Zhang Minqing, Liu Xin, Deng Xueyuan, Cao Shucheng, et al. Covid-19 CT lung and infection segmentation dataset. [Online]. Available: https: //doii.og//0.. wttp:/irrrr....22/eeenoeeeo www.//eeoooo..... 37777777. 2020.
Guoliang Kang, Yunchao Wei, Yi Yang, Yueting Zhuang, and Alexander Hauptmann. Pixel-level cycle association: A new perspective for domain adaptive semantic segmentation. In Advances in Newral Information Processing Systems (Neurips), 2020.
Mare Kéry and J Andrew Royle. Applied Hierarchical Modeling in Ecology: Analysis of distribution, abundance and species richness in R and BUGS: Volume 1: Prelude and Static Models. Academic Press, 2015.
14

---

Published in Transactions on Machine Learning Research (02/2023)
KJ Kiser, S Ahmed, SM Sticb, et al. Data from the thoracic volume and pleural effusion segmentations in diseased lungs for benchmarking chest CT processing pipelines. The Cancer Imaging Archive, 2020.
Jogendra Nath Kundu, Naveen Venkat, Ambareesh Revanur, R Venkatesh Babu, et al. Towards inheritable models for open-set domain adaptation. In Proceedings of the LEEE Conference on Computer Vision and Pullern Recognition (CVPR), pp. 12376 12385, 2020.
Yunsheng Li, Lu Yuan, and Nuno Vasconcelos. Bidirectional learning for domain adaptation of semantic segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6936-6945, 2019.
Hong Liu, Zhangjie Cao, Mingsheng Long, Jianmin Wang, and Qiang Yang. Separate to adapt: Open set domain adaptation via progressive separation. In Proceedings of the IEEE Conference on Computer Vision and Poltern Recognition (CVPR), pp- 2927-2933 2936, 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431- 3440, 2015.
Yadan Luo, Zijian Wang, Zi Huang, and Mahsa Baktashmollagh. Progressive graph learning for open-set domain adaptation. In International Conference on Machine Learning (ICML), pp. 6468-6478, 2020.
Yawei Luo, Liang Zheng, Tao Guan, Junqing Yu, and Yi Yang. Taking a closer look at domain shift: Category- level adversaries for semantics consistent domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2507-2516, 2019.
Jun Ma, Yixin Wang, Xingle An, Cheng Ge, Ziqi Yu, Jianan Chen, Qiongjie Zhu, Guoqiang Dong, Jian He, Zhiqiang He, et al. Towards efficient covid-19 CT annotation: A benchmark for lung and infection segmentation. arXiv preprint oxXw:2004.12552 2020.
Ke Mei, Chuang Zhu, Jiaqi Zou, and Shanghang Zhang. Instance adaptive self-training for unsupervised domain adaptation. arXiv preprint arXio:2008. 12197, 2020.
Dominik Müller, Iñaki Soto Rey, and Frank Kramer. Automated chest CT image segmentation of COVID-19 lung infection based on 3D U-Net. arXiv preprint ar Xrx:::007.00 04 774, 2020.
Yingwei Pan, Ting Yao, Yehao Li, Chong- Wah Ngo, and Tao Mei. Exploring category-agnostic clusters for open-set domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13867-13874 2020.
Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEFF International Conference on Computer Vision (ICCV) pp. 754-763, 2017.
Stephan R Richter, Vibhav Vincet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 102 118. Springer, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognétion (CVPR), pp. 3234-3243, 2016.
Karsten Roth, Latha Pemula, Joaquin Zepeda, Bernhard Schölkopf, Thomas Brox, and Peter Gehler. Towards total recall in industrial anomaly detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 14318-14328, 2022.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3723 3732, 2018a.
15

---

Published in Transactions on Machine Learning Research (02/2023)
Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Proceedings of the European Conference on Compuler Vision (ECCV), pp. 153-168 2018b.
Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recognilion Workshops, pp. 2018.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 746-760. Springer, 2012.
Kai-Sheng Song. Goodness-of of--tttt-tfff- fit tests based on kullback-leibler discrimination information. IEEE Transactions on Information Theory, 481)):1103-1117, 2002.
Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, Jeffrey N Chiang, Zhihao Wu, and Xiaowei Ding. Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis, 63:101693, 2020.
Marco Toldo, Audrea Maracani, Umberto Michieli, and Pietro Zanuttigh. Unsupervised domain adaptation in semantic segmentation: a review. Technologies, 8(2):35, 2020.
Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming Hsuan Yang, and Manmohan Chan- draker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEFE Conference on Computer Vision and Pottern Recognition (CVPR), pp. 7472 7481, 2018.
Yi-Hsuan Tsai, Kihyuk Sohn, Samuel Schulter, and Manmohan Chandraker. Domain adaptation for structured output via discriminative patch representations. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1456-1465, 2019.
Oldrich Vasicek. A test for normality based on sample entropy. Journal of the Royal Statistical Society: Series B (Methodological), 38(1)544-59, 1976.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set. classifier is all you need. In International Conference on Learning Representations, 2021.
Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE Conference on Compuler Vision and Pollern Recognition (CVPR), pp. 2517 2526, 2019.
Haoran Wang, Tong Shen, Wei Zhang, Ling- Yu Duan, and Tao Mci. Classes matter: A fine-grained adversarial approach to cross-domain semantic segmentation. In Proceedings of the European Conference on Compuler Vision (ECCV), pp. 642-659. Springer, 2020.
Qiming Zhang, Jing Zhang, Wei Liu, and Dacheng Tao, Category anchor-guided unsupervised domain adaptation for semantic sogmentation. In Advances in Neural Information Processing Systems (Neurips), pp. 433 443, 2019.
Li Zhong, Zhen Fang, Feng Liu, Bo Yuan, Guangquan Zhang, and Jie Lu. Bridging the theoretical bound and deep algorithms for open set domain adaptation. arXiv preprint Xxx::2006.13022, 2020.
Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaplation for semantic segmentation via class-balanced self- tifftrrainigg In Proceedings of the European Conference on Computer Vision (ECCV), pp. 289 305, 2018.
Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self-training. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pp. 5982 5991, 2019.
16

---

Published in Transactions on Machine Learning Research (02/2023)
A Related Works
In this section, we briefly review two kinds of domain adaptation settings related to open-sel domain adaptation segmentation (OSDAS).
Unsupervised open set domain adaptation (UOSDA) is first proposed by Busto et al. (Panareda Busto & Gall, 2017), where both source and larget domain contain private and shared classes, but we only know shared labels. To relax the requirement of private source labels, Saito et al. (Saito et al., 2018b) introduce a new concept of UOSDA setting where only the target domain has private labels and propose Open Set Poee-Propaggtion (OSBP), a novel adversarial training- UOSDA methods follow this new and realistic setting and cope with it by a coarse-to-fine weighting mechanism (Liu et al., 2019), a self-supervised task (Bueci et al., 2020), self-ensembling (Pan et al., 2020), or inheritable models (Kundu et al., 2020). Additionally, Fang el al. and Zhong el al. (Fang et al., 2020; Zhong et al., 2020) provide theoretical analysis for UOSDA and introduce open set difference, special term that facilitates recognizing unknown target samples. Feng et al. (Feng et al., 2019) explicitly utilize the semantic margin of open set data to make the unknown class apart from the decision boundary and the known classes more separable. Luo et al. (Luo et al., 2020) propose a graph neural network with episodic training and achieve ofate-of-the-ans performance. However, these existing UOSDA methods only focus on classification tasks, and most of them cannot be modified for semantic segmentation by simply convolutionalizing their classification architectures. For example, the state-of-the-art UOSDA method (Luo el. al., 2020) requires source episodes containing each known class in batch to construct graph neural network. This requirement is hard to meet in segmentation tasks which demand large memory to support dense computations. Therefore, modifying existing UOSDA methods for OSDAS is not usually feasible.
method for an open set scenario. Later on,
Closed set domain adaptation for semantic segmentation (CSDAS) has been extensively studied and developed maturely (Toldo et al., 2020). A predominant stream of UCSDA works is adversarial training (AT) based methods which minimize adversarial losses to align the distributions between the source and target domains at input level (Hoffman et al., 2018; Gong et al., 2019), feature level (Hoffman et al., 2016; Vu et al., 2019; Chen et al., 2019; Wang et al., 2020; Hoffman et al., 2018), or output space level (Luo et al., 2019; Saito et al., 2018a; Tsai et al., 2018). Recently, series of approaches (Zou et al., 2018; Mei et al., 2020; Zou et al., 2019; Zhang et al., 2019) based on deep self- relf-training (ST) has become an alternative research direction. These approaches generate pscudo-labels for target samples to provide cxtra supervision so that the net work can be trained under the supervision of two domains. Zou el al. (Zou et al., 2018) propose a dlss-balanced self-training (CBST) framework to overcome the issue of imbalanced target pseudo-labels, To avoid overconfident wrong pscudo-labels, Zou et al. (Zou et al., 2019) further incorporate two types of confidence regularization to CBST. To generate high-quality pscudo-labels, Mci et al. (Mei et al., 2020) propose an instance adaptive selector to generate more accurate pseudo-labels. In addition, several works (Li et al., 2019; Tsai et al., 2019) have been developed by combining AT and ST methods and presented great potential. Despite the well-studied CSDAS mothods, they cannot detect unknown classes and lead to negative transfer due to mismatched label sets. Therefore, current CSDAS methods are not applicable to an open world and cannot solve OSDAS tasks well.
Based on such well-studied CSDAS methods, we propose an end-to-end framework, KRADA, which can modify current CSDAS segmentation methods and adapt them for OSDAS tasks.
B Experiment Setup
B.1 Experiments on Two Synthetic OSDAS Tasks
Data description: For the task SYNTHIA - Cityscapes, SYNTHIA originally includes 9,400 synthetic images and has 16 common classes with Cityscapes. These three classes (wall, light, and bus) are selected to form the unknown class, and we discard those images containing either of the three classes. The remaining data in SYNTHIA contain 750 images, and there are 13 shared known classes in this task. For the task GTA5 Cityscapes, GTA5 initially contains 24,966 images rendered from the GTA5 game engine and has the same 19 category annotations as Cityscapes. Similarly, we choose two classes (fence and sign) as the unknown
17

---

Published in Transactions on Machine Learning Research (02/2023)
class and only retain the images which do not contain the unknown class. The remaining images in GTA5 include 17 categories and 2,277 images. Cityscapes is real-world dataset consisting of a training set with 2,957 images and a validation set with 500 images. We divide the Cityscapes training set into training splits of 2,500 images for training and evaluation splits of 457 images for hyperparameter sclection. The results on the Cityscapes validation set are reported for performance comparison.
Implementation details: For a fair comparison, we adopt the Deeplab-V V2 (Chen et al., 2017) framework with ResNet-101 (He et al., 2016) pretrained on ImageNet (Deng et al., 2009) as the segmentation base network. We implement KRADA on three CSDAS methods: AdaptSegNet (Taai et al., 2018), CLAN (Luo et al., 2019), and FADA (Wang et al., 2020). For each CSDAS method, we additionally duplicate: a last convolution classification module as C which is arranged in parallel with the original classifier C' after the feature extractor. C* is identical as C' expect for the last layer with channel mumber K to output the predicted score map over known classes. Regarding the pseudo-label hyperparamcters in KRADA, 7; 8, and 8 are chosen as 0.05%, 0.99, and 0.1 respectively for these two synthetic segmentation tasks. a set as 0.1, 0.03, and 0.2 for the AdgptSegNee, CLAN, and FADA models equipped with KRADA. Other experimental settings such as discriminator structure, optimization policy, and hyperparameters in three modified models are almost the same as those described in the original papers. All the models are implemented using Python 3.6 and Pytorch 1.7 on a TITAN Tesla V100 GPU.
B.2 Experiments COVID-19 Infection Segmentation in CT Scans
We describe the details of our data used in the COVID-19 infection segmentation task. More specifically, we exploit the public datasets summarized in (Ma et al., 2020) to construct a real-world OSDAS task. Source data includes 30 CT scans which are randomly selected from NSCLC left and right lung segmentation dataset (Kiser et al., 2020; Aerts et al., 2014; Clark et al., 2013) (with CC BY-NC license). The target data consists of 20 COVID-19 CT scans and 10 non-infected CT scans. More specifically, COVID-19 CT scans are publicly available (Jun et al., 2020) (with CC BY-NC-SA license), and each case has the annotations of the left lung, right lung, and infection. Non-infected CT scans randomly selected from MICCAI 2019 StructSeg lung organ segmentation challenge, and only use their left lung and right lung annotations. We divide the target data into two parts for training and testing, and cach part includes 10 COVID-19 cases and 5 non-infected CT scans. Following (Ma et al., 2020), we adjust each CT scan to lung window [-1250, 250] and then normalize to [0,255] for pre-processing. We slice each CT volume into 2D slices and perform the same data augmentation as (Müller et al., 2020). In this task, the pscudo-label hyperparameters 7, B, and 5 are chosen as 1%, 0.99, and 0.001 respectively. set as 0.01, 0.01, and 0.1 for the three models equipped with KRADA. Other experimental setups are similar to the above two synthetic tasks.
C Potential Social Impacts
Our research allows training segmentation model for a new dataset by exploiting existing annotated data, which reduces the annotation cost. Besides, the proposed method can recognize the abnormal and unseen regions in new dataset and give an early warning. This is quite critical and has broader significance in the medical field, especially when it comes to a new disease, and we know nothing about it. However, this method is not entirely mature, and there are potential risks of false alarms and missed detection. Therefore, it cannot be used in clinical medical practice for the time being.
18


Reviews:
[{"review_id": "bxXt7vLwgK", "reviewer": "Reviewer_biFQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ Soling the two task jointly makes sense as It is very likely that the two problems occur when dealing with open world scenario. \n\n\\- However, this task was already proposed by (Saito et al. ECCV18) for classification. Extending it for semantic segmentation does not seem an important contribution in my opinion.\n\n\\- In general the presentation of the paper is ok, but there are some parts that are not clear or misleading and should be corrected. (see next section)\n\n\\- In the beginning of the second page, the authors present the term open world semantic segmentation (OSS) as it is a term that they invented. Instead there are already multiple papers talking about OSS that should be cited. Even more, in this paper, the new classes are just detected as outliers, while in other works (e.g. Cen et al. ICCV21) the method not only detects pixels of new classes, but it is also able to cluster them in meaningful categories. In general there is some confusion with the term open world. For instance, in my understanding when talking about open world I would assume that there is no knowledge at all of the deployment domain, as in domain generalization.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper present a method for open world semantic segmentation in which the method is not only able to distinguish pixels coming from new categories, but also align the classifier to the deployment data that might have a different distribution that the data used for training.\nThe contributions of the paper are the following:\n- this method solves the problem of open-world semantic segmentation, in which there can be new categories not seen during training and the alignment problem, in which even the categories seen during training they might be affected by a domain shift. \n- The method is evaluated on two synthetic to real segmentation datasets for autonomous driving and one for detecting COVID19 from CT scans.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- (Critical) The presentation of the proposed work is misleading in my opinion. the term open world semantic segmentation (OSS) as the task they aim to solve. However, the term OSS normally does not consider the domain alignment. Thus, I think that the authors should propose another name for their task in which is clear that they do unknown class detection and domain alignment jointly.\n\n- (Critical) In the introduction authors present only some papers for closed-set domain adaptation for semantic segmentation. To have a better idea of what has been already proposed, the authors should provide a section of related work in which they distinguish their settings from previous work in related topics, for instance open world approaches and domain adaptation methods for image classification and segmentation. This would provide also better and more interesting baselines to compare with.\n\n- (Critical) The definition in problem 1 is difficult to read and understand and it should be improved. Authors first talk about target domain $X^T$, then they go back to source domain $X^T$ and then back again to target, while presenting the definition of some variables in the wrong location. Also, in most of the cases authors use \"\\mathcal\" for sets, but $\\mathcal M$ is used for the model.\n\n- (Critical) In several parts of the paper (e.g. end of 1st paragraph section 4.1, remark 2) authors mention that they highly fitted pixels are classified as unknown pixels. In my understanding it should be the other way around. Only low-fitting pixels should be classified as unknown. Authors should clarify this point and if needed, correct it.\n\n- (Critical) The evaluation is performed by comparing the proposed approach that learn to select which pixels come from unknown distributions to methods for closed-set domain adaptation (excluding Saito et al. 2018b). It is evident that the proposed method works better than simple domain adaptation approaches when the target domain has new classes, as that is the aim of the proposed method. What would happen when there are no new classes?\n\n- (Critical) The evaluation of the method is done by explicitly removing images containing certain classes form a given training dataset. How are those classes chosen? To have a better evaluation of the proposed approach authors should evaluate the average performance  when different classes are removed (Not only one additional case in the ablation study). \n\n- (Minor) At the end of page 1, authors talk about variations in light for domain shift in CT scans. In that case, in my understanding there is no light, as we are not dealing with natural images.\n\n- (Minor) Not sure what is the advantage of presenting results for the best models (B) and the last iteration model (L). I guess, this evaluation come from previous work, but authors should explain the reason or present only the best model which makes more sense to me.\n\n", "broader_impact_concerns": "No broader impact concerns found. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eLfzItDboz", "reviewer": "Reviewer_2jSe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a relatively simple system that results in decent but incremental gains, particularly in recognizing the unknown class, which the other segmentation systems are not capable of.  Even though it is straightforward, I still have questions on whether even simpler versions of this idea might be as effective, some of which should be included as additional baselines.  In particular, the post-hoc \"unknown\" classifier that simply thresholds on max softmax score should be included as a comparison.\n\nThere is also some related work that could be discussed more as well on open-set recognition in the context of classification (e.g. Vaze et al 2022 https://openreview.net/pdf?id=5hLP5JY9S2d, & related prior works) as well as anomaly detection (e.g. Roth et al 2021 https://arxiv.org/pdf/2106.08265.pdf), which produces low-resolution heatmaps for \"unknown\" class as well.  Note the \"MSP\" baseline used in Vaze et al is the post-hoc thresholding classifer I think should be a baseline.\n\nIn addition, I don't know that the source-to-target difference is necessary --- the same method could be applied to same-domain (source only), training on a split where there are held-out classes in the test set.  That would evaluate the effectiveness on unknown class shifts, not in the presence of target domain shift.  The same method could then be adapted also for cross-domain synthetic-to-real.  This could strengthen the paper by demonstrating integration of the method in multiple settings.\n\n\n\nAdditonal questions and comments:\n\n* How is the p^prior calculated?  Is it the frequency of class pixels?  Or is it optimized in some other way?\n\n    * If it is just a frequency count, the uniform distribution would be even simpler, and could be as effective --- is it?\n\n    * A single vector for p^prior appears to assume an independent class distribution, but often class presence is correlated.  For example, residential areas have buildings, sidewalk, pedestrians, etc., while highways have more medians and overhead signs.  So the prior class distribution could also be modeled with a more sophisticated distribution, that has has nonzero covariance and/or is multimodal.  Is there any benefit to modeling this?\n\n* Are the two classification heads C and C^* necessary?  An alternative would be to use just one head with K+1 units, but two softmaxes, one on all K+1 classes, and one on just the first K.  The second could be used in place of C^*, as it conditions on the pixel being \"known\".\n\n* What is the effect of changing $\\gamma$?  Particularly since the unknown class appears under-predicted, this would be interesting to vary.  Is \"unknown\" predicted more often on the target if this is increased?\n\n* Sec 2 problem statement:  It would be good to also include the words \"source\" and \"target in the definition here, when describing X^S ad X^T (e.g., \"Let X^T be a r.v. following a distribution (called the \"target\" image data)\" or along these lines, similarly for X^S and source).  Right now being presented with X^T at the start of the definition is easy to mistake the T for transpose.\n\n* The citation format in text would be clearer including brackets around authors and year, not just year. Right now these can blend with the rest of the text\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method for cross-domain semantic segmentation that explicitly recognizes unknown classes in the target domain along with source-to-target transfer.  Two pixelwise classification heads are used, one of which classifies the known K source classes, while the other adds a K+1'th unit in addition that classifies the \"unknown\" category.  The predictions from the first are used to create online pseudolabel targets for the second, by targeting \"unknown\" for pixels with softmax prediction closest to the source class frequency prior, using EMA to find a value that thresholds a fixed budget fraction on average.  Interestingly, the pseudolabel bitmap is also used to disable the adversarial feature-discrimination loss used by the cross-domain segmentation systems, as these features must be kept different from the source in order to correctly classify as \"unknown\".  The system is integrated into three different segmentation systems, each evaluated using three synthetic driving datasets transferring to one real (cityscapes).  An additional application in a CT segmentation task for COVID-19 is also presented.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Max softmax probability threshold for \"unknown\" class (MSP baseline)\n\nStudy on different values for fraction $\\gamma$\n\nBoth of these are necessary additions in my view.\n", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B2109MmchD", "reviewer": "Reviewer_r9Vn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n+ This paper considers a new, rarely studied problem in semantic segmentation. It is an interesting and important task in the community.\n\n+ An unknown-class estimation is proposed to detect unknown pixels in the target domain.\n\n+ A known-region-aware domain alignment framework is introduced to perform a safe domain alignment under the open-world setting.\n\n\n*Weaknesses*\n\n- Although this paper solves a new setting, the novelty of the proposed method is somewhat limited. I am not supervised by the unknown-class estimation and the known-region-aware domain alignment methods. At least, I do not well agree these two techniques can meet the requirement of TMLR.\n\n- The motivation of unknown-class pseudo-label generation is not well-explained. Is there any theoretical or experimental supports?\n\n- The improvement of the proposed method is not consistent. For example, in many cases, the improvement of the proposed method is limited.\n\n- Except the final segmentation results, the authors should provide other metrics the verify the benefit of the proposed unknown-class pseudo-label generation. For example, the accuracy of the selected pseudo-labels.\n\n- For the COVID-19 dataset, do we need to ignore the new-infected region (unknown class) or detect? I think it is important to detect the new-infected region. However, in the proposed open-world setting and method, it seems the unknown-region should not be considered during training and detected during a testing, which I think is not reasonable in practice.\n\n- Many recent open-set domain adaptation methods are not considered.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers the problem of open-world semantic segmentation, where the unlabeled data include both domain shift and category shift. To solve this problem, this authors first propose an unknown-class estimation approach and design a  known-region-aware domain alignment framework to carefully align the data distribution of source and target domains. Experiments conducted on two tasks show the benefit of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please solve my concerns in the weaknesses.\n\n- Address the novelty of the proposed method.\n\n- Make the motivation more clear.\n\n- Explain the large improvement of the proposed method.\n\n- Better verify the effectiveness of unknown-class pseudo-label generation.\n\n- State the rationality of COVID-19 dataset under the proposed open-world setting.\n\n- Compare with more recent open-set domain adaptation methods in the community.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to TMLR and for the edits made to the paper based on the reviewer comments.  Overall, two of the three reviewers were quite positive about the paper, particularly after the discussion phase.  One reviewer had some concerns about novelty but, as pointed out by the authors, these concerns do not weigh heavily on the final decision.  Overall it does seem that the technical concerns have been addressed, and the problem studied would be of interest to several in the community.  I'm happy to recommend accepting the paper.
Decision: Accept (poster)