Title: Adversarial Learning for Semi-Supervised Semantic Segmentation
Authors: Wei-Chih Hung
Yi-Hsuan Tsai
Yan-Ting Liou
Yen-Yu Lin
Ming-Hsuan Yang
Venue: ICLR 2018
Keywords: semantic segmentation
adversarial learning
semi-supervised learning
self-taught learning

Abstract: We propose a method for semi-supervised semantic segmentation using the adversarial network. While most existing discriminators are trained to classify input images as real or fake on the image level, we design a discriminator in a fully convolutional manner to differentiate the predicted probability maps from the ground truth segmentation distribution with the consideration of the spatial resolution. We show that the proposed discriminator can be used to improve the performance on semantic segmentation by coupling the adversarial loss with the standard cross entropy loss on the segmentation network. In addition, the fully convolutional discriminator enables the semi-supervised learning through discovering the trustworthy regions in prediction results of unlabeled images, providing additional supervisory signals. In contrast to existing methods that utilize weakly-labeled images, our method leverages unlabeled images without any annotation to enhance the segmentation model. Experimental results on both the PASCAL VOC 2012 dataset and the Cityscapes dataset demonstrate the effectiveness of our algorithm.


Review discussion:
This paper describes techniques for training semantic segmentation networks. There are two key ideas:
- Attach a pixel-level GAN loss to the output semantic segmentation map.
That is, add a discriminator network that decides whether each pixel in the label map belongs to a real label map or not.
Of course, this loss alone is unaware of the input image and would drive the network to produce plausible label maps that have no relation to the input image. An additional cross-entropy loss (the standard semantic segmentation loss) is used to tie the network to the input and the ground-truth label map, when available.
- Additional unlabeled data is utilized by using a trained semantic segmentation network to produce a label map with associated confidences; high-confidence pixels are used as ground-truth labels and are fed back to the network as training data.
The paper is fine and the work is competently done, but the experimental results never quite come together. The technical development isn’t surprising and doesn’t have much to teach researchers working in the area. Given that the technical novelty is rather light and the experimental benefits are not quite there, I cannot recommend the paper for publication in a first-tier conference.
Some more detailed comments:
1. The GAN and the semi-supervised training scheme appear to be largely independent. The GAN can be applied without any unlabeled data, for example. The paper generally appears to present two largely independent ideas. This is fine, except they don’t convincingly pan out in experiments.
2. The biggest issue is that the experimental results do not convincingly indicate that the presented ideas are useful.
2a. In the “Full” condition, the presented approach does not come close to the performance of the DeepLab baseline, even though the DeepLab network is used in the presented approach. Perhaps the authors have taken out some components of the DeepLab scheme for these experiments, such as multi-scale processing, but the question then is “Why?”. These components are not illegal, they are not cheating, they are not overly complex and are widely used. If the authors cannot demonstrate an improvement with these components, their ideas are unlikely to be adopted in state-of-the-art semantic systems, which do use these components and are doing fine.
2b. In the 1/8, 1/4, and 1/2 conditions, the performance of the baselines is not quoted. This is wrong. Since the authors are evaluating on the validation sets, there is no reason not to train the baselines on the same amount of labeled data (1/8, 1/4, 1/2) and report the results. The training scripts are widely available and such training of baselines for controlled experiments is commonly done in the literature. The reviewer is left to suspect, with no evidence given to the contrary, that the presented approach does not outperform the DeepLab baseline even in the reduced-data conditions.
A somewhat unflattering view of the work would be that this is another example of throwing a GAN at everything to see if it sticks. In this case, the experiments do not indicate that it did.
We thank the comments and address the raised questions below.
Q1. Why do the authors present two largely independent ideas?
The novelty of this work is to incorporate adversarial learning for dense predictions under the semi-supervised setting without image synthesis. The adversarial learning and semi-supervised learning are not independent in our work. Without the successfully trained discriminator network, the proposed semi-supervised learning does not work well. The ablation study in Table 6 shows that without adversarial loss, the discriminator would treat most of the prediction pixels with low confidence of, providing noisy masks and leading to degenerated performance (drops from 68.8% to 65.7%).
Q2. Why don’t the author use the full DeepLab model?
We implement our baseline model based on the DeepLab in PyTorch for the flexibility in training the adversarial network. We did not use the multi-scale mode in the DeepLab due to the memory concern in section 4.2, in which the modern GPU cards such as Nvidia TitanX with 12 GB memory are not affordable to train the network with a proper batch size.
Although this issue may be addressed by the accumulated gradient (e.g., iter_size in Caffe), in PyTorch the accumulated gradient implementation still has issues (ref: https://discuss.pytorch.org/t/how-to-implement-accumulated-gradient/3822/12). We have also verified that it does not work in the current PyTorch version.
However, our main point of the paper is to demonstrate the effectiveness of proposed method against our baseline model shown in Table 1 and 2.
In fact, our baseline model already performs better than other existing works in Table 3 and 4.
This paper proposed an approach for semi-supervised semantic segmentation based on adversarial training. Built upon a popular segmentation network, the paper integrated adversarial loss to incorporate unlabeled examples in training. The outputs from the discriminator are interpreted as indicators for the reliability of label prediction, and used to filter-out non-reliable predictions as augmented training data from unlabeled images.  The proposed method achieved consistent improvement over existing state-of-the-art on two challenging segmentation datasets.
Although the motivation is reasonable and the results are impressive, there are some parts that need more clarification/discussion as described below.
1) Robustness of discriminator output:
The main contribution of the proposed model is exploiting the outputs from the discriminator as the confidence score maps of the predicted segmentation labels.
However, the outputs from the discriminator indicate whether its inputs are from ground-truth labels or model predictions, and may not be directly related to ‘correctness’ of the label prediction.
For instance, it may prefer per-pixel score vectors closed to one-hot encoded vectors. More thorough analysis/discussions are required to show how outputs from discriminator are correlated with the correctness of label prediction.
2) Design of discriminator
I wonder if conditional discriminator fits better for the task. i.e. D(X,P) instead of D(P). It may prevent the model generating label prediction P non-relevant to input X by adversarial training, and makes the score prediction from the discriminator more meaningful. Some ablation study or discussions would be helpful.
3) Presentations
There are several abused notations; notations for the ground-truth label P and the prediction from the generator S(X) should be clearly separated in Eq. (1) and (4).
Also, it would better to find a better notation for the outputs from D instead of D^(*,0) and D^(*,1).
Training details in semi-supervised learning would be helpful.
For instance, the proposed semi-supervised learning strategy based on Eq. (5) may be suffered by noise outputs from the discriminator in early training stages. I wonder how authors resolved the issues (e.g. training the generator and discriminator are with the labeled example first and extending it to training with unlabeled data.)
We thank the comments and address the raised questions below.
Q1. How are outputs from discriminator correlated with the correctness of label prediction?
T_semi, # of Selected Pixels (%), Average Pixel Accuracy (%)
In the above table on the  Cityscapes dataset, we show the average numbers and the average accuracy rates of the selected pixels with different values of T_semi as in (5) of the paper. With a higher T_semi, the discriminator outputs are more confident (similar to ground truth label distributions) and lead to more accurate pixel predictions.
Also, as a trade-off, the higher threshold (T_semi), the fewer pixels are selected for back-propagation. This trade-off could also be observed in Table 5 in the paper. We will add more analysis to the paper.
Q2. What’s the performance of D(X,P) compared to D(P)?
We conduct the experiment using D(X,P) instead of D(P) by concatenating the RGB channels with the class probability maps as the input to the discriminator.
However, the performance drops to 72.6% on the PASCAL dataset (baseline: 73.6%). We observe that the discriminator loss stays high during the optimizing process and could not produce meaningful gradients. One reason could be that the RGB distributions between real and fake ones are highly similar, and adding this extra input could lead to optimization difficulty for the discriminator network.
Therefore, it is reasonable to let the segmentation network consider RGB inputs for segmentation predictions, while the discriminator focuses on distinguishing label distributions. Note that, in Luc2016, similarly the discriminator structure on PASCAL does not include RGB images as inputs. We will add more results and discussions in the paper.
Q3. The notation P in (1) and (4) is not clear.
Thanks for the recommendation. We revise (1) and (4) in the paper for better presentations.
Q4. What are the training details in semi-supervised learning?
We include the details of semi-supervised training algorithm in the revised paper. As the reviewer points out, initial inputs may be noisy, and we tackle this issue by applying the semi-supervised learning after 5k iterations.
The paper presents an alternative adversarial loss function for image segmentation, and an additional loss for unlabeled images.
+ well written
+ good evaluation
+ good performance compared to prior state of art
- technical novelty
- semi-supervised loss does not yield significant improvement
- missing citations and comparisons
The paper is well written, structured, and easy to read.
The experimental section is extensive, and shows a significant improvement over prior state of the art in semi-supervised learning.
Unfortunately, it is unclear what exactly lead to this performance increase. Is it a better baseline model? Is the algorithm tuned better, or is there something fundamentally different compared to prior work (e.g. Luc 2016).
Finally, it would help if the authors could highlight their technical difference compared to prior work. The presented adversarial loss is similar to Luc 2016 and "Image-to-Image Translation with Conditional Adversarial Networks, Isola etal 2017". What is different, and why is it important?
The semi-supervised loss is similar to Pathak 2015a, it would help to highlight the difference, and show experimentally why it matters.
In summary, the authors should highlight the difference to prior work, and show why the proposed changes matter.
We thank the comments and address the raised questions below.
Q1. What is the major novelty of this work?
The novelty of this work is to incorporate adversarial learning for dense predictions under the semi-supervised setting without image synthesis. To facilitate the semi-supervised learning, we propose a fully-convolutional discriminator network that provides confident predictions spatially for training the segmentation network, thereby allowing us to better model the uncertainty of unlabeled images in the pixel level. Our model achieves improvement over the baseline model by incorporating this semi-supervised strategy.
Q2. What are the major differences between this work and Luc2016?
The major differences between our work and Luc2016 are listed below:
- We propose a unified discriminator network structure for various datasets, while Luc2016 designs one network for each dataset.
- We show that the simplest one-hot encoding of ground truth works well with adversarial learning. The “scale” encoding proposed in Luc2016 does not lead to a performance gain in our experiments.
- We propose a semi-supervised method coupled with adversarial learning using unlabeled data.
- We conduct extensive parameter analysis on both adversarial learning and semi-supervised learning, showing that our proposed method performs favorably against Luc2016 with the proper balance between supervised loss, adversarial loss, and semi-supervised loss.
Q3. Differences between this work and Pix2Pix (Isola 2017)?
Our discriminator network works on probability space, while Pix2Pix and other GAN works are on the RGB space.
In addition, the target task of Pix2Pix is image translation, and ours is semantic segmentation.
Q4. Difference between this work and constrained CNN (Pathak 2015a)?
In Constrained CNN (CCNN), the setting is weak supervision where image labels are required during training. In our work, we use completely unlabeled images in a semi-supervised setting.
Thus, the constraints used by CCNN are not applicable to our scenario where image labels are not available.
In CCNN, they design a series of linear constraints on the label maps, such as those on the segment size and foreground/background ratio, to iteratively re-train the segmentation network. Our framework is more general than CCNN in the sense that we do not impose any hand-designed constraints that need careful designs for specific datasets. Take the Cityscapes dataset as an example, the fg/bg constraint in CCNN does not work in this dataset since there is no explicit background label. The minimum segment size constraint does not make sense either, especially for thin and small objects that frequently appear in road scenes.
In contrast, we propose a discriminator with adversarial learning to automatically generate the confident maps, thereby providing useful information to train the segmentation network using unlabeled data.

Meta-review:
role: PC
data: comment: The paper presents a reasonable idea, probably an improved version of method (combination of GAN and SSL for semantic segmentation) over the existing works. Novelty is not ground-breaking (e.g., discriminator network taking only pixel-labeling predictions, application of self-training for semantic segmentation---each of this component is not highly novel by itself). It looks like a well-engineered model that manages to get a small improvement with a semi-supervised learning setting. However, given that the focus of the paper is on semi-supervised learning, the improvement from the proposed loss (L_semi) is fairly small (0.4-0.8%).

Decision: Reject