Title: C+1 Loss: Learn to Classify C Classes of Interest and the Background Class Differentially
Authors: Changhuai Chen
Xile Shen
Mengyu Ye
Yi Lu
Jun Che
Shiliang Pu
Venue: ICLR 2022
Keywords: C+1 loss
classes of interest
background class

Abstract: There is one kind of problem all around the classification area, where we want to classify C+1 classes of samples, including C semantically deterministic classes which we call classes of interest and the (C+1)th semantically undeterministic class which we call background class. In spite of most classification algorithm use softmax-based cross-entropy loss to supervise the classifier training process without differentiating the background class from the classes of interest, it is unreasonable as each of the classes of interest has its own inherent characteristics, but the background class dosen’t. We figure out that the background class should be treated differently from the classes of interest during training. Motivated by this, firstly we define the C+1 classification problem. Then, we propose three properties that a good C+1 classifier should have: basic discriminability, compactness and background margin. Based on them we define a uniform general C+1 loss, composed of three parts, driving the C+1 classifier to satisfy those properties. Finally, we instantialize a C+1 loss and experiment it in semantic segmentation, human parsing and object detection tasks. The proposed approach shows its superiority over the traditional cross-entropy loss.


Review discussion:
Strengths:
+ The employed additional loss functions improved the original methods without these additional loss terms.
Weaknesses:
- Lack of novelty. The C+1 classification is a straightforward and widely adopted setting, which is the default for segmentation task, while the paper claims that "We define the C + 1 classification problem ..."; The method itself is basically adding two additional regularization terms using the center loss, which is employed from existing methods.
- With the claim being handling C+1 classification problem, methods from the OSR literature should be compared.
- Though an improvement on overall classification performance is shown, however, no evidence is provided showing that the representation is more compact and the margin has been full filled.
- No evidence is shown to support the conclusion of equation (3). The radii of the small and large balls should be demonstrated (e.g., by feature visualization).
- The writing is unsatisfactory, with grammatical errors and typos even in the abstract.
For example, CoI is never mentioned before the short form appears; "In spite of ... use" ->  "In spite of ... using". The main context contains more issues...
NO.
Indeed the C+1 classification is a widely adopted setting, but all of them treat all classes without difference. As far as we know, we are the first to differentiate the background class from foreground classes. Almost all algorithms of segmentation and detection adopt softmax loss or its variants (such as Focal Loss) as the default loss for classification, which learn the foreground and background with no difference.
Does OSR mean Open-Set Recognition? If it does, we think we don’t need to compare with it because of their difference. In OSR, there aren’t any samples for the background class during training, but the background class should be detected during testing.
However, the C+1 classification problem include many training samples for background class. As far as we know, there are no methods which adopt OSR algorithms as their classifier in the traditional semantic segmentation and object detection tasks, i.e PASCAL VOC, PASCAL Context, LIP datasets, and COCO.
We have added a feature visualization on appendix of the newly submitted paper.
I appreciate the authors' response.
However, as agreed by the majority of reviewers, the paper lacks novelty and the quality needs significant improvement. The response did not add extra evidence to address these concerns.
Thus, I would like to keep my original rating.
I mention Open-Set-Recognition(OSR) because OSR also considers the open-set classes, which corresponds to the C+1th class in the context of this paper, while OSR methods can achieve good results even without training with any open-set/background classes.
Hence, better performance would be expected if the background training data is used, which should be validated experimentally.
However, it could also be the case that adding the extra background training data leads to worse performance because of overfitting to the trained background. In this case, it is questionable to add the extra background training data. Unfortunately, this is not validated in the paper.
* Significance. The problem addressed by the author is important (the background class is present in most of the vision benchmarks) but few work (to my knowledge) attempted to tackle it in a rigorous way.
* Novelty. The solution proposed by the authors (decompose the loss function into three terms, to enforce some constraints of the last layer feature space) is original, easy to implement and has the potential to inspire others. The properties identified by the authors (which translate into these loss terms) are well discussed and intuitive.
It might however be surprising  to maintain the basic loss (cross entropy preceded by a softmax) as such (since it probably acts against the compactness and background margin properties enforced by the two other terms).
* Results. The authors report convincing results on four different datasets and two different tasks. The experiments (ablation study) validate the claims of the introduction.
While the quantitative analysis consistently show improvement wrt to baseline or the sota, I believe that more challenging experiments could have
been performed for binary classification tasks (eg, human detection/classification, road sign detection), for which the background is often very diverse.
* Clarity. The paper is fluid, a pleasure to read.
NO.
Maybe it’s more appropriate to call the first term as separability loss, which makes all classes separable. The compactness drives the representation space of C classes of interest more compact which is beneficial for the separability of C classes of interest. The background margin drives the representation space of background far away from the C classes of interest, which enhance the separability of background and C classes of interest.
So the three loss terms are mutually beneficial, not conflicting.
In this paper, we focus on validating our idea on some general C+1 classification tasks.
Indeed, binary classification tasks are interesting. And we will consider some explorations on them in future.
The paper tackles an interesting and fair aspect in the field of multiclass classification.
However, the neither the idea nor the kind of solution are new.
In addition, the experimental results are not promising and show just slightly different results compared to arbitrarily selected baselines. From this points of view, there is neither a novel contribution nor provides the paper thrilling new insights to the tackled problem.
In addition, the paper is not written and structured very well.
For instance, the mathematical writing and the bibliography need to be seriously checked!
NO.
OCRNet and FCOS are the SOTA methods of semantic segmentation and object detection respectively, not the arbitrarily selected basleines. We select a representative method DeepLabV3+ as baseline for ablation study, and then transfer our method to different algorithms on different semantic segmentation dataset.
Furthermore, we have validate our method on different tasks, including human parsing and object detection. All of this proved the generality of our method.
Strengths
- The introduced C+1 loss is widely applicable to many kinds of tasks that require classification.
- The experiments are done extensively on multiple tasks to demonstrate the effectiveness of C+1 loss: object detection, semantic segmentation, human parsing.
Weaknesses
- C+1 loss is a simple and straightforward combination of existing loss functions already explored in previous papers `Wen et al. (2016)`. The background margin loss is almost identical to the conventional margin ranking loss. If you take things that are working really well and combine them in a simple way, it is likely that the combination will work better than any of the individual things.
- The naming of "basic discriminability" is so arbitrary and not supported by anything. Why does it have to be "basic"?
- When the paper mentions it uses center loss for L_compact, it does not cite the any papers about center loss.
- The proposed properties of C+1 loss are not backed by any theoretical foundations or insights.
- Except for DeepLabV3 on Pascal VOC, the performance gains are not so significant (most are less than 1%) in the evaluated tasks.
NO.
Indeed, “basic discriminability” is not the most appropriate term, we consider replacing it with “separability”.  Now we have cited the paper about center loss in the new submitted draft. We have revealed the necessity of the proposed properties. For OCRNet+ours on PASCAL VOC, the performance gain is about 1%. For FCOS(R50)+ours and FCOS(X101-64x4d)+ours on COCO, the performance gains are 1.2 and 1.5 for small object respectively. And small object detection is a challenging problem. Most importantly, our method doesn’t bring any more cost for inference.
As far as we know, we are the first to propose the C+1 classification problem present all around the machine learning area. And we propose to learn the C classes of interest and background class differentially by a uniform abstract C+1 loss. These are the main contributions and innovation points of us.
However, the instantialized C+1 loss is just one instance of the abstract C+1 loss, we could design any new loss accorded with our proposed properties. We have added a feature visualization on appendix of the newly submitted paper.

Meta-review:
role: PC
data: comment: This paper presents work on classification with a background class.  The reviewers appreciated the important, standard problem the paper considers.  However, concerns were raised regarding presentation, empirical evaluation, clarity, novelty, and signficance of the work.  The reviewers considered the authors' response in their subsequent discussions but felt the concerns were not adequately addressed.  Based on this feedback the paper is not yet ready for publication in ICLR.

Decision: Reject