Learning State-Based Node Representations from a Class Hierarchy for Fine-Grained Open-Set Detection
Authors: Spandan Pyakurel
Qi Yu
Venue: ICML 2025 poster (2025)
Keywords: Openset detection
hierarchical classification
Primary area: deep_learning->robustness

Abstract: Fine-Grained Openset Detection (FGOD) poses a fundamental challenge due to the similarity between the openset classes and those closed-set ones. Since real-world objects/entities tend to form a hierarchical structure, the fine-grained relationship among the closed-set classes as captured by the hierarchy could potentially improve the FGOD performance. Intuitively, the hierarchical dependency among different classes allows the model to recognize their subtle differences, which in turn makes it better at differentiating similar open-set classes even they may share the same parent. However, simply performing openset detection in a top-down fashion by building a local detector for each node may result in a poor detection performance. Our theoretical analysis also reveals that maximizing the probability of the path leading to the ground-truth leaf node also results in a sub-optimal training process. To systematically address this issue, we propose to formulate a novel state-based node representation, which constructs a state space based upon the entire hierarchical structure. We prove that the state-based representation guarantees to maximize the probability on the path leading to the ground-truth leaf node. Extensive experiments on multiple real-world hierarchical datasets clearly demonstrate the superior performance of the proposed method.

Learning State-Based Node Representations from a Class Hierarchy for Fine-Grained Open-Set Detection
Spandan Pyakurel | Qi Yu
Abstract
Bird
Openset Sperrow
Sparrow
Fine-Graid Bnee-Grained Openset Detection (FGOD) poses fundamental challenge due to the similarity be- tween the openset classes and those closed-set ones. Since real-world objects/entities tend to form hierarchical structure, the fine-grained rela- tionship among the closed-set classes as captured by the hierarchy could potentially improve the FGOD performance. Intuitivelyy the hierarchical dependency among different classes allows the model to recognize their subtle differences, which in turn makes it better at differentiating similar open-set classes even they may share the same parent. However, simply performing openset de- tection in top-down fashion by building local detector for each node may result in poor detec- tion performance. Our theoretical analysis also reveals that maximizing the probability of the path leading to the ground-truth leaf node also results in a sub-optimal training process. To systemati- cally address this issue, we propose to formulate novel state-based node representation, which constructs state space based upon the entire hier- archical structure. We prove that the state-based representation guarantees to maximize the proba- bility on the path leading to the ground-truth leaf' node. Extensive experiments on multiple real- world hierarchical datasets clearly demonstrate the superior performance of the proposed method.
Seagull
Sparrow
Callfomia
Ivory
Fox
Grasshopper Sparrow
Flnng Necked
Gull
Gull
Sparrow
Pheasant
Openset
Bird
Figure 1. With the support of a hierarchy, fine-grained openset detection assign openset data sample into different parent nodes as shown: Le Conte Sparrow + Openset Sparrow, Ring Necked Pheasant - Openset Bird
et al., 2020b; 2021; Yang al., 2020; Zhang & Patel, 2016). They typically quantify an openset score to measure the difference between the openset sample and all the closed- set classes as whole. However, the semantic similarity between openset samples with closed-set classes may vary significantly. As the similarity increases, openset detection becomes more challenging and such problems are referred to as near OOD detection (Fang et al., 2022; Ren al., 2021; Fort al., 2021). Theoretical results have also been devel- oped to strictly quantify the difficulty of these problems from the learnability perspective (Fang et al., 2022).
In many real-world settings, the fine-grained relationship between the closed-set classes can be precisely captured through a hierarchical structure. Such a hierarchy could help improve the detection performance of some openset examples despite having high semantic similarity with cer- tain closed-set classes. As shown in Figure 1, the closed-set training data consists of four types of birds, which can be fur- ther grouped into two parent classes, Seagull and Sparrow. Due to migration of birds from other regions, one may need to identify previously unseen species. Identifying the closest related species of these new birds can help expand the exist- ing hierarchy and provide insights into migration patterns. By training model that adequately captures the hierarchical dependencies among different closed-set classes, it can be better equipped to more accurately recognize openset sam- ples with different levels of similarity with the closed-set ones. For example, when being presented with a Le Conte Sparrow, the knowledge learned from the hierarchy could
1. Introduction
Openset detection aims to identify data samples that are not part of the known set (a.k.a., closed-set) of classes in the training data. Most existing methods try to assign the detected openset samples into a single openset class (Sun et al., 2020; Bendale Boult, 2016; Vaze et al., 2021; Chen
'Rochester Institute of Technology, Rochester, New York. Cor- respondence to: Qi Yu <qi.yu@rit.edu>.
Proceedings of the 4 International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning State-Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Peleconiform Seabird
Seabird
Aquatic Bird
Bird
Passeriform Bird
Oscino
Corvine
Cormorant
Pelecaniform
Openset
Crow
Seabird
American Crow
Openset
Cormorant
Hierarchical Precision: %
Hierarchical Precision: 16
Ground truth
State-FGOD
Global
Brandt Cormorant
Figure 2. Frror prediction analysis for global detector that assigns openset sample from Brandt Cormorant to totally wrong class due to lack of considering the hierarchical dependencies.
allow the model to recognize some important but subtle dif- ferences from the two known Seagull classes. Meanwhile, also more different from the two Seagull classes. As result, the model able to perform fine- fine-grained openser detection (FGOD) recognize Le Conte Sparrow openset sparrow as shown in Figure 1. Similarly, when facing a Ring Necked Pheasant, the model could recognize that it is different from both Seagull and Sparrow while pos- sessing some key characteristics of birds, and hence assigns it as an openset bird.
path probability does not guarantee to decrease the proba- bility of other nodes that are not the children of the nodes on the path, which compromises the detection performance, as evidenced by our experiments.
To avoid large error accumulation of local detectors, a global detector can be trained that maximizes the probability of the ground truth node by considering all the leaf nodes. For open-set detection, besides evaluating an open-set score, an extra child node can be added to each non-leaf node denoting its open-set child class (Leee al., 2018; Pyakurel & Yu, 2024), which transforms open-set detection as a multi- class classification problem that only considers all leaf nodes (see Figure 8 in the Appendix for an illustrative example), However, global detector trained in such a way ignores the detailed hierarchical dependencies among different classes. Our thorough theoretical analysis reveals that the global detector formulation indeed leads to a sub-optimal training process causing large hierarchical detection errors. Such errors can be more precisely captured by the hierarchical performance metrics that measure the number of correct predictions out of all predictions along the hierarchy to reach the leaf node as shown in Figure 2.
As illustrated by the example above, detecting openset sam- ples that share semantic similarity with the closed-set classes (i.e., near OOD detection) can be achieved by conducting FGOD when the closed-set classes can be organized into a hierarchical structure. Such hierarchical structures com- monly exist in real-world settings as evidenced by many benchmark datasets (e.g., Imagenet and Cifarl (0). To achieve fine-gra une-grained detection, an openset sample should be assigned the right parent node in the hierarchy based on its difference to classes at different levels. Following the hierarchy, FGOD can be conveniently conducted in a top-down fashion by building a local detector at each non- leaf node. Each detector performs the classification of child nodes under the non-leaf node while quantifying an openset score. If detector obtains a high openset score, an openset sample is detected. Otherwise, one child node is selected based on the classification result and the process continues unless a leaf node is reached.
To address the fundamental limitations of both local and global detectors for FGOD, we propose to learn novel state-based node representations from a hierar- chy of closed-set classes to advance the frontiers of fine-grained open-set detection. Through the formu- lation of legal states, we can avoid eтror accumula- tion like in local detectors while effectively capturing fine-grained hierarchical dependencies between different nodes, including (parent - chila), (ancestor + des cendant), and (sibling - sibling). Such a formulation ensures that the probabilities assigned each node (both leaf and non-leaf) strictly follow their hierarchi- cal dependencies with theoretical guarantees. This implies that node probabilities are derived from a global view of the entire hierarchy. It is worth noting that there are a few exist- ing works, such as (Deng et al., 2014; Chen et al., 2022), that perform state formulation, but they are designed for hierarchical classification instead of fine-grained openset
However, using a sequence of local detectors in top-down fashion may result in a poor FGOD performance. Each detector is trained by only considering its own child node distribution, which fails fully utilize fine- ine-grained depen- dencies from nodes at lower levels those located other parts of the hierarchy. This usually leads to local detection decisions that are sub-optimal. To train more robust detec- tors, one can collectively consider all the ancestor nodes on the path to the ground-truth leaf node. By choosing the path with the highest overall probability, the model can more ef- fectively avoid poor local decisions. Nevertheless, a deeper analysis reveals that solely maximizing the (ground-truth)
2

---

Learning State-Basch Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
detection. Our novel theoretical analysis shows that the use of state-based representations in model training guar- antees to decrease the probabilities of all non-ground truth nodes while increasing the ground-trudh truth node's 's probability. Since these non-ground truth nodes include both leaf and non-leaf nodes, the prediction of the proposed method, even when wrong is close to the ground truth node in the hierar- chy as shown in Figure 2, resulting a higher hierarchical precision. Our contribution is threefold:
Hierarchical classification. Based on how the work lever- ages the hierarchical structure of data, it can be divided into two types: (i) modify network architecture, and (ii) Constraint on the loss function. First line of work (Cerri et al., 2014; Chen et al., 2020a; Peng et al., 2018; Du et al., 2020; Zhao et al., 2021; Chen et al., 2019) modifies the network architecture to adapt the hierarchical structure of data. Among the state-of-the-ar Zhao et al. (Zhao et al., 2021) uses higher-order attention to discover relations in the form of graph of embeddings. Similarly, Du et al. (Du et al., 2020) adopts progressive training and jigsaw puzzles to focus on both fine-grained and coarse-grained features required to identify hierarchical features. Graph-Neural- Network is also leveraged to construct a graph between class and attributes (Chen et al., 2019). Another line of work (Giunchiglia & Lukasiewicz, 2020; Deng et al., 2014; Chen et al., 2022; Chang et al., 2021) imposes constraints on the loss function to ensure hierarchical relations are being followed. The goal of the work in this setting is to improve the performance of hierarchical classes seen in the training. ProTeCt (Wu et al., 2024) proposes a tree-cut loss function for taxonomic openset detection. This work deals with iden- tifying samples different granularity in the hierarchy, not the samples from the open- world. Li et al. leverages the hier- archical relationship to perform hierarchical classification at the pixel level. Different from ours, this work does not con- sider the open-world setting. To handle fine-grained openset detection, our method ensures hicrarchical constraints while performing openset detection in the hierarchy.
a thorough theoretical analysis that unveals fundamental limitations of local and global detectors when performing open-set detection in a hierarchical structure,
formulation of novel state-based node representations from hierarchy of closed-set classes along with a FGOD process that avoids error accumulation in open-set detec- tion while effectively capturing fine-gn theeggrined hierarchical dependencies between different nodes,
a formal analysis of the proposed state-FGOD model that ensures an optimal training process and guarantees to improve the hierarchical detection performance.
Extensive experimentation conducted on multiple real-world hierarchical datasets demonstrates the superior performance of the proposed method.
2. Related Works
Openset detection. Openset detection usually rely on a score to differentiate openset samples from closed-set classes. Openmax (Bendale & Boult, 2016) redistributes the closed-set probabilities and assigns a probability as a score to the openset class. Some works utilize the maximum softmax probability from the classification of closed-set classes for detection (Hendrycks & Gimpel, 2016; Vaze et al., 2021). Vaze et al. (Vaze et al., 2021) replace the probability with logit and utilize the maximum logit score for openset detection. Some works (Chen et al., 2020b; 2021; Yang et al., 2020) focus on training prototypes of closed-set classes in the embedding layer and learning a representation of openset class far from all the prototypes. Based on the distance of openset sample from the closed-set set prototypes, detection is performed. A line of research quan- tifies uncertainty measure to identify openset samples based on evidential uncertainty (Sensoy et al., 2018; Malinin & Gales, 2018: Charpentier et al., 2020), free energy (Liu et al., 2020; Wang al., 2021; Du et al., 2021). Few works (Fang et al., 2022; Ren et al., 2021; Fort al., 2021; Lang et al., 2024; Sun et al., 2023; Fu et al., 2023) particularly focus on near-OOD detection, where openset samples are semantically similar to the closed-set classes. Our work extends beyond near-OOD detection, which not only detects a near-OOD sample but also identifies its closest parent in the hierarchy of the closed-set classes.
Fine-grained openset detection. Fine-grained openset de- tection is relatively under-explored in the literature. There are two ways of dealing the problem: (i) Local Detector and (ii) Global Detector. For the former, a local openset detector can be applied each non-leaf class of the closed- set hierarchy. For each local openset detector, (Lee et al., 2018) 8) quantifies the confidence score based on KL diver- gence between the model and uniform probability distribu- tion and (Wang et al., 2022) quantifies uncertainty based on fuzzy logic. The global detector trains a single classifier to identify the sample between closed-set set and openset classes. Several strategies are proposed to leverage samples from closed-set classes as openset ones (Lee et al., 2018) 8). (Ruiz & Serrat, 2022) trains prototypes of closed and open set classes using a cosine loss. E-HND (Pyakurel & Yu, 2024) leverages an evidential loss to further separate closed and open-set classes by allocating fine-grained evidences. In this work, we address the fundamental limitations of both local and global detectors as outlined in the introduction by learning novel state-based node representations from class hierarchy to advance fine-grained openset detection. An- other line of work, Generalized Category Discovery (GCD) (Rastegar et al., 2023) targets a related problem similar to FGOD of assigning openset samples to a specific category. FGOD leverages an existing hierarchical structure and de-
3

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
fines openset class for every non-leaf class in the hierarchy so that openset test samples can be assigned to those openset classes. GCD, on the other hand, does not leverage existing hierarchy but learns to categorize the classes into a hierar- chical structure. When a hierarchy is not present for the dataset, GCD approaches can be applied to obtain the hier- archy before the FGOD methods are applied. In that sense, GCD and HND are complementary to each other.
detector performs classification of child nodes while quan- tifying an openset score. If the score is higher than a pre- determined threshold, openset is detected. Otherwise, the sample is classified as one of the child nodes. The process starts from the root node and stops if openset is detected or reaches the leaf node of the hierarchy. For model training, we define known D and openset Dú data samples for each classifier lc. If a training sample belongs descendant classes of non-leaf node lc, it is assigned to Dic- otherwise to Di:- Using Di.- a loss function (eg, cross-entropy) can be defined to guide the model to correctly classify its child nodes. Some regularization term is usually included to en- sure a high openset score for samples from Dic- Additional details and an algorithm summarizing the detection process is presented in the Appendix.
3. Methodology
We first formally define the problem of fine-grained openset detection (FGOD) with hierarchy of closed-set classes. We present a standard process that builds a local detector on each node and performs openset detection along with the hierarchy. We provide novel theoretical insight on why it leads to a sub-optimal training process even considering all the ancestor nodes on the path leading the ground-truth leaf node. Such theoretical insight inspires the design of the statebbsed node representations from hierarchy that guarantees an optimal FGOD training process.
0.4
0.6
0.9
0.1
0.51
0.49
3.1. Problem Formulation
Given training dataset:
Durain (Iize Un J 1- data sam- ple In belongs to one of the K classes yn y {1, K3. For the test dataset Diea = {I'n, Un > the test label yhe may or may not belong to train label set V. In general openset detection, the goal is to identify whether a test sample is from the closed-set of classes and if so, assigns it into the correct class. It can be formally defined as f In'' + DU(K" + 13, where label K + refers to the openset class.
0.36
0.04
0.306
0.294
Figure 3. Local detector augmented with path probability
While using sequence of local detectors offers an intuitive way to tackle FGOD, the final prediction depends on mul- tiple classifiers. Each classifier makes a decision based on the local view only. If single classifier makes a mistake, no matter how well other classifiers perform, the final pre- diction will be wrong. Thus, it fails to take advantage of fine-grained views from the hierarchy, As shown in Figure 3 for a test sample with ground-trute truth node ④ First, node 3 is predicted, instead of node 2, and then node 6 is predicted as the final prediction. However, it fails to examine that nodes ④ and ( have a close to uniform distribution, while node 4 dominates the probability over node 3 If we could leverage fine- fnnegrriied probabilities from all the descendants, this issue could be mitigated. An intuitive solution is to augment the local detector method by considering all the ancestor nodes of the ground truth leaf node and maximize the the probability of the entire path. Specifically, for each leaf node in the hierarchy, we define a path from the root node to the Icaf node. For each path, we calculate path prob- ability by multiplying the local probabilities (conditioned the parent node) of all the nodes that lie in the path.
Openset detection becomes more challenging when the the semantic similarity between openset samples and closed- set classes increases. By leveraging the hierarchy obtained from the classes associated with labels V/ we formulate the problem of Fine-Graide Grained Openset Detection (FGOD) with hierarchy. For openset test samples, the goal of FGOD is to identify the node at which the sample becomes openset. To formally define the problem, we first introduce few notations. For a node in hierarchy H. we refer to its parents, children, descendants, ancestors, and siblings as P(y). D(w), A(y) and (v)).. The hierarchy contains leaf nodes ((H) and lon-leaf nodes Ni Next, we assume that each non-leaf node y has extra child Ovy), denoting the openset child, along with its closed-set children C(y). The set of all the openset nodes from the hierarchy is O(H). Formally, the goal of FGOD can be defined as h: x. - y O
Ay E C(H), ph(y)-
II p(a/P(a)) aEA())
(I)
3.2. Why is a Local Detector Sub-optimal for FGOD with a Hierarchy?
For each classifier, the local probability is obtained by the softmax operation on the logits of its child nodes. Therefore. the path probabilities are based on logits of only those nodes, whose parent classifiers are in the path. For example, for the training sample from node ① in Figure 3, only logits
By leveraging the hierarchy of closed-set classes, a local openset detector can be built on each lea- node. Each
4

---

Learning State-Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
optimization longer considers local view only, and guar- antees an update of all the parameters in the right direction. We denote H'' as an augmented hierarchy H. with openset nodes (HH) (see Figure 4 for an example). The leaf and non-leaf nodes of H' becomes L(H') = L(H) U (HH) and Ni(H') =N(()))
Subsumption Edge
Exclusion Edge
Seagull
Sparrow
Openset
5
10
The hierarchy 'H' can be represented as a graph G (V, E.) consisting of nodes V = {ui, u2,.,.urr, where M = I + (iH') The relationship between two nodes uy and vi can be defined as an edge (vi, Ui), where each node is assigned the value of 1 or 0 to denote the node is either active or inactive in the edge. There are two types of edges in the graph: subs sobbsumpiion edges E, CV x V and exc 1 usi on edges F.e C V V. A subsumption edge (U,, 12) E E, is a directed edge that refers to node iy being parent of ey. Similarly, an exclusion edge (v, v2) E E is undirected edge that refers to nodes v4 and vy being mutually exclusion to each other. The graphical nodes for the hierarchy of Bird along with subsumption and exclusion edges are illustrated in Figure 4. First, we define the legal assignment of the values for each edge, referred to as local legal states. Next, we define the legal assignment of the val- ues to all the nodes, referred to as global legal states. The legal assignment for the Eg is (1, (1, (1, (0, 0), mean- ing when the child node is active, the parent node has to be active. Similarly, the legal assignment for the E, is (0, 1). (0, 0), (1, O), meaning both of the nodes can not be active. The values except the legal assignments are illegal. With the definition of local legal states, we define a global legal state below.
Openset Seagull
California Gull
Ivory
Grasshopper Sparrow
Openset Sparrow
Gull
Sparrow
Figure 4. An example of open-set augmented hierarchy: Blue edge
represents subsumption; Green edge represents exclusion.
U1
U2
V/
U4
Ug
U6
U/
Ug
Ug
U1O
0
0
0
0
0
0
80
1
0
0
0
0
0
0
0
84
1
1
0
0
1
0
0
0
0
0
85
1
1
0
0
0
1
0
0
0
0
$6
1
1
0
1
0
0
$7
1
0
1
0
0
0
0
1
0
0
88
1
0
1
0
0
0
0
0
1
0
89
1
0
0
0
1
810
Figure 5. An example of global states
from nodes 2. 3, 4 and 5 are used in path probability calculation. Logits from the rest of the nodes are not used, making the optimization from path probability regulariza- tion sub-optimal, as shown in the following theorem.
Theorem 3.1. (Sub-optimality of FGOD with path prob- ability) Given a hierarchy H and a training sample (x. y)- Aly) is the set of nodes belonging to the path from root to leaf node y. Then, the cross-entropy loss function based on path probability does not guarantee decrease in logits of node y' & C(a), Va E A(y).
Definition 3.2 (Global Legal State). Given a graph G =
(V, Eg, Ee), a global legal state s E {0, 13M assigns val- ues to all the nodes V such that the following three condi-
tions are satisfied: C₁: V(E, 0z) E., (v%, U₁) * (0, 1). Cz: V(a, U2) E Ee, vy) # (1, and C2: (Eu L(H'), = 1) V (V),u=0).
Proof. (Proof Sketch) We provide the detailed proof in Ap- pendix C. First, we define path probability for ground truth label y as a function of logits of the nodes in hierarchy, then evaluate the loss based on the probability of path pv)). Next, we calculate the gradient of the loss function for
Remark. For each state, C3 ensures that one of the Icaf nodes is active or none of the nodes is active. Since we include openset nodes ((H) as leaf nodes of H', without the condition (A E L(H'). it will result in invalid global states. We also need to allow a state where the openset sample lies outside the entire hierarchy, which is given by Ye, = O. All the legal global states can be represented in the form of a matrix, defined as global state matrix S. The resulting dimension of S is (IC(H)) + 1) x M. An example of the legal global states is presented in Figure 5, where nodes and states follow Figure 4. It is important to note that each state (except for su) corresponds to leaf node in open-set augmented hierarchy H', so performing classification on the states is equivalent to conducting the multi-class classification as performed in the global detec- tors. More importantly, each state captures the important fine-grained dependencies among the ancestor nodes that
dg= "h(ylzzj and non-ground truth logits Iv' " We observe that gradient By dependent on logits of children of its parent only (local optimization). Similarly, for non-ground truth label g & C(a), Va E A(y). the value of gradient Uy' = O. There is update for those non-ground truth logits. Therefore, there is no guarantee of a decrease in logits of node y' & C(a), E A(y).
ground truth logit 9y
3.3. Learning State-Based Node Representations for Efffettive FGOD
To solve the problem of sub-optimality of FGOD using local detectors, it is essential to leverage all important hierarchi-
cal dependencies among the nodes so that the probability
5

---

Learning State-Based Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Vision Transformer
Node Probabilities
Model Training
Head
State Probabitties
Input
8
State Representation
3
Prompts
Figure Conceptual diagram of model training State-FGOD
lead to the corresponding leaf node. If we look state $6. it corresponds to assigning sample to California Gull, where nodes 222 and are active and all the other nodes in the hierarchy are inactive. Those active nodes are ancestors of California Gull, Further, inactive nodes are siblings of either California Gull or its ancestors. Similar interpretations can be made about all the other states, where the index of the state corresponds to the node index from Figure 4 and the red color signifies openset cases.
With the complete knowledge of children of node Una the equality holds. Otherwise, the inequality indicates the pres- ence of an openset node associated with Un- In our formu- lation, we support the presence of an extra openset node.
Model training and inference. We capture the hierarchi- cal relationship between all the nodes through state represen- tations. Next, we train state-based detector that classifies all the leaf nodes in the hierarchy H'. We remove one node Um at a time. This creates a dynamic hierarchy H.' Um- When om is removed, all the samples that belong to tm will be treated as openset samples with respect H' Um- Since the openset samples can occur at different levels or positions within the class hierarchy, this process allows the model to learn to detect diverse types of openset samples, de- pending their specific location within the hierarchy. It is designed to address the unique requirement for fine- -ee-grained open-set detection. For a training input x with ground truth y. Um = or U/ E A(y), then the ground truth label is changed to the openset class as y = O(P(m.)) Other- wise, the ground truth label is not changed, and the sample belongs to closed-set classes. We train single model pa- rameterized by 0. For each removed node Um and training sample (Ir, y). the training loss maximizes ground truth node probability p(y (r. 0, H' Un)) The loss function is:
Evaluating the state and node probabilities. Consider a network h parameterized with 0. We can obtain the logit from for all the nodes V E G represented as hv1 Ja;; (),),,... hh lx; f), (),... |x; (z; f). First, we calcu- late an un-normalized probability for each of the state by multiplying the exponent of logit from all the nodes that active in state
p(s|x)= II (P (x:00) #1))
where ((s..=1[s) is an indicator function that takes value of 1 when Um = 1. The un-normalized state probabilities p is of dimension [C(''') + which can be normalized
Z Σ (()2)), p())2) (()(2) Z
SES
We can further obtain node probabilities p(um (r)) by sum- ming all the state probabilities where the node Um is active:
L = E
E Fim EV (v.y)
In plular, 0, H' vm))
(6)
The conceptual diagram of the proposed method along with model training is illustrated in Figure 6. The inference process using the proposed State-FGOD is summarized in Algorithm 2 of the Appendix. Next, we show that mini- mization of the cross entropy loss function using state-based formulation leads to optimal model training in Theorem 3.4.
p(um (x) Σ (()
SES
Proposition 3.3. For any node Um- the parent-child relation satisfies the probability constraint:
Σ p(clæ),
Pum Ix
Theorem 3.4. (Optimal model training from state-based node representations) Given a hierarchy H' and a training sample (x. y), the ground truth logits are 2 8), Va A(y) and the non- non-ground truth logits are heaa la; 0), Val &
cEC(mm)
Pomla p(c|z) + p(O(mm)))))
(5)
(EC(Um)
6

---

Learning Maaee-Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
A(y). Then, the cross-entropy loss function based on state probability guarantees the increase in value of ground truth logits and decrease in the value of non-ground truth logits.
fine-tuning. Additionally, we conduct an ablation study to evaluate different backhone architectures and also compare different fine-tuning methods.
Baselines. There are two categories of methods that can be adapted for FGOD: (i) Local detector and (ii) Global detector methods. For local detector, the following base- lines can be leveraged: top-down (TD), maximum softmax probability (MSP) and hierarchical classification with extra Node (HC-EN). For global detector, the following baselines are compared: Dual Accuracy Reward Trade-off Search (DARTS) (Deng et al., 2012), Relabel (Lee et al., 2018), Srave-One-Out (LOO) (Lee al., 2018), Evidential learn- ing (Sensoy et al., 2018; Pyakurel & Yu, 2024), TD+LOO (Lee et al., 2018) and E-HND (Pyakurel & Yu, 2024). The details are presented in Appendix.
Proof. (Proof Sketch) First, we define cross-entropy loss C based on node probability p(y]z). Next, we calculate the gradient of the loss function for the ground truth logit dc
de 9y = ah(ylz)> and non-ground truth logits QV' = Jhly')a) (x) We observe that gradient By is dependent on logits of all nodes (global view). When logit of non-ground truth in- creases, the gradient magnitude increases, updating the model parameters such that it increases the logit of ground truth node. Similarly, for non non-ground truth label y', the gra- dient By updates the model parameters such that decreases the value of the non-ground truth logit hyy'(z) [z:). Therefore, this guarantees decrease in logits of non-ground truth node y'. The detailed proof is in Appendix C.
4.1. Comparison Settings and Evaluation Metries
We compare the performance of the proposed method with the baselines using top-1 accuracy. For an openset sam- ple { Vvr) with prediction In. it is correct if Un ((P(wx)) The number of correct predictions out of closed- set samples is called Closed-set Accuracy (CA), and openset samples are Openset Accuracy (OA). For global detector methods, we can adjust the bias to the logits of the leaf and non-leaf nodes to obtain different CA and OA. We present the comparison of global detector methods with AUC and OA 50% CA as OA@50.
Along with the proofs, we have further illustrated the con- cept of gradient analysis of Theorems 3.1 and 3.4 using the representative example from Figure 4 in Appendix C.3. As pointed out in the introduction, building a global detector by performing multi-class classification over all the Icaf nodes in Ht' may lead to a high hierarchical detection error. The following Corollary provides the root cause for that, which explains why a global detector is sub-optimal.
Corollary 3.5 (Sub-optimality of global detectors). Given hierarchy H' and training sample (x, y). the ground truth labels are a E A(y) and the non- goo-ground truth labels are a' & A(y)- The cross-entropy loss based on probability from only leaf labels, y and sf & A(y) A sf E L(H') does not increase the value of non-leaf ground truth logits.
4.2. Fine-Grained Openset Detection Results
We present the openset accuracy and AUC results in Table 1. The comparison reveals the superior performance of our method. We observe that the global detector-based meth- ods, in general, have better performance than local detector- based methods. In Tiny Imagenet, the performance gap is more prominent the dataset has 12 levels in the hierar- chy, and the error accumulation from the detectors becomes more severe. Among local detector methods, the elimination of setting thresholds on openset score improves the perfor- mance, as given by HC-EN baseline for the Tiny Imagenet dataset. For other datasets, local detector methods based on different openset score quantification methods, MSP and KL divergence score in TD method, also show competitive performance. This suggests that the performance of the local detector- based method is impacted by the number of hierarchy levels, quality of openset scores, and thresholds on the openset scores. In general, due to the decision of local classifiers. the local detector methods achieve lower FGOD performance in comparison to global detector methods. For global detector methods, the proposed State-FGOD consis- tently outperforms other methods on all the datasets (im- provement in AUC of CUB:+5.45, Tiny Imagenet:+2245 and AWA2: (WA2:++2.0 +2.82 w.r.t. the berf-performing baseline). This is further demonstrated by AUC curve Figure 7. Among the baselines, LOO, and E-HND methods demonstrate competi-
Existing global detector methods do not fully leverage fine-grained hierarchical dependencies to train the model, and hence non-leaf logits are not optimized for fine- grained openset detection. As a result, these methods may assign the openset sample to the incorrect parent node, lead- ing to lower hierarchical precision as shown in Figure 2.
Remarks.
4. Experiments
In this section, we first present the datasets and baselines used in the experiments. Next, we discuss the comparison settings and evaluation metries. We then show the FGOD results and ablation studies to justify the proposed method.
Datasets and implementation details We use the fol- lowing world hicrarchical datasets for the experiments. (i) Tiny Imagenet (Le & Yang, 2015): a subset of Ima- geNet, (ii) CUB 200-2200 (Welinder et al., 2010): images of fine-grained species of bird (iii) Animals With Attributes 2 (AWA2): images of animals (Lampert et al., 2014). Details for each dataset are presented in Appendix D. For both the compared algorithms and the proposed method, we experi- ment with ViT-B/16 using prompt- based parameter-efficient
7

---

Learning State-Based Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Table 1. Comparison with Local and Global Detectors
CUB
Tiny Imagenet
AWA2
Method
CA/OA
HMean/AUC
CA/OA HMean/AUC
CA/OA
HMean/AUC
(a) Local Detector Methods
TD
72.47/18.75
29.79/
50.43/9.20 62.68 77.52
15.56/
80.42/33.83
47.62/
MSP
72.62
36.33/-
13.42/-
89.12/30.
45.19/
HC-EN
78.26/8.30
15.01/-
48.83 11.56
18.69/-
89.77/26.00
40.41/
(b) Global Detector Methods
DARTS
50.00/46.74
48.32/41.82
50.00/17.15
25.54/12.75
50.00/38.
33.36/34.29
Relabel
50.00/50.80 50.95
50.47/41.33
50.00/22.02
30.58/ 17.58
50.00/30
30.00
59.52/32.68
Evidential
50.00/43.. /4.34
06.43/33.44
50.00/19.35
27.90/14.53
50.00/46
/4.99
48.42/30 2/22//6.3
LOO
50.00/48. /8.34
49.15/38.
50.00/21.53
30.10/17.26
50.00/
48.83
49.40/
49.40/45.31
/45.31
TD+LO0
50.00/44.17
46.90/32.77
50.00/ 19.37
27.92/ 14.87
50.00/35
/300//35.622
41.35/25
/20.40
E-HND
00.00//4.43
52.12/ /45.59
50.00/21.45
30.02/17.33
11.31
50.00/47.05
48.47/41.66 /1.69
48.47/
State-FGOD
50.00/64.47
56.31/51.04
50.00/24.71
33.07/ 20.03
50.00/ 53.32
51.60 /48.13
(a) CUB
(b) Tiny Imagenet
(c) AWA2
Openset Class Accuracy
Openset Class Accuracy
Openset Class Accuracy
0.2
0.4
0.5
0.4
0.3
0.3
LOG
E-HND
0.1
E-HND
0.2
E-HIND
0.2
TD+LO0
TD+LO0
TD+L00
0.1
0.1
Evidential
state-FGDD
state-FGDD
state-FGon
0.0
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.0
0.2
0.4
0.6
0.8
1.0
Closed-set Class Accuracy
Closed-set Class Accuracy
Closed-set Class Accuracy
Figure 7. Closed-set accuracy vs Openset accuracy curves
Table 2. Comparison using hierarchical precision and recall
Hierarchical Precision
Hierarchical Recall
HarmonicMean
Method
Closed-set
Openset
Closed-set
Openset
Closed-set
Openset
TD
96.88
90.36
84.69
87.05
90.37
88.67
LOO
95.96
83.86
87.22
92.31
91.38
87.88
TD+LOO
94.67
84.75
85.03
86.43
89.59
85.58
E-HND
97.23
85.83
86.36
92.07
91.47
88.84
State-FGOD
98.06
90.68
87.40
92.81
92.42
91.73
tive performance.
from the hierarchy. For sample (æ, y), if the prediction is ÿ. the hierarchical metrics are defined as:
Hierarchical performance measure. We conduct this study to understand the performance of our method w.r.t hierarchical performance measure. We use the CUB dataset with hierarchical metrics: hierarchical precision, hierarchi- cal recall, and harmonic mean. Apart from leaf-level ground truth, hierarchical metrics also measure whether the an- cestors of the ground truth leaf node are being correctly predicted. Hi Hierarcht cal Precis a sion (HP) mea- sures the number of nodes correctly predicted by the model out of all the nodes predicted in the hierarchy. Similarly, Hierarchiss icl Recall (HR) measures the number of nodes correctly predicted out of the ground truth nodes
Common Ancestors(y, ()
HP
(7)
(Ancestors
Common Ancestors ()
HR
(8)
|Ancestors())
The comparison table using hierarchical metrics is presented in Table 2. Along with our method, we compare perfor- mance of TD method (a representative method from local detectors) and several representative methods from global detectors. As can be seen, local detector has poor openset recall. The error accumulation of local detector affects the
8

---

Learning Mate--ased Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Table Impact state-based representation
Table Impact path probability
Method
SP?
@A@50
AUC
PP?
TD
MSP
HC-EN
46.74
41.82
X
72.47
72.62
78.26
DARTS
CA
V
57.02, 10.228)
46.71 (6.71(14.80)
V
(2.63(+0.16)
(2.17(-0.45)
(7.58(-0.68)
Relabel
50.95
41.35
OA
X
18.75
24.23
8.30
/
59.80
45. 45.71 (44.3))
(44.36)
V
21. 21.16(+2.41)
28. (+1.0)
10.05
LOO
55.31 55..9 (+6.97)
48.34
46.23
38.45
HMean
X
(2.77(+2.98)
29.79
40.62 (44429)
36.33
17.79(+2.78)
15.01
recall performance, as the ground truth node lies in leaf nodes of hierarchy. On the other hand, global detector doesn't fully utilize the fine-grained relationships in the hi- erarchy resulting in poor openset precision. The proposed State-FGOD is able to avoid error accumulation while ef- fectively capturing important dependencies among different nodes in hierarchy, which achieves good balance on both hierarchical metries for closed-set and openset classes.
space, which helps learn hierarchical relationships between classes in the input space.
Impact of com- ponent: The conceptual di- w/o Prompt
Table Impact of component
Method
OA @5@50
AUC
55.31
46,23
agram of the proposed method is presented in Figure 6. To better learn the hierarchical relationships, we have two components in the proposed method: (i)
w/o State
48.34
38.45
State-FGOD
64.47
51.04
Top-2 Accuracy. Ta-
Table 3. Top-2 Accuracies
ble 3 compares repre- sentative global detec- tors using top-2 accu- racy on the CUB dataset. The performance im- provement shows the effectiveness of the proposed method. By leveraging the relationships among all the nodes, it ef- fectively improves the performance of FGOD.
Method
OA@50
AUC
LOO
70.08
62.49
TD+LOO
57. .4
48.39
Prompt Tuning (ii) State representation. We study the impact of each component in CUB dataset. We present the results in Table 7. As we remove each component, we see a significant drop in the performance, justifying the importance of the components.
EHND
84.58
74.08
State-FGOD
87.24
76.72
4.3. Ablation Studies
Impact of path probability. In our methodology, we dis- cuss how local detector methods are not able to leverage the fine-grained probability information make decisions leading error accumulation. We use path probability identify the path in hierarchy instead of all the classifier's predictions. For each local detector method, we compare the performance with and without path probability for CUB dataset in Table 6. With the use of path probability, we see improvement of both closed-set and openset performance in most of the methods. This study shows the importance of utilization of fine- fine-grained probabilities for FGOD.
Impact state-representtttin in global detectors. We study how the state formulation impacts the global detec- methods. We apply state formulation representative methods: DARTS, Relabel, and LOO CUB dataset and present our results in Table 4. Using state formulation in global detector methods results in significant performance improvement. This study further highlights the importance of the utilization of hierarchy relationships between nodes.
Impact of back-
Table 5. Impact of backhone
bone. We study the impact of dif- ferent backbones to the proposed method. We experiment with CUB dataset and architectures: ResNet101, 15 (He al., 2016), and ViT-B/16. For ResNet-based methods, we use linear layer on top of Resnet features for fine-tuning. For VIT, we leverage popular parameter-efficien finettuiing methods: prompt (Jia et al., 2022), bias (Cai et al., 2020), and adapter (Rebuffi et al., 2017). The results are presented in Table 5. Among different settings, ViT-B/16 fine-tuned with prompt, leads to superior performance. Prompt fine-tuning allows training learnable parameters in the input
Backbone @5
AUC
5. Conclusion
ResNet101
45.63
34.39
ResNet152
44.34
33.73
Openset detection poses significant challenge when the samples are semantically similar to the closed-set classes. In this work, we leverage the hicrarchical structure of the closed-set classes to detect the openset samples in a fine- grained manner. We formulate legal states to fully capture the hierarchical relationship between all the nodes in the hierarchy that allows us optimize the performance of recognizing openset samples different levels in the hier- archy. Further, we perform theoretical analysis to show the fundamental advantage of the proposed method. Our ap- proach exhibits superior performance, as validated through extensive experimentation on various real-world hicrarchi- cal datasets.
ViT-B/16 + Adapter
59.58
47.12
ViT-B/16 + Bias
58.45
47.72
VIT-B/16 Prompt
64.47
51.04

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Impact Statement
Deng, J., Ding, N., Jia, Y., Frome, A., Murphy, K., Bengio, S., Li, Y., Neven, H., and Adam, H. Large-scale object classification using label relation graphs. In Computer Mision-ECCV 2014: 13th Luropean Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 48-64. Springer, 2014.
In this work, we leverage the hierarchical relationships to perform fine-grained openset detection. Such hierarchical relations are useful by providing more information about the openset data samples that leads to informed decision. The proposed approach can be potentially applied in many fields like wildlife monitoring, studying the nature of malware attacks, and identifying the parent class of traffic signs for self-driving cars.
Du, R., Chang, D., Bhunia, A. K., Xie, J., Ma, Z., Song, Y.ZI,, and Guo, J. Fine-grained visual classification via progressive multi- granularity training of jigsaw patches. In European Conference on Computer Vision. pp. 153-168. Springer, 2020.
Du, X., Wang. 7., Cai, M., and Li, Y. Vos: Learning what you don't 't know by virtual outlier synthesis. In International Conference on Learning Representations, 2021.
References
Fang, Z., Li, Y., Lu, J.. Dong. J.. Han, B., and Liu, F. Is out-of- distribution detection learnable? Advances in Neural Informa- tion Processing Systems, 35:37199-37213, 2022.
Bendale, A. and Boult, T. E. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Fort, S., Ren, J., and Lakshminarayanan, B. Exploring the limits of out-of-distributi detection. Advances in Neural Information Processing Systems, 34:7068-7081, 2021.
Cai, H., Gan, C., Zhu, L.., and Han, S. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33:1 1285-112222 1297, 2020.
Fu, Y., Liu, Z., and Zhang. Z. Progressive learning vision trans- former for open set recognition of fine- fine-grained objects in remote sensing images. JEEE Transactions on Geoscience and Remote Sensing, 2023.
Cerri, R.. Barros, R. C., and De Carvalho, A. C. Hierarchical multi-label classification using local neural networks. Journal of Computer and System Sciences, 80(1):39-56, 2014.
Chang, D., Pang. K.. Zheng, Y., Ma. Z., Song. Y-Zz,, and Guo, J. Your" flamingo" is my" bird": Fine-grained, not. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11476- 11485, 2021.
Giunchiglia, E. and Lukasiewicz, Coherent hierarchical multi- label classification networks. Advances in neural information processing systems, 33:9662-9677 2020.
He, K., Zhang, X., Ren. S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Charpentier, B., Zügner, D.. and Günnemann, S. Posterior network: Uncertainty estimation without ood samples via density- based pseudo-counts. Advances in Neural Information Processing Systems, 33:1333311111 1367, 2020.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclas- sified and out- dfittrrdiittiuttion examples in neural networks. In International Conference on Learning Representations, 2016.
Chen, B., Huang. X., Xiao, L.. Cai. Z., and Jing, L. Hyperbolic interaction model for hierarchical multi-label classification. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7496-7503, 2020a.
Jia, M., Tang. L.., Chen, B.-C., Cardie, C., Belongie, S., Hariha- ran, B., and Lim, S.-N. Visual prompt tuning. In European Conference on Computer Vision. pp. 709-727. Springer, 2022.
Chen, G., Qiao, L., Shi. Y., Peng, P., Li, Huang. T., Pu. S., and Tian, Y. learning open set network with discriminative recip- rocal points. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt III 16. pp. 507-522. Springer. 2020b.
Lampert, C. H., Nickisch, H., and Harmeling. S. Attribute-bssed classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36 (3):453 465, 2014. doi: 10. 1110//TPMM.200
Chen, G., Peng, P., Wang, X., and Tian, Y. Adversarial reciprocal points learning for open set recognition. IEFE Transactions on Puttern Analysis and Machine Intelligence, 44(11))80658800 1, 2021.
Lang, N., Snæbjarnarson, V., Cole, E., Mac Aodha, O., Igel, C., and Belongie, S. From coarse to fine-grained open-set recogni- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17804 17814, 2024.
Chen, J., Wang, P., Liu, J., and Qian, Y. Label relation graphs enhanced hierarchical residual network for hierarchical multi- granularity classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4858-4867, 2022.
Le. Y. and Yang. X. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Lee, K., Lee, K., Min, K., Zhang, Y., Shin, J., and Lee, H. Hi- crarchical novelty detection for visual object recognition. In Proceedings the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1034-1042, 2018
Chen, T., Xu, M., Hui, X., Wu, H., and Lin, L. Learning semantic- specific graph representation for multi- label image recognition. In Proceedings of the IEFF/CVF international conference computer vision, pp. 522-531, 2019.
Liu, W., Wang. X., Owens, J., and Li, Y. Energy-based based out-of- distribution detection. Advances in neural information process- ing systems, 33:21464 2145, 2020.
Deng, J., Krause, J., Berg. A. C., and Fei-Fei, L. Hedging your bets: Optimizing accuracy-speeiffcity trade-offs in large scale visual recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3450-3457 IEFF, 2012.
Malinin, A. and Gales, M. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018
10

---

Learning State- Bate-BBsed Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Ncal, L., Olson, M., Fem, X., Wong, W.-K., and Li, Open set learning with counterfactual images. Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Yang, H.-M., Zhang, X.-Y., Yin, F., Yang, Q. and Liu, C.-L. Convolutional prototype network for open set recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (5):2358-233) 2020.
Peng, H., Li, J., Hc, Y., Liu, Y., Bao, M., Wang, L.,, Song, Y., and Yang, Q. Large-scale hicrarchical text classification with recursively regularized deep graph-enn. In P Proceedings of the 2018 world wide web conference, pp. 1063-1072, 2018.
7hang, H. and Patel, V. M. Sparse representation-basee open set recognition. IEEE transactions pattern analysis and machine intelligence, 39(8):1690-1666 2016.
Zhao, Y., Yan, K., Huang, F., and Li, J. Graph-based high-order relation discovery for fine-grained recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10099-15088, 2021.
Pyakurel, S. and Yu, Q. Hicrarchical novelty detection via fine- grained evidence allocation. In Salakhutdinov, R., Kolter, Z., Heller. K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Ma- chine Learning Research, pp. 41280-411 11280-41302 PMLR, 21-27 Jul 2024. URL https::/proceede/e 2223/pyyauuss124.... html.
mlr. press/
Rastegar, S., Doughty, H., and Snock, C. Learn to categorize or categorize to learn? self-coding for generalized category discovery. Advances in Neural Information P Processing Systems, 36:72794-72818, 2023.
Rebuffi. S.-A.,
Bilen, H., and Vedaldi, A. Learning multiple visual domains with residual adapters. Advances in neural information procexsing systems, 30, 2017.
Ren, J., Fort, S., Liu, J., Roy, A. G.. Padhy, S., and Lakshmi- narayanan, B. A simple fix mahalanobis distance for im- proving near-ood detection, arXiv preprint arXiv:2106.0902. 2021.
Ruiz, I. and Serrat, J. Hierarchical novelty detection for traffic sign recognition. Sensors, 22 438449,3,, 2022.
Sensoy, M.. Kaplan, L., and Kandemir, M. Evidential deep learn- ing quantity classification uncertainty. Advances neural information processing systems, 31, 2018.
Sun, J., Wang, H., and Dong, Q Hicrarchical attention network for open set fine- grained image recognition, IEEE Transactions on Circuits and Systems for Video Technology, 2023.
Sun, X., Yang, Z., Zhang, C., Ling, K.-V., and Pong, G. Condi- tional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13480-13489, 2020.
Vaze. S. Han, K., Vedaldi, A., and Zisserman, A. Open-set recog- nition: A good closed-set classifier is all you need. In Interna- tional Conference on Learning Reprexentations, 2021.
Wang. H.. Liu, W., Bocchieri, A., and Li, Y. Can multi-label classification networks know what they don't know? Advances in Neural Information Processing Systems, 34:29074-2906 2021.
Wang. Y., Hu, Q.. Chen, H., and Qian, Y. Uncertainty instructed multi-gramularity decision for large-scale hierarchical classifica- tion. Information Sciences, 586:644-666 661, 2022.
Welinder, P., Branson, S., Mita, T., Wah, C., Schroff. F., Belongie, S., and Perona, P. Caltech-ucsd birds 200. 2010.
Wu. T.-.,, Ho, C.-H., and Vasconcelos, N. Protect: Prompt tun- ing for taxonomic open set classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16531-16540, 2024.

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Learning State-Based Node Representations from a Class Hierarchy for Fine-Grained Open-Set Detection
Supplementary Material
Organization. In section A, present the summary of all the notations used in the paper. In section B, discuss the details of local and global detectors. In section C, we provide the proofs and illustrations of theorems presented in the main paper. In section D, we discuss details of experiments and provide additional results. In section E, we discuss potential societal impacts of the work, along with the limitation and future works. In section F, we provide the source code.
A. Summary of Notations
Notation
Description
H Durin N
A training hierarchy of closed-set classes Training Dataset
Total number of data samples in training Set of all leaf labels training dataset Set of all leaf nodes; is a function of hierarchy Set of all non-leaf nodes; is a function of hierarchy
y
L(H)
(HH)
P(v)
Parents of node y
C(y)
Children of node y
D(y)
Descendants of node
A(y)
Ancestors of node Siblings of node
Sa(v)
Dust
Test Dataset
O
An openset class belonging to y
(
of openset classes of hierarchy A resulting hierarchy of H'' augmented with O(H)
H'
Di.
Known dataset for classifier lc
Di
Openset dataset for classifier Ic
ph(y)
Path probability of path from root to node y
9y
Gradient of loss w.r.f ground truth logit Gradient of loss w.r.t non-ground truth logit
Jy
G
Graph representing hierarchy H
V
All the nodes of the graph G
E,
A set of all the subsumption edges in the graph G
E.
A set of all the exclusion edges in the graph G
A global state assigning binary values to all the nodes in G Global state matrix of G
S
M
Total number of nodes in G
h
FGOD network
h(um lz;
Logit from FGOD network for node Um Un-normalized probability for state
((s]a:)
p(s(ar)
Normalized probability for state
p(umla) (z)
Probability for node Um
FGOD network parameters
FGOD network parameters when node a removed from H'
B. FGOD using Local and Global Detectors
In this section, we provide additional details on using local and global detectors for FGOD.

---

Learning State-Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Algorithm 1 FGOD with hierarchy
Require: A of thresholds À for all local detectors (one for each non-leaf node).
Current local classifier, Ic <- root node
while Stopping Condition + 1 do Probability distribution of (c: E [R(C())
Openset score: o E Prediction from g: orgmar(p) if S Ate then le: predicted child node, y if Ic L(H) then
Final Prediction + Ic, Stopping Condition f- 1
end if else
Final Prediction + Olc)) Stopping Condition +-
end if
end while
Local detector based FGOD. Algorithm summarizes the open-set detection process using local detector. Based on how openset is determined, existing general openset detection methods can be adapted for FGOD, Here, we consider some commonly used open-detection strategies. For example, the openset score can be quantified using the KL divergence between a uniform distribution and model predicted distribution over the child nodes (eg, the top-down method (Lee et al., 2018)), model predicted maximum softmax probability (MSP) (Vaze et al., 2021), free energy based evaluation (Liu et al., 2020), or evidential uncertainty (Sensoy et al., 2018). In what follows, we first use the top-down method as concrete example present specific loss function in (9) and then generalize into a more abstract form in (10):
CH
E (t,y)E (eEN((H)
E
lc; | [c))
E ([U (a,))DD
(9)
E {c)e Cic)
(10)
(E))((H)
where C). maximizes the probability of ground truth class for each classifier using known samples D and L? is a regularization term for the classifier assign high openset score to openset samples in Di
We summarize the inference process of FGOD with hierarchy in Algorithm 1.
In addition to using the KL divergence to specify the confidence, may consider other options, including (i) Maximum Softmax Probability (MSP) (Vaze et al., 2021). evidential uncertainty (Sensoy et al., 2018), energy score (Liu et al., 2020), and Maximum Logit (Vaze et al., 2021) as openset score. For MSP baseline, the training is the same as the Top-down method, as in Eq. (9). For other baselines, we use the loss function according to evidential loss, where the first loss to output high evidence for the correct class, and KL regularizer is used train the novel samples output uniform KL divergence. The evidential based loss function is:
[C(te)]
HC--videnttall
E
E (f.
you yul[n((t]))) In(aw))
(7))
+
E
KL le; D(I < 1,1,..,,,110
(1I)
(x.))
2

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
For each method, if the following conditions hold, the sample is not detected openset.
Maximum Softmax Probability: max(Pr(.]ees 0)) 2 Ate
(12)
Evidential Uncertainty: c(lc)]
S
Ave
(13)
St
[C(io]]
Energy Score: In Σ
exp exph (x; 0) S Ale
(14)
Maximum Logit Score: max max(1u (z; 0)) 2 A/e
(15)
Instead of relying on the openset score, samples in Di can be treated as an extra openset child node Oyy) (Neal et al., 2018).
In this way, if the classifier predicts the highest probability for Oyy). it detects the sample as openset. Figure provides a specific example on how to add an extra node to each non-leaf node in the hierarchy that transforms open-set detection in hierarchy into multi-class classification problem, where each leaf node correspond score comparison no longer depends pre-defined threshold. If the highest probability of any child node is lower than that of the extra novel node, we can classify the sample as openset. A loss function can be defined accordingly:
class. In this case, the openset
C
E E [- 1mply/2,, 0)) +
|- (m((c((()))))
(16)
(cEN((H) (x.g)EDk.
(x.y)e
Similarly, each detector leverages the openset score to find out if the sample is openset or not. For this method, if the following condition holds, the sample is not openset.
max(pp(|=,,c p(l(
(17)
Details of Global Detector Methods
Dual Accuracy Reward Trade-off Search (DARTS) (Deng et al., 2012) is adapted for FGOD by training a classifier using leaf nodes and obtaining the expected rewards for all the leaf nodes and openset non-leaf nodes.
Relabel (Lee et al., 2018) trains a single classifier using leaf and openset non-leaf nodes by relabeling some of the leaf nodes openset ancestor nodes.
Leave-One-Out (LOO) (Lee et al., 2018) trains a classifier with leaf and openset non-leaf nodes. For openset nodes, each node from the hierarchy removed one at time to use it as an openset parent node.
Top Down + Leave-One-Our (TD+LOO) (Lee et al., 2018) uses input from Top Down method to LOO training.
Evidential learning (Sensoy et al., 2018; Pyakurel & Yu, 2024) uses the evidential loss train a classifier following the LOO strategy. Evidential uncertainty is used for openset detection.
Evidential- Hierarchical Novelty Detection (E-HND) (Pyakurel & Yu, 2024) trains the model to separate evidence margin of closed-set and openset classes.
Open-set Detection using State-FGOD.
The inference process of State-FGOD is summarized in Algorithm 2.
C. Proofs of Theorems
C.1. Proof of Theorem
Proof. Let us assume the ground truth node as y. For simplicity purpose, we denote the logit output by the neural network
by o. For example: we denote hyllz) by oy. The path probability of the node is given by:
ph(y)z) II p(a(P(a);x)
(18)
AEA(s)
(P())
X eov + Σ EbbES(l)) Cou @P + EAVES Cw x x
(19)
3

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Detector 1
Bird
Bird
Detector 2
Detector 2
Seagull
Sparrow
Seagull
Sparrow
Openset
Openset California
Ivory
Fox
Grasshopper
Openset
California
Ivory
Fox
Grasshopper
Bird
Sparrow
Gull
Gull Sparrow Sparrow
Sparrow
Gull
Gull
Sparrow Sparrow
(a)
(b)
Figure 8. diagrammatic representation of (a) Local detector method: There are three classifiers on Bird, Seagull and Sparrow, respectively. (b) Global Detector method: single global classifier for all leaf nodes and openset non-leaf nodes.
Algorithm Inference using State-FGOD
Require: A set of biases b.
Require: A trained State-FGOD model h parameterized with 6.
Input: A test data sample (Iiwr, Un')
Logits for all the M nodes: h(vi [xw' 0), // znr; 0), Ju(um (an ".. 0) nonnoomalized probabilities for each state p(s( using Eq. (2).
Un-normalized probabilities for each leaf node Um E C(H()
p(vml ) = E.es (s( (v.. ( (1s))
for in biases do
Add bias logits of openset nodes, Vum O(H), p(umlaw) p(uml Obtain normalized probabilities for all leaf nodes of H'' p
Select node with the highest probability argmar(p)
end for
The definition of cross-entropy loss for path probability becomes.
e.v
(P(())
C
log e.v + E she
(oP(s) + Eaves,(PO)) ew X
(20)
First, we take the derivative of loss w.r.t ground truth logit og. All the terms except the 1s one not function of oy so we treat them as a constant con. Similarly, we denote the normalization of the first fraction as Zy.
d d
log c.v + EsbESh co
x con
(21)
doy doy
Using chain and product rule derivative
Zy
con X Zye C"
X
x 0 con
(22)
6
Zy
Z3
Zy
x
x
con
x !e (Zy - -evv]
(23)
coy
z2
cy
(24)
6Pv ExbeS,()) eoab
The gradient is negative, meaning minimizing the loss function increases the ground truth logit o,, If the value of oy is low, the gradient magnitude high. Similarly, for a high value of non-ground truth sibling leaf logit, Osb. sb So(y), the gradient magnitude is high. Except for the non-ground truth sibling leaf logits, the gradient magnitude is not dependent

---

Learning Shate-Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
on logits of other nodes, which implies local optimization. Next, take derivative w.r.t non-ground truth leaf logit, sb Syy). The terms other than the 1st not the function of sb, so we treat them constant con.
d d
e.
do
do
log
E
(bb e
con
(25)
Using chain and product rule of derivative
Zy
c"y
x 0 + con Zy 0 - e00b
(26)
e²v con
Zzz
Zy
1
je Ea
(27)
ev
con
z
(28)
e.v + Σ sb) al
The gradient is positive, meaning minimizing the loss function decreases the non non-ground truth sibling leaf logit. If the value of Ost is high, the gradient magnitude is high. Similar gradient analysis can be obtained for all non-ground truth nodes, y E C(a), Va E A(y). Next, we take derivative w.r. a non-ground truth logit of the node y' & C(a), Va E A(y), which are child of the node in the path of the root node to the ground truth node.
dC
(29)
doy
The gradient as the loss is not function of the y'. So, minimizing the loss function does not update the logits of these nodes. Therefore, it does not guarantee the decrease of all the non-grov non-ground truth logits.
C.2. Proof of Theorem
Proof. The state probability of the node is given by:
p(y
II ct (aEAA())
(30)
aEA(y)
p(ylar)
p(y
(31)
1 Σ CEECNY p([[x)
(32)
1 Eiecoee Ed og
The cross-entropy loss for state probability becomes
Eue Pa
L log
(33)
1 Eiec(ryy Coeee 9
First, we take the derivative of the loss w.r.t ground truth logit oy. We denote the normalization of the first fraction Z.
Ze Zu on - -Cu or ECCCACo) Oa
dC
Z
(34)
do,
EeCA(l
oa
Zz
Z
e on
x 1 ChEEAAor 0a
(35)
EecA(y) 60
72
EoCAty) On
(36)
EiEchh Ead on
The gradient is negative, meaning minimizing the loss function increases the ground truth logit oy. If the value of o, is low; the gradient magnitude high. For a high value of all the non- non-ground truth logits, the gradient magnitude is high. Next, we
5

---

Learning Mlate-Basel Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
take the derivative w.r.t non-ground truth leaf logit y/-
dC
Z
(EEAA)))) 90 Ea' Pat
x
(37)
doy
e On
Z²
Z
X
(CC) On
X e
(38)
A
Z2
EeCCAG Odl
(39)
1+ EiECCHH (ZoeE On"
The gradient is positive, meaning minimizing the loss function decreases the non-ground truth leaf logit og. If the value of of is high, the gradient magnitude is high. Therefore, it guarantees the decrease of all the non-ground truth leaf logits. Next, we take the derivative of the loss w.n.t non non-ground truth non-leaf logit ya-
dC
Z
(40)
doy
E
2 x 0 - (
EE (WW,))VCCCH)) eDeen oa)
x
(41)
Z
(42)
Z
The gradient is positive, meaning minimizing the loss function decreases the non-ground truth leaf logit og If the value of 0y, is high, the gradient magnitude is also high. With this analysis, all the values of non-ground truth logits of decrease by minimization of the loss function.
C.3. Illustration of theorems
In this section, we illustrate the key concept introduced by Theorems 3.1 and 3.4. We use Figure 4 representative example. For each of the nodes, we have an associated logit value defined as: Seagull: 02. Sparrow: og, Openset Bird: 04, Openset Seagull: og, California Gull: og. Ivory Gull: 07, Fox Sparrow: og. Grasshopper Sparrow: og and Openset Sparrow: 010- Let's take a sample from California Gull, the path probability can be defined as:
ph(California Gull[z) exp(02 + 06)
(43)
Zpa
where, Zpp = exp(o22 + 05) + exp(02 + 08) exp(02 07) exp(03 os) exp(03 + 0s) + exp(o3 + 07) + exp(04 +
05) +exp(o+ + 06) exp(01 + 07). The loss function that maximizes the path probability ph (Caaliffoii Gull(z) is defined as:
[exp(02 + 06)
_ = log
(44)
Zph
In theorem 3.1, there are three types of gradient analysis: (i) gradient for ground truth logit, (ii) gradient for non- non-ground truth logit, and (iii) gradient for non-ground truth logit with special condition. First, we analyse gradient w.r.t logit of the ground truth class 0g:
Zph
Zpa exp(02 + 0a) - exp(o2 + os) /exp(02 + os) + exp(os + 06) + exp(o: - 06)
dos
exp(o2 + 06)
2
Zph
lexp(02 + 08) + explos 06) + exp(os 06)]
Zpz
cxp(oe)(exp(0z)) + cxp(os) cxp(o)) cxp((o)))
(exp(02) + exp(o3) exp (exp(])]eexp(() + exp(os) + exp(or))
exp(o8)
exp(os) + exp(os) exp(
6

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Table 8. Gradient Summary
Method
Node
Gradient
Remark
Path Proh.
Seagull
Local Optimization:
Magnitude only depends on the ground ruth node Seagull and its siblings (Sparrow and Openset Bird).
State Prob.
Seagull
[1 cimes x (e 27
Global Optimization: The gradient magnitude depends on all the nodes.
Path Prob.
Sparrow
Local Optimization:
Magnitude only depends the non-ground truth node Spurrow and its siblings (Seaguil and Openset Bird).
State Prob.
Sparrow
"nt this xxe 4. 40
Zxp
Global Optimization: The gradient magnitude depends on all the nodes.
Path Prob.
Openset
Local Optimization:
Magnitude only depends the non-ground truth node Openset Bird and its siblings (Scagull and Sparrow).
Bird
State Prob.
Openset
Global Optimization: The gradient magnitude depends on all the nodes.
Bird
Path Proh.
Openset Seagull
ena coEieehi-pe
Magnitude only depends on the non-ground truth node Openset Seagull and its siblings (Califomia and Ivory Gull). Global Optimization: The gradient magnitude depends on all the nodes.
Local Optimization:
State Prob.
Openset
tug tur
Zap
Sparrow
Path Proh.
California
200 2% 0" 4
Local Optimization:
Magnitude only depends on the ground truth node California Gull and its siblings (Ivory Gull and Openset Seagull). Global Optimization: The gradient magnitude depends on all the nodes.
Gull
State Prob.
California
[1 Zop
Gull
Path Prob.
Ivory Gull
ex7 t +e"
Magnitude only depends on the non-ground truth node Ivory Gull and its siblings ( Openset Seagull and California Gull). Global Optimization: The gradient magnitude depends on all the nodes.
Local Optimization:
State Prob.
Ivory Gull
014422or 7ip
Path Prob.
Fox Spar-
There is gradient update.
row
State Prob.
Fox
Spar-
"n tuy Zip
Global Optimization: The gradient magnitude depends on all the nodes.
row
Path Proh.
Grasshopped Sparrow
There gradient update.
State Prob
Grasshopper
"y buy top
Zap
Global Optimization: The gradient magnitude depends on all the nodes.
Sparrow
Path Prob.
Openset
There is no gradient update.
Sparrow
State Prob.
Openset
Global Optimization: The gradient magnitude depends on all nodes.
Zoyy
Sparrow
Here, the gradient is ve, and the gradient magnitude only depends on the siblings of California Gull: Openset Seagull and Ivory Gull. The gradient update increases the value of og. We can observe that, the increase of the value is dependent only on the local view (the ground truth class and its siblings.)
Next, we analyse gradient w.r.t a non- non- -oonnrdun truth class 07.
d.Pa
Zpm
Zen 0 - exp(02 + 08) + 07) + exp(03 + 07) + exp(o4 + 07)]
doy
exp(o2 + 06)
23
explo2 + 07) + exp(os + 07) + exp(os++ 07)
Zpz.
+ cxp(o=)[exp(o)) + cxp(s)) explos)
(exp(02) + exp(os) + exp( x (exp(os) + exp(o6) + exp(o7)
exp(07)
exp(os) + cxp((s) + exp(or)
Here, the gradient is +ue, and the gradient magnitude only depends on the ground truth and its siblings (local optimization). Finally, let's analyse gradient w.r.t a non-grone non-ground truth class s/ & C(a), Va E A(y) as presented in theorem 3.1. In the representative example of 4, the symbols represent the following for the image of California Gull:
7

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
y California Gull
A(y) (Bird, Seagull, California Gull}
Va A(y), (Seagull, Sparrow, Openset Bird, Opensest Seagull, California Gull, Ivory Gull)
Non-ground truth nodes with condition: y' C(a), E A(y) = { Fox Sparrow, Grasshopper Sparrow, Openset Sparrow
If take the gradient w.r.t non-ground truth node with condition, for example 08-
dL
(45)
dog
Since, the gradient value is 0, there is no update for the non non-ground truth logits with condition. It should be noted that, in this particular example, the non-ground truth nodes with condition are from the leaf level in the hierarchy. However, as the depth of hierarchy increases, nodes from non-leaf levels are also included in the non con-ground truth nodes with condition.
For theorem 3.4, we have two types of gradient analysis. (i) ground truth logit and (ii) non-ground truth logit. The state probability for California Gull be defined
P(California Gull[a:) exp(01 + + 06)
(46)
Zap
where Zsp + exp(o1 + 01) exp(o1 + 02 + os) + exp(o1 + 02 + o6) + exp(o1 + 02 + 07) exp(o1 + 03 + 08) +
exp(o) 03 0g) exp(o) 03 + 010) The corresponding loss function defined as:
- log exp(o1 + 02 + 08)
(47)
Zop
First, taking gradient for ground truth class, California Gull we have:
dC
Zep
Zap exp(o1 + 02 + 08) exp(oi + 02 exp(() 02 06)
dog
exp(o1 + 02 06)
Z2p
explo1 + 02 + 06)
Zep
The gradient is vv and the magnitude is dependent of all the nodes (global view). Next, taking gradient for non-ground truth class Fox Sparrow:
d.
Zsp
Zap x 0 exp(o1 + 02 + 0B) exp(o) + 03 + 08)
dog
exp(oI + 02 + +08)
22
exp(o1 + 03 + 08)
Zop
We see that all the gradient is +ve and magnitude is dependent on all the nodes. Similarly, there is not any nodes with zero gradient update. The gradient summary of all the nodes for sample of California Gull in the example with the remark presented in Table 8.
D. Details of Experiments and Additional Results
In this section, we present more details on the experiments along with some additional results.
D.1. Details of Datasets
We perform experiments on three real-world hierarchical datasets. The details of each dataset are summarized in Table 9, including (i) the Total number of samples, (ii) the Total number of classes, (iii) Known class split: total number of classes used the training (iv) Openset class split: total number of classes used in testing as openset classes (v) Number leaf nodes in the training hierarchy (vi) Number of non-leaf nodes in the training hierarchy and (vii) Maximum depth associated with the training hierarchy.
8

---

Learning Mate-BBaedd Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Table 9. Description of real- world hicrarchical datasets
Dataset Name
Tiny Imagenet
CUB-200-2011
AWA2
Total Number of Samples
120, 000
12, 000
37, 37,000
Total Number of Classes
200
200
50
Known Class Split
150
150
40
Openset Class Split
50
50
10
Leaf Nodes Training Hierarchy
150
150
40
Non-Leaf Nodes in Training Hierarchy
86
43
21
Max Depth of Training Hierarchy
12
7
7
Table 10. Significance Test
CUB
Tiny Imagenet
AWA2
NA
AUC
NA
AUC
NA
AUC
Baseline State FGOD
53.72-0.339
4..54
21.4
17.1640..9
48.75 0.23
44.86:0..5
64,29 + 0.17
50.79 + 0.17
24.53 + 0.50
19,83 t 0.39
53.24 +: 0.30
47.71 1 0.31
p-value
1.48 x 10 10-11 1I
5.77 x 10-12
2.06 10
7.07 10
2.87 x 10-12
3.64 x 10 10-10
D.2. Implementation Details
For the proposed method and baselines, we use ViT-B/16 as the baseline, and prompt as parameter efficient fine-tuning method. We follow the default configuration of prompts, learning rate, and number of epochs as presented by (Jia et al., 2022). For the taxonomy definition, we define a pre-defined numpy file, along with state representation. The state representation defined in the NumPy file is leveraged in defining the loss function 6. The code is written in Python using PyTorch as a library. We conduct all the experiments in NVIDIA A100 with 32GB memory. For the known-nove split, we follow (Pyakurel & Yu, 2024). The conceptual diagram of model training is presented in Figure 6. For the EHND method, use the value of 31 and B2 suggested by the paper (Pyakurel & Yu, 2024). For relabel method, use the relabing rate of 20%.
D.3. Significance Test
For the results presented in the main paper, the strongest baseline for CUB is E-HND. Similarly, for Tiny Imageet and AWA2, the strongest baseline is Relabel and I.00 respectively. In this section, we run our method along with the strongest baseline for 5 different seeds to test whether the improvement of the proposed method is statistically significant. The mean of the performance, the standard deviation of the proposed method, and the strongest baseline, along with the p-value of the t-test is presented in Table 10. The lower value of p-values justifies the statistical significance of the improvement from the proposed method.
D.4. Additional Results
Experimental results on Stanford Cars and Cifar100 In this section, we compare the proposed method on additional datasets along with some representative global detectors: LOO, Relabel, and EHND. For 100, we select 40 classes as closed-set classes. We construct the hierarchy of using those 40 classes as leaf classes resulting in 21 non-leaf nodes and depth of 3. Similarly, for Stanford Cars, select 166 classes closed-set set classes. We construct the hierarchy using those 166 classes as leaf classes resulting in 10 non-leaf nodes and a depth of 3. The comparison results are presented in Table 11. From the comparison, we see that the performance of the proposed method is superior to the baselines justifying the effectiveness of the proposed methodology.
Qualitative study. We start by showing how the path probability mitigates the shortcomings of local detector methods. For an openset sample of Le Conte Sparrow, we use the baseline Hierarchical Classifier with extra nodes with and without path probability in Figure 9a). The ground truth of the openset sample is Sparrow. We provide probabilities from local detectors in the box of its corresponding child nodes, and path probabilities in blue color at the end of each path. The correct way of inference for local detectors is Bird - Passeriform Bird + Sparrow. Then, the detector Sparrow should allocate a high openset score so that it would not be further classified to the child node. However, we see that Oscine Bird gets the
9

---

Learning State-Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Table 11. Comparison results on additional datasets
Stanford Cars
Cifar100
Method
OA
AUC
OA
AUC
LOO
33,15
28.31
22.71
18.38
Relahel
29.82
26.75
18.89
16.26
EHND
41.47
28.81
15.89
12.42
Stae-FGODD
47.03
35.05
24.97
20.69
Le Conte Sparrow
Bird
Bird
Passeriform
kingfisher
Passeriform
kingfisher
Bird 0.89
0.0021
Bird 0.9994
0.00000202
Sporrow
Oscine Bird
Oscine Bird 0.1186
0.343
0.576
Sparrow
0.8614
Nelson Sharp Tailed Sparrow
Fox Sparrow
Warbler
Cape Glossy Starling
Warbler
Cape Glossy Starling
0.46
0.168
0.008
0.0024
0.001
Nelson Sharp Tailed Sparrow
Openset Sparrow
0.0072
0.000071
0.051
0.0024
Myrtle Warbler
Palm Warbler
0.0505
0.5218
Myrtle Warbler
Palm Warbler
0.014
0.034
0.003
0.008
0.0009
0.0004
(a)
(b)
Figure 9. Detection result on sample of Le Conte Sparrow, an openset sample of Sparrow for (a) local detector vs local detector with path probability and (b) State-FGOD
higher probability (0.576) than the correct prediction Sparrow (0.343) and makes the mistake of GrainGr Openset Detection. However, if we use path probabilities for inference as provided for all the given paths in Figure 9()), the path from Bird -> Nelson Sharp Tailed Sparrow with path probability (0.051) is selected. Next, the local detector Sparrow has a high openset score of 0.58, and therefore provides the final prediction as Openset Sparrow. In this way, path probability mitigates the error from Passeriform Bird detector.
However, there is still one issue that Sparrow class has a lower probability than Oscine Bird. Although the final prediction is correct, the correct non-leaf nodes still can have a lower probability value than their siblings. If we observe the prediction from the proposed method (State-FGOD) in Figure 9(),, the ground truth Openset Sparrow has the highest probability (0.5218) among all the leaf nodes of the hierarchy. Additionally, non-leaf node Sparrow (0.8614) has the highest probability than its siblings (eg, Oscine Bird with probability of 0. 0.1186). Similar is the case for Passeriform Bird. This probability distribution is possible for the proposed method due to the state formulation as discussed in Proposition 3.3.
For the next openset sample from Chestnut Sided Warbler, we compare the local detector with path probability with the proposed method "State-FGOn in Figure 10. We observe that one of the child nodes Green Kingfisher has higher logit value, and high probability, making the path probability higher for the path of Bird - Green Kingfisher highest in Figure 10(a). Because of this, the path probability of one of the child nodes of ground truth node Warbler is not the highest, making Green Kingfisher a wrong final prediction of FGOD. However, in Figure 10(b). we observe that the proposed method assigns the highest logit to Openset Warbler, making a correct FGOD prediction.
Computational Cost Analysis: We conducted a computational cost analysis. We define the input dimension of the classifier head as D. the total number of leaf nodes in the class hierarehy as N, and the depth of the hierarchy as h. For VIT
10

---

Learning State-Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Chestnut Sided Warbler
Bird
Passeriform Bird
kingfisher
Bird
0.593
0.344
Passeriform Bird
kingfisher
Ringed Kingfisher
Oscine Bird
Kingfisher
0.159
0.151
0.056
Oscine Bird
Green Kingfisher
Ringed Kingfisher
0.078
0.033
Warbler
0.0000253
0.000004
0.70
Warbler
Golden Winged Warbler
Myrtle Warbler
Golden Winged Warbler
Openset Warbler
0.356
0.029
0.379
0.009
0.013
0.0011
(a)
(b)
Figure 10. Detection result on sample of Chestnut Sided Warbler, openset sample of Warbler for (a) local detector with path probability vs (b) State-FGOD
and ResNet backbone, the value of D is 768 and 2048, respectively. For the Tinylmagenet, CUB, and AWA2, the value of N is 236, 193 and 61, respectively. The corresponding values of h for these datasets are 12, 7, and respectively. So, note that D > N > h.
For local detectors, we assume the output dimension of each classifier to be n on average, which is no more than 5 in our datasets. So, we have n < N. The time complexity the logit calculation is O(D x n x h). The top-down inference for receiving the final prediction takes O(n h). The total complexity is O n x h) + O(n x h), and overall complexity O(D X n x h). For global detectors, the classifier output dimension is N, resulting O(D N). The prediction complexity is O(N). Hence, we obtain total complexity of O(D N) + O(N) and overall complexity of O(D x N). For the proposed method, we need to aggregate the probabilities by utilizing the global states, which is an extra step compared to global detectors. This is obtained using matrix multiplication of logits and global states, adding the complexity of ( Thus, the total complexity is O(D x N) + O(N) + O(N2). For the datasets we used, we have D> > N, resulting an overall complexity of O(D x N), which is the same as the global detectors.
logit calculation complexity of
We use the VIT backbone calculate the inference time (in milliseconds for one test sample) for the proposed method and representative baselines. The analysis is presented in Table 12. The total inference includes the time of both the logits calculation and the final prediction. Since the number of leaf classes follows the order Nriny > NCUB > NAW A2- the corresponding inference times (both total and logit calculation) adheres to the same trend TimeTiny TimecuB >> Time 42. Among all the datasets, local detectors require less time than global detectors, which is due to smaller output dimension n < N and top-down inference < N. Finally, the overall complexity analysis revealed that the logits calculation has the dominant time complexity. This is reflected in a substantial portion of the total inference time occupied by logits calculation. In comparison to other global detector methods, the proposed method requires slightly higher inference time, which is due extra step of calculating global state probabilities. Overall, the inference time of all the methods is fairly fast, making them usable in practical setting.
E. Limitations, and Future Work
In this work, we evaluate the proposed approach using various benchmark datasets to validate the superior performance of the proposed method. We acknowledge that the real openset dataset is unbounded for a given training hicrarchy. To capture the nature of such data, we utilize the samples from the closed-set classes to train open-set classes as well.

---

Learning State- Based Node Representations from Class Hierarchy for Fine-Grained Open-Set Detection
Table 12. Inference and Logits Caleulation Time (in milliseconds)
Method
Metric
TinyImagenet
CUB
AWA2
Total inference
0.0009189
0.0007314
0.0002797
TD
Logits calculation
0.0007530
0.0005900
0.0001950
Total inference
0.0009213
0.0007297
0.0002807
MSP
Logits calculation
0.0007530
0.0005900
0.0001950
Total inference
0.0009225
0.0007464
0.0002794
HC-EN
Logits calculation
0.0007530
0.0005900
0,0001950
Total inference
0,3191200
0.2956500
O. D.1330400
LOO
Logits calculation
0.3138200
0.2930120
0.1304500
Total inference
0.3186700
0.2959900
0.134120
E-HND
Logits calculation
0,3138200
0.2930120
0.1304500
Total inference
0.3204700
0.2984500
0.1365200
STATE-FGOD
Logits calculation 0.3138200
0.2930120
0.1304500
would be interesting to generate meaningful openset samples and use them to augment training data to further improve the performance of FGOD. We will investigate the data augmentation to aid fine-grained openset dataset as a future work.
F. Source Code
The source code is provided in the repository:
12


Reviews:
[{"review_id": "HrmtxzCpNs", "reviewer": "Reviewer_1R2j", "summary": "This paper considers fine-grained open-set detection. The proposed approach utilizes hierarchical relationships to determine if a sample does not belong to any of the known classes, but still might be related to them, by e.g. sharing a superclass with some of them. The paper proposes to leverage the hierarchical structure of labels to learn state-based node representations. \nThe hierarchy is represented as a graph where the nodes represent possible classes. The set of possible classes is extended with open-set classes - each non-leaf node gets an extra child node.  One legal state consists of nodes that may be active at the same time. The model outputs a logit for each node. The probability of the state is the product of exponent of logits of the state's active nodes normalized with probabilities of all possible legal states. The model is trained to maximize the probability of the correct state.\n\n# Score after rebuttal\nI have increased my rating after the rebuttal as the authors have addressed my main concerns regarding the training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments. Our main responses are summarized below.  \n\n**Q1: Reference not discussed**\n\nThank you for suggesting this relevant work.  We would like to clarify that the primary focus of this work is to perform hierarchical classification at the pixel level. For example, for a pixel that is part of a Fox Sparrow, the goal is to identify it as Fox Sparrow, Sparrow and Bird as present in the class hierarchy: `Bird → Sparrow → Fox Sparrow`. Different from ours, this work does not consider the open-world setting. In contrast, our goal is to tackle the more challenging openset detection problem by leveraging a known class hierarchy. For the same training hierarchy as mentioned above, we aim to identify a  Le Conte Sparrow as an openset sample, since it does not belong to any known classes in the hierarchy. Our approach first detects the sample as openset, and then identifies the closest parent class (i.e., Sparrow).  We will include a discussion of this work in the related work section.\n\n**Q2: Clarification of the training process**\n\nThe design and the training process of the proposed State-FGOD model are presented in Section 3.3. Before describing the training process (right above Theorem 3.4), all the key technical concepts are presented in detail in earlier parts of this section. The overall training loss is shown in equation (6). During the training process, we remove one node $v_m$ at a time. This creates a dynamic hierarchy $\\mathcal{H}' \\setminus v_m$ (lines 315-316 in the paper and more examples are shown here: https://anonymous.4open.science/r/FGOD_training-37D9/README.md). When $v_m$ is removed, all the samples that belong to $v_m$ will be treated as openset samples with respect to $\\mathcal{H}' \\setminus v_m$. The goal is to learn a more compact representation of $\\mathcal{H}' \\setminus v_m$ that can help to detect openset samples that fall outside of this hierarchy.  For a training input $x$ with ground truth $y$, if $v_m = y$ or $v_m \\in \\mathcal{A}(y)$, then the ground truth label is changed to the openset class as $y = \\mathcal{O}(\\mathcal{P}(v_m))$. Otherwise the ground truth label is not changed, and the sample belongs to closed-set classes.  For each removed node $v_m$ and training sample $(x,y)$, the training loss maximizes ground truth node probability $p(y|x, \\theta^{\\mathcal{H'}\\setminus v_m})$. \n\n**Q3: Calculation of probability $p(y|x)$ in dynamic hierarchies**\n\nWe present the calculation of state probability of California Gull in equation (46). This is equivalent to probability $ p(y|x, \\theta^{\\mathcal{H'}})$, where sample $x$ belongs to the California Gull class.  However, in equation (6), it contains different parameters that depend on dynamic hierarchies, which are defined in lines 315-316 in the paper and further clarified in our response to Q2. \nAs reviewer mentioned, we obtain logits from a single model. We aggregate these logits to obtain unnormalized state probabilities as given by equation (2). The calculation of probability in dynamic hierarchy $p(y|x, \\theta^{\\mathcal{H'}\\setminus v_m})$ leverages the same unnormalized state probabilities and only differs in normalization factor $Z$.  To obtain $Z$ when $v_m$ is removed, we only sum unnormalized probabilities of those states, which remain in the dynamic hierarchy $\\mathcal{H'}\\setminus v_m$. \n\n**Q4: Definition of $p(y|x)$**\n\n For a sample with ground truth $y$, $p(y|x, \\theta^{\\mathcal{H'}})$ is the node probability (lines 310-311 in paper) in the hierarchy $\\mathcal{H'}$. When node $v_m$ is removed, $p(y|x, \\theta^{\\mathcal{H'}\\setminus v_m})$ is defined to represent node probability in the corresponding dynamic hierarchy. Node probability is the sum of all the state probabilities where node $v_m$ is active, as explained in lines 289-290. For any ground truth node $y$, only one state is active, making the node probability equal to state probability corresponding to the node $y$. \n\n**Q5: how the \"open-set\" nodes are trained**\n\nAs explained in our response to Q2, we remove one node $v_m$ at a time during model training. All the samples that belong to the removed node will play the role as openset samples with respect to $\\mathcal{H}' \\setminus v_m$. \n\n**Q6: Role of bias**\n\nThank you for raising this point. The role of bias is explained in lines 374-375. There is a trade-off between closed and open set performances. To capture the trade-off in evaluation, we define a fixed set of biases $\\mathbf{b}$ in [-20, 20].  For each $b\\in\\mathbf{b}$, we add this term to the logits of open-et nodes. When the value is positive, it favors openset nodes in prediction, and when the value is negative, it favors closed-set nodes. For each $b$, we obtain a pair of closed-set and openset accuracy. We plot them to obtain a curve and use Area Under Curve as an overall evaluation metric.", "claims_and_evidence": "The paper claims that the proposed probability formulation considers all of the nodes during training, as opposed to conventional approaches that train independent classifiers along the hierarchy. This is shown through two proofs that seem correct. Good experimental results further support the proposed method.", "methods_and_evaluation_criteria": "The proposed approach is evaluated on three fine-grained datasets and follows previous work.", "theoretical_claims": "I went over the proofs in the appendix and they seemed correct.", "experimental_designs_or_analyses": "The experimental setup follows previous work and seems sound.", "relation_to_broader_scientific_literature": "The method improves on previous work on fine-grained open-set detection.", "essential_references_not_discussed": "[a] Li et al. Hierarchical semantic segmentation, CVPR 2022, this paper also enforces coherent probabilities along hierarchies", "other_strengths_and_weaknesses": "The paper is somewaht difficult to follow at certain points. A lot of space is dedicated to describing local detection methods leaving some parts of the proposed method unclear, especially with regards to training. See questions", "other_comments_or_suggestions": "No comments", "questions_for_authors": "I am still unclear about exact training of the method. \nQ1) Eq 46 seems to suggest that p(y|x) is equal to p(s|x) and that there is a single model that produces those probabilities. Eq 6 on the other hand contains different parameters that depend on dynamic hierarchies, that are not explained anywhere.\nQ2)  Also, p(y|x) is not defined previously.\nQ3) It is also not clear to me how the \"open-set\" nodes are trained exactly. If only the closed set classes are used during training, I would expect those nodes to always have a gradient which lowers them and that they would \"die out\" given enough training. What is the signal that would increase the logits for those nodes? Algorith 2 mentions a bieas that is added during inference? How is that bias determined?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YvyG1wLHhM", "reviewer": "Reviewer_JNfG", "summary": "This paper addresses the challenge of Fine-Grained Open-Set Detection (FGOD) by leveraging hierarchical structures among known classes. It introduces a state-based node representation that systematically captures fine-grained dependencies between different hierarchical levels. Unlike traditional local or global detectors, this approach reduces error accumulation and ensures optimal training through a mathematically justified method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments. Our main responses are summarized below.  \n\n\n**Q1: Sibling relationship for theorem 3.1**\n\nPath probability based optimization (Theorem 3.1) indeed considers sibling relationships. As we mentioned in the proof sketch, gradient update is dependent on the logits of the children of its parent only. For a class, children of its parent refers to its siblings. So, the sibling relationship contributes to the training. However, it does not guarantee a decrease of logits of other incorrect nodes that are not children of the nodes in the path. The failure to consider other incorrect nodes is the reason for sub-optimality. \n\nIn addition, when we provide details of the proof in the appendix, we show that the gradient of the ground truth label (Equation 24 and lines 767-769) is dependent on $o_{sb}$, which refers to the logit of sibling $sb$.\n\n**Q2:  The writing could be optimized.**\n\nWe will carefully incorporate all the reviewers' suggestions to improve the presentation of the revised paper.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "For theorem 3.1: Simply maximizing the probability of the path leading to the ground-truth leaf node does not guarantee a decrease in logits of incorrect nodes that are not children of the nodes in the path. The proof assumes that only local parent-child constraints affect optimization, but does not explicitly discuss how information from other hierarchical paths (e.g., sibling relationships) might indirectly contribute to improved training.", "experimental_designs_or_analyses": "Yes, the main results(Table 1)， the experiments is suitable", "relation_to_broader_scientific_literature": "Open-Set Recognition (OSR): Traditional OSR approaches focus on detecting whether a test sample is from a known or unknown category. Earlier works like OpenMax  estimated open-set likelihood by recalibrating softmax scores. Other methods, such as Maximum Softmax Probability (MSP) and Evidential Learning , improved uncertainty quantification for OSR.\nHierarchical Classification: Standard hierarchical classification methodsstructured predictions according to taxonomy trees but were limited to closed-set settings.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The author analyzes the shortcomings of the baseline through intuitive explanations and rigorous proofs, thereby introducing the method they propose. This style of writing aligns well with scientific research practices and is persuasive.\n2 The proposed method exhibits relatively superior performance.\n\nWeakness: \nThe writing could still be optimized.", "other_comments_or_suggestions": "In fact, I am not familiar with this field, so I am unable to provide an accurate assessment of the contributions of this paper. Please, AC, prioritize the opinions of other reviewers.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6hBGCiqD8A", "reviewer": "Reviewer_hS4h", "summary": "This paper handles fine-grained open-set detection, which features a hierarchical structure of semantic classes. The authors discussed the limitations of existing local and global detectors, with theoretical analyses unveiling the suboptimality of these detectors. Then a state-based approach is proposed by the authors to effectively capture the dependencies among the nodes in the hierarchy. A formal analysis is provided to validate the optimal training process, which is further supported by significant performance gains in the experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments. Our main responses are summarized below.  \n\n**Q1: Difference between open-set detection, near-OOD detection, and OOD detection?**\n\nThe goal of open set detection is to identify test samples that are semantically different from the closed-set classes used in the training. Some studies also refer to open-set classes as Out-Of-Distribution (OOD) and closed-set classes as in-distribution. In that sense, the goal of OOD detection is the same as open set detection. That being said, Out-of-distribution is also a more general term used to refer to any test distribution that does not match the training distribution. To this end, the definition of OOD is not limited to open set classes but may also refer to label shift, subpopulation shift, or domain shift as well.\n\nBased on how similar open-set classes are with the closed-set one, open-set detection can be categorized into near-OOD and far-OOD detection. For near-OOD detection, although open-set classes are semantically different classes, they share similarities with closed-set classes.  As an example, the closed-set classes include Husky and Golden Retriever and openset classes may include other types of dogs, wuch as German Shepherd and Bulldog. For far-OOD detection, open-set classes are much more distinct from the closed-set ones. For example, the closed-set classes may include cars and airplanes while the open-set ones may cover different animals.", "claims_and_evidence": "Yes. The authors have provided detailed theoretical analysis and experiment evidences.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The theoretical analysis of the local/global detector and the proposed state-based approach was checked, and no issues were found.", "experimental_designs_or_analyses": "The authors have clearly clarified the experiment settings including datasets, baseline setup, and evaluation metrics. The effectiveness of the proposed method is supported by the performance gains over baseline methods. And the ablations on sub-modules / designs are also provided.", "relation_to_broader_scientific_literature": "The findings in this paper on hierarchical openset detection are related to hierarchical classification that handles closedset semantic categories.", "essential_references_not_discussed": "To my limited knowledge, no related works are missing.", "other_strengths_and_weaknesses": "The paper is well written with clear motivation and smooth logic flow.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "Just out of curiosity, can the authors explain the difference between open-set detection, near-OOD detection and OOD detection?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOU7oosa2G", "reviewer": "Reviewer_YgaW", "summary": "This paper introduces a structured state representation for fine-grained open-set detection (FGOD). Prior approaches fall into two categories: local and global detectors. Local detectors iteratively multiply probabilities along a tree path but suffer from error accumulation. Global detectors flatten probabilities and predict all at once but ignore tree structures. The proposed State-FGOD addresses these issues by combining both approaches: 1) computing node logits for all tree paths, then 2) normalizing them to obtain the global probability. The paper provides theoretical justifications for previous limitations and demonstrates State-FGOD achieving state-of-the-art performance on FGOD benchmarks, supported by hierarchical performance measures.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper. We appreciate your constructive comments. Our main responses are summarized below.  \n\n**Q1: Computational Cost**\n\n Thank you for the comment. We conducted a computational cost analysis. We define the input dimension of the classifier head as $D$, the total number of leaf nodes in the class hierarchy as $N$, and the depth of the hierarchy as $h$. For ViT and ResNet backbone, the value of $D$ is $768$ and $2048$, respectively.  For the TinyImagenet, CUB, and AWA2, the value of $N$ is $236$, $193$ and $61$, respectively. The corresponding values of $h$ for these datasets are $12$, $7$, and $7$, respectively. So, we note that $D>N\\gg h$.\n\nFor local detectors, we assume the output dimension of each classifier to be $n$ on average, which is no more than 5 in our datasets. So, we have $n\\ll N$. The time complexity of the logit calculation is $O(D\\times n\\times h)$. The top-down inference for receiving the final prediction takes $O(n \\times h)$. The total complexity is $O(D\\times n\\times h)$ + $O(n \\times h)$, and overall complexity is $O(D\\times n\\times h)$. For global detectors, the classifier output dimension is $N$, resulting in a logit calculation complexity of $O(D\\times N)$. The prediction complexity is $O(N)$. Hence, we obtain a total complexity of $O(D\\times N) + O(N)$ and an overall complexity of  $O(D\\times N)$. For the proposed method, we need to aggregate the probabilities by utilizing the global states, which is an extra step compared to global detectors. This is obtained using matrix multiplication of logits and global states, adding the complexity of $O(N^2)$. Thus, the total complexity is  $O(D\\times N) + O(N)+ O(N^2)$. For the datasets we used, we have $D> N$, resulting in an overall complexity of $O(D\\times N)$, which is the same as the global detectors. \n\nWe use the ViT backbone to calculate the inference time (in milliseconds for one test sample) for the proposed method and representative baselines. The total inference includes time  of both the logits calculation and final prediction.  Since the number of leaf classes follows the order $N_{Tiny} > N_{CUB} > N_{AWA2}$,  the corresponding inference times (both total and logit calculation) adheres to the same trend $Time_{Tiny} > Time_{CUB} > Time_{AWA2}$. Among all the datasets, local detectors require less time than global detectors, which is due to smaller output dimension $n < N$ and top-down inference $h < N$.  Finally, the overall complexity analysis revealed that the logits calculation has the dominant time complexity. This is reflected in a substantial portion of the total inference time occupied by logits calculation. In comparison to other global detector methods, the proposed method requires slightly higher inference time, which is due to extra step of calculating global state probabilities. Overall, the inference time of all the methods is fairly fast, making them usable in a practical setting.\n\n| Method     | Time  | TinyImagenet  | CUB    | AWA2        |\n|------------|----------------------|---------------|-------------|-------------|\n| TD      | Total inference      | 0.0009189     | 0.0007314   | 0.0002797   |\n|            | Logits calculation   | 0.0007530     | 0.0005900   | 0.0001950   |\n| MSP        | Total inference      | 0.0009213     | 0.0007297   | 0.0002807   |\n|            | Logits calculation   | 0.0007530     | 0.0005900   | 0.0001950   |\n| HC-EN      | Total inference      | 0.0009225     | 0.0007464   | 0.0002794   |\n|            | Logits calculation   | 0.0007530     | 0.0005900   | 0.0001950   |\n| LOO        | Total inference      | 0.3191200     | 0.2956500   | 0.1330400   |\n|            | Logits calculation   | 0.3138200     | 0.2930120   | 0.1304500   |\n| E-HND      | Total inference      | 0.3186700     | 0.2959900   | 0.1341200   |\n|            | Logits calculation   | 0.3138200     | 0.2930120   | 0.1304500   |\n| STATE-FGOD | Total inference      | 0.3204700     | 0.2984500   | 0.1365200   |\n|            | Logits calculation   | 0.3138200     | 0.2930120   | 0.1304500   |\n\n**Q2: Reference not discussed.**\n\nThank you for suggesting the reference of Generalized Category Discovery (GCD) [1]. GCD and FGOD both target a related problem of assigning openset samples to a specific category. FGOD leverages an existing hierarchical structure and defines openset class for every non-leaf class in the hierarchy so that openset test samples can be assigned to those openset classes. GCD, on the other hand, does not leverage existing hierarchy but learns to categorize the classes into a hierarchical structure. When a hierarchy is not present for the dataset, GCD approaches can be applied to obtain the hierarchy before the FGOD methods are applied. In that sense, GCD and HND are complementary to each other. We will include a discussion in the revised papper. \n\n\n\n[1] Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery. NeurIPS 2023.", "claims_and_evidence": "All claims are well supported.", "methods_and_evaluation_criteria": "Intuitive and clearly explained.", "theoretical_claims": "It seems correct.", "experimental_designs_or_analyses": "Well conducted.", "relation_to_broader_scientific_literature": "FGOD is a practical yet underexplored problem in the ML community. This paper could have a broader scientific impact, such as in fine-grained bird species recognition.", "essential_references_not_discussed": "The related works are generally well-surveyed, including recent studies. The only missing one I noticed is InfoSieve [1], which addresses fine-grained category discovery by unsupervisedly learning codes (or states) for the class tree. However, InfoSieve targets a related but different problem and does not leverage known class structures, distinguishing it from this paper. Therefore, citing it is not necessary.\n\n[1] Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery. NeurIPS 2023.", "other_strengths_and_weaknesses": "I believe the paper addresses an important yet underrecognized problem, and the proposed state representation seems to be an effective approach to solving it. Therefore, I think the paper is worth accepting as a reference for future research.\n\n---\nOne downside is its computational cost, as it aggregates computations over all leaf nodes, making it more expensive than local and global detectors. Discussing inference complexity and including a table comparing computation times across different approaches would be useful.  \n\nWith $N$ leaf nodes in a well-balanced class tree of depth $\\log N$, the computational complexity is $O(\\log N)$ for local detectors, $O(N)$ for global detectors, and $O(N \\log N)$ for this approach, which remains manageable for large-scale classes. In the worst case, where depth is $O(N)$, complexity could reach $O(N^2)$, but most practical cases likely involve a reasonably balanced tree.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by four experts in the field and finally received all positive scores: Weak accept, Weak accept, Accept and Accept.
The major concerns of the reviewers are: 
1.	some details about the training of the method,
2.	the efficiency of the proposed method.
The authors address the above concerns during the discussion period. Hence, I make the decision to accept the paper.
Decision: Accept (poster)