TopoZero: Digging into  Topology Alignment on Zero-Shot Learning
Authors: Yang Liu
Fei Wang
Jiankang Deng
Chen WeiTao
Lei Shang
Baigui Sun
Xuansong Xie
Venue: Submitted to ICLR 2023 (2023)
Keywords: Zero-Shot Learning
Structure Alignment
Persistent Homology

Abstract: Common space learning, associating semantic and visual domains in a common
latent space, is essential to transfer knowledge from seen classes to unseen ones
on Zero-Shot Learning (ZSL) realm. Existing methods for common space learning
rely heavily on structure alignment due to the heterogeneous nature between
semantic and visual domains, but the existing design is sub-optimal. In this paper,
we utilize persistent homology to investigate geometry structure alignment,
and observe two following issues: (i) The sampled mini-batch data points present
a distinct structure gap compared to global data points, thus the learned structure
alignment space inevitably neglects abundant and accurate global structure
information. (ii) The latent visual and semantic space fail to preserve multiple
dimensional geometry structure, especially high dimensional structure information.
To address the first issue, we propose a Topology-guided Sampling Strategy
(TGSS) to mitigate the gap between sampled and global data points. Both theoretical
analyses and empirical results guarantee the effectiveness of the TGSS.
To solve the second issue, we introduce a Topology Alignment Module (TAM)
to preserve multi-dimensional geometry structure in latent visual and semantic
space, respectively. The proposed method is dubbed TopoZero. Empirically, our
TopoZero achieves superior performance on three authoritative ZSL benchmark
datasets.

Under review as a conference paper at ICLR 2023
TOPOZERO: DIGGING INTO TOPOLOGY ALIGNMENT ON ZERO-SHOT LEARNING
An Anonymous authors Paper under double-blind review
ABSTRACT
Common space learning, associating semantic and visual domains in a common latent space, is essential to transfer knowledge from seen classes to unseen ones on Zero-Shot Learning (ZSL) realm. Existing methods for common space learn- ing rely heavily on structure alignment due to the heterogeneous nature between semantic and visual domains, but the existing design is sub-optimal. per, we utilize persistent homology investigate geometry structure alignment, and observe two following issues: (i) The sampled mini-batch data points present distinct structure gap compared to global data points, thus the learned struc- ture alignment space inevitably neglects abundant and accurate global structure information. (ii) The latent visual and semantic space fail preserve multiple dimensional geometry structure, especially high dimensional structure informa- tion. To address the first issue, we propose Topology-guide Sampling Strategy (TGSS) to mitigate the gap between sampled and global data points. Both the- oretical analyses and empirical results guarantee the effectiveness of the TGSS. To solve the second issue, we introduce a Topology Alignment Module (TAM) preserve malti-dimensional geometry structure latent visual and semantic space, respectively. The proposed method is dubbed TopoZero. Empirically, our TopoZero achieves superior performance on three authoritative 7SL. benchmark datasets.
this pa-
INTRODUCTION
Given large amount of training data, deep learning has exhibited excellent performance on various vision tasks, e.g., image recognition He et al. (2016); Dosovitskiy et al. (222)) object detection
Lin al. (2017); Liu et al.] (202T). and instance segmentation He et al. (2017 7): Bolya et al.] (2019). However, when considering more realistic situation, e.g., the testing class does not appear at the training stage, the deep learning model fails to give prediction on these novel classes. To remedy this, some pioneering researchers Lampert et al. (2014); Mikolov et al. (2013) point out that the auxiliary semantic information (sentence embeddings and attribute vectors) is available for both and unseen classes. Thus, by employing this common semantic representation, Zero-Shot Learning (ZSL) was proposed to transfer knowledge from seen classes to unseen ones.
Common space learning, enabling a significant alignment between semantic and visual information on the common embedding space, is mainstream algorithm for ZSL. Existing approaches for com- mon space learning can be divided into categories: algorithms with distribution alignment and 2) structure and distribution alignment. Typical methods in the first category employ various encoding networks directly align the distribution between visual and semantic domains, e.g., vari- ational autoencoder in SSchnneeld et al. (201)). bidirectional latent embedding framework in Wang Chen (2017), and deep desual-semante embedding network in Tsai al. (2017). Even though these methods encourage distribution alignment between visual and semantic domains, the align- ment on the geometry structure is usually neglected. Note that the structure gap naturally exists in these two domains due to their heterogeneous nature Chen al. (2021c). To mitigate the structure gap for promoting alignment between visual and semantic domains, HS VA Chen et al. (2021c) was proposed and become pioneering work in the second category. Inspired by the successful structure alignment work Lee et al. (2019) in unsupervised domain adaptation, HSVA introduces a novel hier- archical semantic- viiuall adaptation framework to align the structure and distribution progressively.

---

Under review as a conference paper at ICLR 2023
(m))
X
Global data points
Sampled data points Random incremental data point
TGSS incremental data point
X
XR
(m++))
X(met) x(m)
D D (((mm
(m
Dy (X (m,x())))) (m+1))
[a] Random sampled m data points
(b) Random Incremental data point
(c) TGSS incremental data point
Terther Jentr
325
452
Aliva 0-dim topalogical feature
(o) Allvve 1-dim topological feature
feature
Allvi 2-dim toallloooggii
Figure 1: Motivation Illustration. (a)-(c) Based the same random sampled data points (m)
in (a), the sampled batch data points from our Topological-guidee Sampling Strategy (TGSS) closer to the global data points compared to those sampled from random sampling strategy (DH (X, x < Dy (X, x x Combining this illustrator example with our theoretical analysis guarantees that our TGSS mitigate the structure gap between mini-batch and global data points. (d)-(f) Compared to the input space, HSVA latent space can only preserve O-dimensional topological features, indicating some high dimensional structure representation is lost during the dimension reduction phase. In contrast, our TopoZero latent space can preserve more accurate topo- logical features by taking advantage of our proposed Topology Alignment Module.
Although HSVA empirically works well, we discover that there exist two issues in HSVA's structure alignment module. To clarify our findings clearly, we first introduce some background information tool
in terms of Persistent Homology Zomorodian & Carlsson (2005). Persistent homology for computing topological features" of a data set at different spatial resolutions. More persistent features can be found over wide range of spatial scales and represent true features of the underlying geometry space. We first introduce the concept of simplicial homology. For a simplicial complex R, i.e. a generalised graph with higher-order connectivity information such as cliques, simplicial homology employs matrix reduction algorithms assign R family of groups, namely homology groups The d-th homology group HR(R) of R contains d-dimensional topological features, such as connected components (d = 0), cycles/tunnels (d 1), and voids (d = 2). Homology groups typically summarised by their ranks, thereby obtaining a simple invariant signature of a manifold. For example, circle in R22 has (a connected component). Based on these background knowledge, we further introduce how to compute a Persistent Homology when given point cloud X. Firstly, we denote the Vietoris-Rips complex Vietoris (192)) of X at scale as (X.). Then, we can obtain the Persistent Homology PH)))(X))of Vietoris-Rips complex (X)., which consists of persistence diagrams {D1, Dz, ...) and persistence pairs (mI, T2, The d-dimensional persistence diagram Da contains coordinates with the form (a, b), where a refers to a threshold e at which d-dimensional topological feature appears and b refers to a threshold e' at which it disappears. The d-dimensional persistence pairs contains indices (i.j) corresponding to simplices Sj.8j E VX()), which create and destroy the corresponding topological features determined by (a. E Da. Note that more detailed background knowledge (e.g., simplex, Vietoris-Rips complex) is introduced in Section A.
feature with d 1 (a cycle), and one feature with d =
'Connectvviy- hased features, e.g., connected components in O-dimensional, cycles in I-dimensional, and voids 2-dimensional topological features
2

---

Under review as a conference paper at ICLR 2023
Based on the powerful geometry feature analysis ability of Persistent Homology, we discover 2 problems in the existing sfate-of-the-art (sota) structure alignment module [Cenn et al. (2021c): (i) Due to the limitation of batch size, the underlying geometry structure of mini-batch samples can not represent global samples'. Thus, when applying structure alignment metric (i.e., sliced Wasserstein discrepancy [ee et al. (2019)) on random sampled H mini-hatch visual and semantic data points, we can only achieve a local-level structure alignment, indicating the accurate global geometry in- formation is lost inevitably. (ii) HSVA utilizes sliced Wasserstein discrepancy to align latent visual and semantic space for bridging structure alignment. Actually, this implementation requires an as- sumption that the latent visual and semantic space can represent their underlying geometry structure adequately. To verify the correctness of this assumption, we adopt persistent homology to visualize the underlying geometry structure of input space and latent space on the visual domain. As shown in Fig. [d) ( (f), there is distinct gap between the blue dash line and the orange dash line, which is further expanded in the latter two images, representing that the HSVA latent visual space loses abundant geometry structure, especially for I-dimensional and 2-dimensional topological features. The rationale is that after dimensionality reduction (namely curse of dimensionality Wang & Chen (017) ), the topological structure is difficult to maintain.
In this paper, we devise a TopoZero framework to achieve a more desirable structure alignment by solving 2 aforementioned issues. Concretely, our TopoZero adopts CADA-VAE [Schönfeld et al.. (2019) as the distribution alignment module and develops a Topology Alignment Module (TAM) with 2 following novelties. (i) To alleviate the structure gap between the sampled mini-batch data points and global data points, we propose a Topology-guided Sampling Strategy (TGSS) to ex- plicitly and progressively mine the topology-preserving data point into the sampled mini-batch data point. Moreover, the theoretical analysis illustrated in Section A guarantees the advantage of our TGSS. Besides, as shown in Fig. (b) (c), we further visualize the advantage of our TGSS in an illustrator example: based on the same random sampled data points X X(m), xx (mm++) and
(m++)) are constructed by our TGSS and random sampling strategy, respectively. Obviously, the Hausdorff Distance DH, X/XX xx(++1) between ((=+1) and global data points X is bounded by (T,xx((+))) indicating our TGSS can alleviate the gap between sampled data points and global DH(X,, ( R
data points compared to random sampling strategy. (ii) To preserve the topological structure for vi- sual and semantic latent space, we develop a dual topological-awanl branch as well as a topological- preserving loss to learn a topological-invarian
latent representation. Moreover. based on the open- source tool Ripser we compute the persistent homology to analyze the multi-dimensional topo- logical features from input space, HSVA latent structure space, and TopoZero latent structure space on the visual domain. Given a set of data points, ripser can compute the corresponding persistent homology, which consists of persistence diagrams [ ...} and persistence pairs (D1, D2. Thus based on the obtained persistence diagrams and persistence pairs, we can calculate the num- ber of alive 0/1/2-dimensional topological features under different threshold e. As such, we draw the Fig. [ (d)-(f), where the line represents the trend of the number of alive topological features under different threshold e. As revealed from these visualization results, by taking advantage of our proposed TAM, the dileeiiimmesional topology feature gap between our TopoZero latent space and input space is negligible.
2 RELATED WORKS
Zero-Shot Learning. In recent years, the ZSL realm has attracted many researchers' attention
Zhang & Saligramal 2016); Li et 2017 Zhu al. (2019)); Fu al. (0155): Ye & Guo (2017):
Yu & Lee (2019)); Chen et al. (018)) 8). One typical branch to solve the ZSL problem is learn- ing common embedding space for aligning semantic and visual domains, termed common space
learning. Early common space learning methods focus on framework designation for better distribu- tion alignment. Wang et al. Waang& Chen (2017) have proposed a bidirectional latent embedding framework with two subsequent learning stages. [iu et al. (2018) maps visual features and seman- tic representations of class prototypes into a common embedding space to guarantee the seen data is compatible with seen and unseen classes. CADA-VAE Liu et al. (2018) have demonstrated that
"Exittngg methods all adopt random sampling strategy generate mini-batch data points. 3 A metric that can measure the bounded distance hetween two persistence diagrams.
Available https: / thub. com/ ki pser/ riser
3

---

Under review as a conference paper at ICLR 2023
only variational autoencoders well distribution alignment loss, can achieve significant distribution alignment in common space. However, as pointed out from HSVA Chen et al. (202lc), due to the heterogeneous nature of the feature representations in semantic and visual domais, the distribution and structure variation intrinsically exists. Motivated by this, Chen et al. CChen et al.) (2021c) propose hierarchical semantic-visua adaptation framework for aligning structure and dis- tribution progressively. Thus, the structure alignment in ZSL emerges with a new state-of-the-art performance on the task of common space learning.
Persistent Homology. Persistent homology, a tool for topological data analysis, is used for un- derstanding topologiaa features at different dimension. Concretely, persistent homology can detect multi-dimensional topological features (holes, circles, connected components) under various dimen- sions for the underlying manifold of set of sampled data points. Based on this property, persistent homology has been applied to a vast body of scenarios, e.g., characterizing graphs in Archambault al. (2007); Carrière et al. (002)): Li et al. (2012), analysing underlying manifolds |ae et al. 2017): Futagami et al. (2019), topological preserving autoencoder in Moor et al. (2020). In this paper, by leveraging persistent homology, we discover that the latent visual and semantic space can not preserve dineeiiiimnsi topological features. Furthermore, to improve the geometry repre- sentation of latent space in both domains, we propose a Topology Alignment Module for encoding multi-dimensional topological representation explicitly.
METHODOLOGY
To begin with, we formulate the task of ZSI.. Assume have set of seen samples S for training, and a set of unseen samples for testing only, where S = [(x","", "²")""))"""""""""""" ) xx E xx* E Y Y²,a* E A) be training set. x: is seen image feature, which is extracted from the pre-trained CNN backbone (ResNet-101 101 He et al. (2016) is adopted in this paper). y" and a" are x² corresponding class label and semantic vector, respectively. Analogously, let that Y n Y = Ø. The objectiveness of conventional ZSL (CZSL) is to learn a classifier for mapping unseen image features into unseen categories, i.e., Fessi X + )" while the challenging generalized ZSL (GZSL) focus on learning classifier to map image features to both seen and unseen categories, i.e., Fozst
{(x" y x=" X" y" Y"). Note
> )" Y
As shown in Fig. 2 our TopoZero contains two parallel alignment modules, Distribution Alignment Module and Topology Alignment Module Specifically, we directly adopt the architecture of CADA- VAE [Schnnfeld al. (2019) guided sampling strategy and dual topological-awant branch are proposed to mitigate the geometry structure gap between mini-batch and global data points and preserve multi-dimensional topological structure on both visual and semantic domains, respectively.
Distribution Alignment Module. While for our TAD, topology-
3.1 TOPOLOGY-GUIDED SAMPLING STRATEGY
To bridge a structure gap between mini- and global data points, we propose a Topology-guides Sampling Strategy (TGSS) as well as a theoretical analysis to guarantee its superiority.
3.1.1 DESCRIPTION
Algorithm [ describes how our TGSS samples mini-hatch samples from global data points. First, we random sample 6/2 data points (Xs2) from global training samples (X). After that, we select the incremental data point Imaa according to Equ. [ Then, we construct a set of candidate set (C) by Equ. 2 and random sample b/2 data points from form Cyini- Finally, the mini-batch sampled data points are constructed by integrating X82, Zmaz and Cmini. The advantage of our TGSS relies heavily on the selection of Imazs which proved by the following theoretical analysis.
Emaax E X, x'aax E X62, s.t. dist(Fmar xmaz) = d22)
(1)
C(Emary d) = (fro. Ix. a X, TT dist(ri, xmaa) < d)
(2)
"h represents the size of hatch training samples
4

---

Under review as a conference paper at ICLR 2023
ET
x
CREC
CNN
LRBO
Et
D
Cuce
x
Red Logs Gray Wings
Ed
Dd
Coa
White Head
Long Wing Shape
Loce
Ed
Dd
Figure 2: The proposed TopoZero framework. Based on our proposed TGSS sampling strategy, we can obtain a batch of visual features x and corresponding semantic embeddings Then æ and are fed into the parallel topology alignment module and distribution alignment module. For TAD, the encoder F% and F first encode æ and a to get latent topological visual representation 2 and topological semantic representation 2/g respectively. Then the decoder D' and D decode 2 and zt to reconstruct visual and semantic feature, which is optimized by reconstuction loss A and LAE- The visual and semantic latent topological representation is optimized by Ltp and L4p to preserve multi-dimensional structure information. LT is also applied to align 2 and 2 For the distribution alignment module, we adopt the framework of CADA-VAE, which consists of two variational autoencoders and optimized by LACF, LK LDA, and LCA.
where T denotes a set of sampled data points from X and dist represents distance metric (Eu- clidean Distance in this paer). du refers to the Hausdorff distance Hutteelocher et al. (1993) between X and Xb2- Then, we revisit the definition of Hausdorff Distance that dn( Y) max { supzex d(xr, Y), supygy d(X,y) 3)] }. which measures how far two subsets of metric space are from each other. Informally speaking, the Zmar represents the farthest data point in X to the sampled X52 when adopting Hausdorff Distance metric. Thus, by integrating the Zmaz into X12 the Hausdorff Distance between X and X32 can be reduced, indicating the gap between sampled and global data points is also mitigated according Theorem [ advantage of our TGSS relies heavily the selection of Imazz provide theoretical analysis to guarantee its superiority. Besides, the introduction of Cmini is to maintain the representation of local topology structure surrounding from Tmaz.
Moreover, considering that the
3.1.2 THEORETICAL ANALYSIS FOR TGSS
The core design of our TGSS is the procedure of selection 2mar (line 4 in Algorithm I which can eliminate the structure gap compared to random sampling strategy. Here, we further pro- vide a theoretical analysis to guarantee the advantage of this selection procedure. Before we carry out our analysis, we define a few important definitions and notations. For a point cloud X == fai...., >m> C Rd. denote (m) be a subsample of X with cardinality m. Based on X(m) and the procedure of TGSS's selection Imans the constructed set is denoted as x (m+1) While for random sampling strategy, we have X x(m+1), Thus, we have:
x x(m+1) T ={ {x(m) Uæ, x Imaz)
(3)
x (m+1)) (x((m) Uz, E X X((mm)
(4)
where Vmar is defined Equ. 0
Theorem 1. Moor et al. (2020). Let X be a point cloud of cardinality n and (m) be one subsample of X of cardinality m, i.e. (m) C X, sampled without replacement. We bound the probability
of the persistence diagrams of X exceeding a threshold in terms of the bottleneck distance as
P(d,(DX, ( x > S ((dn(XXXX)) x())) 22)
(5)
5

---

Under review as a conference paper at ICLR 2023
Algorithm Topology-guidedd Sampling Strategy
Input:
X set of whole training samples.
b the size batch training samples.
Output:
T is prepared mini-batch samples for an epoch.
1: init T:T
2: for each iteration in epoch do
random sample training data from X'T: (T: Xx2;
4: compute the incremental data point Smaan by Equ. 11
5: construct set of data points by Equ. 2 C C(Zmazy du (Xuz, X));
6: Xuz X12 U Imaxi
len(C) < b/2 then
M random select b/2 Ien(C) data points from X/ X
9:
Xb2 = X,2 UCUM;
10:
else
11:
M random select 6/2 data points from C:
X12 X1z MM;
13:
end if
14:
T = TUUXs
15:
end for
16: return
Theorem 2. Let A xxx(/)))) R=x(m+1) be the distance matrix between samples of X and x(m+1) and A xxx(++11 E R=x(m++) be the distance matrix between samples of X and x (m)+1) The XT x(m+t) and XR (m++)) are both sorted to ensure that the first (m+1) )) rows correspond to the columns the m subsampled points with diagonal elements au 0. Assume that the entries aij in both matrix are independent and follow a same distance distribution FD when i > (m + 1). For Ay x(m+s), the minimal distances 6, for rows with i > (m + follow distribution Letting 7' maxigigu 64 with corresponding distribution Fzz- For Ax x,xin xdm the minimal distances 5, for rows with > (m + follow a distribution Fx" Letting z" == maxisisn 8 with a cor- responding distribution Fz the expected Hausdorff distance between X and X x(m+1) is bounded by:
Exd X (
(m(m++))) EXdu(X, (m++1))
(6)
We include its proof in Section A Theorem. illustrates that compared to random sampling strategy (x(m+1)), the sampled batch data points (X T (+++)) from our TGSS closer the global data points with Hausdorff Distance metric, which constitutes the upper bound bottleneck distance between persistence diagrams (Theorem [ Thus, since bottleneck distance is usually used to measure the distance between two persistence diagrams in the topological space [Beketayev et al. conclude that compared to random sampling strategy, the
(014); [ubbenik al. (2010), sampled batch data points from our TGSS are closer to the global data points in the topological space.
3.2 TOPOLOGY ALIGNMENT MODULE
As shown in Fig. (a)-(c), HSVA, a state-of-the-art common space learning method by taking structure alignment into account, fails to preserve multi-dimensional topological features. Specifi- cally, the terrible structure representation in the latent space inevitably leads sub-optimal struc- ture alignment. To remedy this, we propose a Topology Alignment Module, consisting of a dual topology-awan branch and formation into latent visual and semantic space for conducting more desirable structure alignment.
loss, to encode multi-dimensional topological in-
Our Dual Topology-awan aware Branch is illustrated in Fig. which contains two autoencoders for obtaining topological-awe aware latent representation in visual and semantic domains. Specifically, the encoder Ft, / F encodes image feature (x) semantic vector (a) into latent space and obtain visual

---

Under review as a conference paper at ICLR 2023
and semantic topological-aware latent representation Zaam) and Z After that, the decoder D2 / D2 decodes 7 Zz(m) for reconstructing the latent representation into x / a. We first apply reconstruction loss optimize our Dual Topology-aware Branch:
AA LREC = (E) - x
A LREC = (D - all
Then we utilize the topology-preserviigg loss proposed by Moor et al. (020) to preserve multiple dimensional topological features the latent visual and semantic space, which is calculated by the following steps: 1) Given a batch of visual feature X and semantic embeddings X(m) our dual lopology-aware branch can obtain corresponding latent representation, Z!m² and _(m² 2) We cal- culate the distance matrix between samples of X!m) and x!!),, termed x(()) The corresponding (in)
persistent homology of (mm) is recorded as )x(")) = (D* .T x Analogously, for x!m), z(m) and %(("),
obtain corresponding distance matrix x((m)s A Az(m)) and A Az)))
?=a" 3) Finally, we retrieve the value of O-dimensional / I-dimensional / 2-dimennional persistence diagram from distance matrix with indices provided by pxx!"
persistence pairings π xx" π and π
the persistence pairings, namely Do
A ((m)
PD
(nn)
Through this computation process,
get the 0/1/2 -dimensional persistence diagrams in ( x(m) _!! and z(!m), which are opti-
mized by the following presseevigg loss:
LTP
Σ k=0
D
X(m)
D 11?
CTP
Σ
1
Xmm)
D
1
(10)
0m0
Finally, to encourage interaction between visual and semantic domains in the topological space, we directly minimize the 1.2 distance between latent visual topological representation and latent semantic topological representation:
CTA = ( Z(m) 112
(1I)
(12)
3.3 DISTRIBUTION ALIGNMENT MODULE
Since CADA- VADA Schonfeld et al. 2019) serves as our distribution aligument module, we directly revisit it in framework. Our distribution alignment module adopts variational autoencoders Kingma & Welling (2014) to obtain latent representation in visual and semantic domains, respec- tively. Concretely, the encoder Ed Ed encodes image feature (x) / semantic vector (a) into latent space and obtain visual and semantic latent representation 2d and 2d. Then, the decoder Dd / D decodes 2 za for reconstructing the latent representation into x a. We apply standard VAE loss to optimize:
AV LBCE 3LKL Egg(w) log (x 3D (Dx (E
(13)
AV = LBCE - 3LKI. = ( [log (x 3D (E:
(14)
where DKL represents the Kullback-Leibler divergence and p(z) a prior distribution (standard Gaussian distribution (00 1) in this paper). The binary cross-entropy loss LECE is served as the reconstruction loss. Following Schonfeld et al. (2019). 3 serves as the balanced weight to measure the importance of DKL-
Distribution alignment loss is formulated as:
LDA - 11 11 (0))
(8
(15)
'Due the page limited, we provide more detailed computation process in Section

---

Under review as a conference paper at ICLR 2023
where || || is the squared matrix Frobenius norm, and cross- cross-alignment loss is formulated as:
Cc = = (a)(())
(16)
Cc - la
(17)
3.4 TOPOZERO OBJECTIVE FUNCTION
Our TopoZero is optimized by the following objective function:
Zopozero - + +1 (Cip LTp) + x2 * LTA + + (CCA + CC + LİVE + LV (DA)
(18)
where Åι X2, and λ are the balanced weight to measure the importance of each module in our TopoZero. the branch of TAM, Ae and Li aim to obtain latent visual and semantic representation. Lip and Lir assist the latent visual and semantic representation to preserve multi- dimensional topology structure. LTA associates semantic and visual latent representation a com- mon space. While for the branch of distribution alignment module, all the objective functions keep the same with those in CADA-VAB Zhu et al. (2019)).
3.5 ZERO-SHOT PREDICTION
After the optimization of TopoZero, we need to train Fozst and Fezst for predicting unseen or image features 2:
samples. Given
obtain the latent distribution representation E with reparametrization trick Kingma & Welling (2114) and topological representation E (x*). Analoguously, for unseen image semantic vector a" have = and = Then dd
concatenate zg* and =ge- ([= x²x-) to serve as seen training data and ((=g,, for unseen one.
After training F Fyzel and Fesel, we use [2dt 2xx and [2 2 inference.
EXPERIMENTS
In this section, we first claborate on implementation details and authoritative benchmark datasets in the field of ZSL. Then compare TopoZero with existing state-of-the-art ZSL methods. Finally, we provide some qualitative and quantitative analysis to illustrate the advantage our TopoZero. Due the limitation of page size, several parts are placed on Appendix
4.1 DATASETS AND IMPLEMENTATION
Datasets. We verify TopoZero popular ZSL benchmark datasets, including CUB Welinder et al. (2010), SUN Patterson & Hays 2012), and AWA2 Xian et al. (2018a). CUB contains 11788 images of 200 bird classes (seen/unseen classes 150/50) with 312 attributes. SUN consists of 14340 images from 717 classes (seen/unseen classes 645/72) with 102 attributes. AWA2 includes 37322 images of 50 animal classes (seen/unseen classes 40/10) with 85 attributes. Finally, we adopt the "split version 2.0" mode Xian et al. (2018h) to conduct data splits on CUB, SUN, and AWA2.
Network Architecture. As illustrated in Fig. 2 our TopoZero contains 2 Encoders and 2 Decoders, which are basic Multi-Layer Perceptions with 2 fully connected (FC) layers and 4096 hidden units. The dimension of latent variable in the distribution alignment and topology alignment module are both set 64. The architecture CZSL and GZEL classifier is a single FC layer.
Optimization Details. Our TopoZero is optimized by Adam optimizer with initial learning rate 10 The total training epoch ZopoZero is set 100 with a batch size 50. For training final CZSL and GZSL classifiers, the training epoch, batch size, and initial learning rate are set 25, 28, 10-3 respectively.
Evaluation Protocols. Following the standard evaluation protocol Xian al. 2018a). TopoZero evaluated by the top-1 accuracy. For CZSL, we only compute the accuracy on unseen classes. While for GZSL, both calculate the accuracy of seen and unseen classes.
8

---

Under review as a conference paper at ICLR 2023
Results (%) of the state-of-the-art models on CUB, SUN and, AWA2 datasets. The best result is masked in bold. The symbol indicates available result.
Table 1:
CUB
SUN
AWA2
Methods
CZSL
GZSL
CZSL
GZSL
CZSL
GZSL
H
Non Common Space
QFSL Song al.
2018
58.8
33.3
48.1
39.4
56.2
30.9
52.1
72.8
60.7
L.F
67.5
26.4
81.6
39.9
65.55
9.8
87.4
17.6
SGMA Zhü al A196
71.0
36.7
71.3
48.5
68.8 8
37.6
87.
52.5
al
38.9
78.7
LFGAA
Liü ml
2015
67.6
36.2
80,9
50.0
61.5
18.5
40.0
25.3
68.1
27.0
93.4
41.9
SP-AEN Ten el 1l.
201N
55.4
34.7
70.6
46.6
59.2
24.9
38.6
30.3
58.5
23.3
90.9
37.1
POZSL
43.2
51.4
46.9
35.1
35.3
35.2
31.7
70,9
43.8
CRNe Zhang & Shi 2019
45.4
56.8
50.5
34.1
36.5
35.3
Cacheux al.
30.4
DVBE Min
2020
53.2
60.2
56,5
45,0
37.2
40.7
63.6
70.8
67.0
FREE Chen et
271hh
55.7
59.9
57.7
47.4
37.2
41.7
60.4
75.4
67.1
Space
DeViSE Frome al. ReViSE Tsa et a. 2017
2013
23.8
53.0
32.8
16.9
27.4
20.9
17.1
74.7
27.8
37.6
28.3
32.3
24.3
20.1
22.0
46.4
39.7
42.8
DCN
28.4
60.7
38.7
25.5
37.0
30.2
SGAL Yu & Lec
9019
44.7
47.1
45.9
31.2
42.9
36.1
81.2
55,1
65.6
CADA- VAB Schomleta
2019
$7.9
51.6
53.5
52.4
53.5
52.4
DOE-ZEL Geht
66.4
57.6
VGSE Xu aral, 2022
56.8
24.1
45.7
31.5
41.1
25.5
35.7
29.8
66.7
45.7
66.7
54.2
HSVA Chen
62.8
52.7
58.3
55.3
63.8
59.3
76.6
66.8
TopoZero (Ours)
64.3
54,9
59.9
57.3
64.7
49.4
40.9
44.7
70.6
59.1
80.0
68.0
For determining the performance of GZSL in unified criterion, the harmonic mean (defined as H = (2 x x US S + U)) is adopted in this paper.
4.2 COMPARISON WITH STATE-OF-THE-AAE
Results on Conventional Zero-Shot Learning. Tab. Treports the CZSL results our TopoZero and recent state-of-the-art (sota) methods 3 ZSL datasets. Considering that attribute-based sota methods Huynh & Elhamifar (2020); Chen et al. (202 exploit the advantage of pre-trained NLP models GloVE and generation- based sota methods Xian et al. (2018b): Yu et al. (2020) take advan- tage of data augmentation, methods involving these branches are not taken into account in this part. Compared to methods only with distribution alignment, our TopoZero illustrates significant improvement of 6.4%, 3. %6, and 8.0% CUB, SUN, and AWA2 datasets at least. While compared to HSVA Chen et al. (202lc) with distribution and structure alignment, our TopoZero also achieves great improvement of 1.5%, 0.9% CUB and SUN datasets, respectively. Such a significant performance directly verifies the effectiveness of topology alignment for ZSL task.
Results on Generalized Zero-Shot Learning. By looking at the challenging GZSL results in Tab. 2. our TopoZero also achieves a dominant harmonic mean performance of 57.3%, 44.7%, and 68.0% on CUB, SUN, and AWA2 datasets, respectively. Both superiority results of Topo CZSL and GZSL settings demonstrate that our TopoZero is better than HSVA on structure alignment.
CONCLUSION
In this paper, we propose TopoZero framework improve structure alignment for common space learning methods. To begin with, we discover that existing structure alignment approaches confront two challenging issues: 1) sampled mini-batch data points present distinct gap compared to global ones; 2) latent visual and semantic space lose some nigh-dimensional structure information due to the 'curse of dimensionality." To solve these two problems, Topology-guides sampling strategy and Topology Alignment Module are proposed to construct our TopoZero. Furthermore, provide a theoretical analysis as well as visualization results to guarantee the advantage of our TopoZero, namely excellent multi multi-dimensionall The extensive and superior experiment results demonstrate that our TopoZero has a great potential to advance the ZSL community.
and aoplogy-aliigmmet ability. Finally,

---

Under review as a conference paper at ICLR 2023
REFERENCES
Daniel Archambault, Tamara Munzner, and David Auber. Topolayout: Multilevel graph layout by topological features. IEEE transactions on visualization and computer graphics, 13(2):305-317, 2007.
Woong Bae, Jaejun Yoo, and Jong Chul Ye. Beyond deep residual learning for image restoration: Persistent homology-guider manifold simplification. In Proceedings of the IEEE conference computer vision and pattrrn recognition workshops, pp. 145-153, 2017.
Kenes Beketayev, Damir Yeliussizov, Dmitriy Morozov, Gunther H Weber, and Bernd Hamann. Measuring the distance between merge trees. In Topological Methods in Data Analysis and Visu- alization III. pp. 151-165. Springer, 2014.
Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9157-91 2019.
Peter Bubenik, Gunnar Carlsson, Peter T Kim, and Zhi-Ming Luo. Statistical topology via morse theory persistence and nonparametrie estimation. Algebraic methods in statistics and probability 11, 516:75-92, 2010.
Yannick Le Cacheux, H. Borgne, and M. Crucianu. Modeling inter and intra-class relations in the triplet loss for zero-shot learning. In ICCV, 2019.
Mathieu Carrière, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda. Perslay: neural network layer for persistence diagrams and new graph topological signatures. In International Conference Artificial Intelligence and Statistics, pp. 2786-2796. PMLR, 2020.
Long Chen, Hanwang Zhang, Jun Xiao, W. Liu, and S. Chang. Zero-shot visual recognition using semantics semaanics-preservingg adversarial embedding networks. In CVPR, pp. 1043-1052, 2018.
Shiming Chen, Ziming Hong, Guo-Sen Xie, Jian Zhao, Xinge You, Shuicheng Yan, and Ling Shao. Transzero++: Cross attribute-guided transformer for zero-shot learning. arXiv preprint arXiv:2112.08643, 2021a.
Shiming Chen, Wenjie Wang. Beihao Xia, Qinmu Peng, Xinge You, Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learning. In ICCV, 2021b.
Shiming Chen, Guo-Sen Xie, Yang Yang Liu, Qinmu Peng, Baigui Sun, Hao Li, Xinge You, and Ling Shao. Hsva: Hierarchical semantic-visual adaptation for zero-shot learning. In NeurlPS, 2021c.
Shiming Chen, Ziming Hong, Yang Liu, Guo-Sen Xie, Baigui Sun, Hao Li, Qinmu Peng, Ke Lu, and Xinge You. Transzero: Attribute-guide transformer for zero-shot learning. AAAI, volume 2, pp. 3, 2022a.
Shiming Chen, Ziming Hong, Guo-Sen Xie, Wenhan Yang, Qinmu Peng, Kai Wang. Jian Zhao, and Xinge You. Msdn: Mutually semantic distillation network for zero-shot learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7612-7621, 2022b.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mathhis Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 11999, 2020.
Andrea Frome, G. S. Corrado, Jonathon Shlens, Bengio, J. Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In NewrIPS, 2013.
Zhenyong Fu, Elyor Kodirov, Tao Xiang, and S. Gong. Zero-shot object recognition by semantic manifold distance. In CVPR, pp. 2635-2644 2015.
10

---

Under review as a conference paper at ICLR 2023
Rentaro Futagami, Noritaka Yamada, and Takeshi Shibuya. Inferring underlying manifold of data by the use of persistent homology analysis. In International Workshop on Computational Topology in Image Context, pp. 40-53. Springer, 2019.
Yuxia Geng, Jiaoyan Chen, Wen Zhang, Yajing Xu, Zhuo Chen, Jeff Z. Pan, Yufeng Huang, Feiyu Xiong, and Huajun Chen. Disentangled ontology embedding for zero-shot learning. In Pro- ceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp- 443-453, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Daniel P Huttenlocher, Gregory A. Klanderman, and William J Rucklidge. Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence, 15(9): 850-863, 1993.
D. Huynh and E. Elhamifar. Fine-grained generalized zero-shot learning via dense attribute-based attention. In CVPR, pp. 4482 4492, 2020.
Diederik Kingma and M. Welling. Auto-encoding variational bayes. In ICLR, 2014.
Christoph H. Lampert, H. Nickisch, and S. Harmeling. Attrbbte-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 06:453-465, 2014.
Chen-Yu Yu Lee, Tanmay Batra, M. H. Baig, and Daniel Ulbricht. Sliced wasserstein discrepancy for unsupervised domain adaptation. In CVPR, pp. 10277-10287, 2019.
Geng Li, Murat Semerci, Bülent Yener, and Mohammed J Zaki. Effective graph classification based on topological and label attributes. Statistical Analysis and Data Mining: The ASA Data Science Journal, 544):265-283, 2012.
Li, X. Lan, Y. Liu, L. Wang, and N. Zheng. Compressing unknown images with product quantizer for efficient zero-shot classification. In CVPR, pp. 5458-5455) 2019.
Y. Li, D. Wang, Huanhang Hu, Yuetan Lin, and Yueting Zhuang. Zero-shot recognition using dual visual-semantic mapping paths. In CVPR, pp. 5207-5215, 2017.
Y. Li, Junge Zhang, Jianguo Zhang, and Kaiqi Huang. Discriminative learning of latent features for zero-shot recognition. In CVPR, 2018.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection, In ICCV, 2017.
Shichen Liu, Mingsheng Long. J. Wang, and Michael I. Jordan. Generalized zero-shot learning with deep calibration network. In NeurIPS, 2018
Yang Liu, Jishun Guo, Deng Cai, and X. He. Attribute attention for semantic disambiguation in zero-shot learning. In ICCV, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Tomas Mikolov, Ilya Sutskever, Kai Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In NewrIPS, pp. 3111-3119, 2013.
Shaobo Min, Hantao Yao, Hongtao Xie, Chaoqun Wang, Z. Zha, and Yongdong Zhang. Domain- aware visual bias climinating for generalized zero-shot learning. In CVPR, pp. 12661-12670, 2020.
11

---

Under review as a conference paper at ICLR 2023
Michael Moor, Max Horn, Bastian Rieck, and Karsten Borgwardt. Topological autoencoders. In International conference on machine learning, pp. 7045-7054, PMLR, 2020.
G. Patterson and J. Hays. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In CVPR, pp. 2751-2758, 2012.
Jeffrey Pennington, R. Socher, and Christopher D. Manning, Glove: Global vectors for word repre- sentation. In EMNLP. 2014.
Edgar Schönfeld, S. Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In CVPR, pp. 8239-8247, 2019.
Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata, Generalized zero-and few-shot learning via aligned variational autoencoders. In CVPR, 2019.
Jie Song. Chengchao Shen, Yezhou Yang. Y. Liu, and Mingli Song. Transductive unbiased embed- ding for zero-shot learning. CVPR, 2018.
Yao-Hung Hubert Tsai, Liang-Kann Kang Huang, and R. Salakhutdinov. Learning robust visual-semantic embeddings. ICCV, pp. 3591-3600, 2017.
Leopold Vietoris. Über den höheren zusammenhang kompakter räume und eine klasse von zusam- menhangstreuen abbildungen. Mathematische Annalen, 97(1):454-477 1927.
Q. Wang and Ke Chen. Zero-shot visual recognition via bidirectional latent embedding, Interna- fional Journal of Computer Vision, 124:356-388 2017.
P. Welinder, S. Branson. T. Mita, C. Wah. Florian Schroff, Serge J. Belongie, and P. Perona. Caltech- uesd birds 200. Technical Report CNS-TR-2010-001, Caltech, 2010.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. TPAMI, 2018a.
Yongqin Xian, T. Lorenz, Schiele, and Zeynep Akata, Feature generating networks for zero-shot learning. In CVPR, pp. 5542-5551, 2018b.
Guo-Sen Xie, L. Liu, Xiaobo Jin, F. Zhu, Zheng Zhang, J. Qin, Yazhou Yao, and L. Shao. Attentive region embedding network for zero-shot learning. In CVPR, pp. 9376 9385, 2019.
Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Vgse: Visually-grounded semantic embeddings for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9316-9325, 2022.
Meng Ye and Yuhong Guo. Zero-shot classification with discriminative semantic representation learning. CVPR, 2017.
H. Yu and B. Lee. Zero-shot learning via simultaneous generating and learning. In NeurlPS, 2019a.
Hyconwoo Yu and Beomhee Lee. Zero-shot learning via simultaneous generating and learning. arXiv: 1910.0946, 2019b.
Y. Yu, Zhong Ji, Han, and Z. Zhang. Episode-based prototype generating network for zero-shot learning. In CVPR, pp. 14032-14041, 2020.
Zhang and G. Shi. Co-representation network for generalized zero-shot learning. In ICMI., 2019.
Ziming Zhang and Venkatesh Saligrama. Zero-shot learning via joint latent similarity embedding. In CVPR, pp. 6034-6042, 2016.
Pengkai Zhu, Hanxiao Wang, and Venkatesh Saligrama. Generalized zero-shot recognition based on visually semantic embedding. CVPR, pp. 2990-2998, 2019a.
Yizhe Zhu, Jianwen Xie, Z. Tang, Xi Peng, and A. Elgammal. Semantic-guided multi-attente localization for zero-shot learning. In NeurlPS, 2019b.
Afra Zomorodian and Gunnar Carlsson. Computing persistent homology. Discrete & Computational Geometry, 33(2):249-274, 2005.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
B PROOF OF THEOREM! 2
Proof. First, we derive the distribution Fa. (v) and Fa" (v):
Fx (v) P(6, = sy) P(6; > y) - P((
min
oij > v)
(19)
155m+1
= -1- aig v) = 1 (1--1-- (1-(FFb(t) m+1
(20)
j
1 (1 (p()))=t
yy < Bldæ (X, X(m))]
(21)
else
Analogously, we have:
(v) PP66" 5) = =1 - PO6, > v) = 11 -P( aij > u)
(22)
=1 =1- - (1-1111 FD(l)) mT
(23)
(24)
For convenience, we denote 1 - (1 FD(1)) as Fo(y). Next, we derive the distribution (F2)(2) and F2z((2) ) of Z' and z", respectively:
(2) P(Z' 22) = P( max 8 5152== P(
n
8, S2)
(25)
m-lciin
m+lcisn
=f -{
(a(2)(
2 (X, X(m))]
(26)
1
else
Analogously,
(2) = P(z" 5 = P( max
8, 5 = P(
N
8 z)
(27)
m+icisn
m+lcisn
(2
E<d (X,,
(28)
Fa(2) else
Thus, we have:
to0
(1 - Fl))) dz - 1 (2) dz
0
EzinPy [Z7 =
(29)
0
400
(1 (2)
(30)
Eldx ((.
(1 (1-Fe (2)) de f
(1- (2)) dz
(31)
Eldn (X.X(
t00
Jo
(1- F2())m-)) m())) +
(1 - 1)dz
(32)
(X,
(X,
fo
(1 F2(2)=
(33)
13

---

Under review as a conference paper at ICLR 2023
and:
+00
FF EFz
12-1= 1
/
(1 (2))
(34)
(1 (1- (z)) f
t00
(1 Fzz (z)) dz
(35)
(1 Fa (=)=)=mm-1))
F(dn(X,x(mm) /
(1- (-Fa(z)) da (36)
(37)
Finally,
EF of 12'1 Ez ~F z" 12"1= = f +0
(Fa(z) - 1) 00
(38)
(X
=> Ez [6 S Ezz ~F =/ 1="1
(39)
F(ddu(X, Xx,,ximmt x(m+t))] I Eldn(XX (a x(m+1l)]
(40)
C PERSISTENT HOMOLOGY
Here, we further provide several explanations on the definition of simplex, simplicial complex, ab- stract simplicial complex and Vietoris- complex. (a) Simplex: In geometry, simplex is generalization of the notion of a triangle or tetrahedron to arbitrary dimensions. The simplex is so-named because represents the simplest possible polytope made with line segments in any given dimension. For example, a O-simplex a point, a I-simplex is line segment, and a 2-simplex is triangle. (b) Simplicial Complex: topology,
common to "glue together" simplices to form simplicial complex. A simplicial complex is a set composed of points, line segments, triangles, and their n-dimensional counterparts. The strict definition of simplicial complex is that sim- plicial complex K is set of simplices that satisfies the following conditions: 1) Every face of simplex from K is also K; 2) The non-empty intersection of any two simplices σ1, 02 E K is face of both 0 and Ø2. (c) Abstract Simplicial Complex The purely combinatorial counterpart to a simplicial complex is an abstract simplicial complex. (d) Vietoris-Rips complex: In topology, the Vietoris-Rii Rips complex, also called the Vietoris complex Rips complex, is way of forming topological space from distances in a set points. It is an abstract simplicial complex that can be defined from any metric space M and distance d by forming simplex for every finite set of points that has a diameter at most 5. That is, it is a family of finite subsets of M. in which we think of subset of k points forming (A:1) (e1)-dimensional simplex (an edge for points, triangle for three points, tetrahedron for four points, etc.); if a finite set S has the property that the distance between every pair points in S is at most then we include S simplex in the complex. As illustrated in Moor et al. 2020). we can compute the persistent homology of a set of data points X based on this background information.
D COMPUTATION PROCEDURE OF TOPOLOOYYPPESSEVIIGG Loss
Here, we further introduce how retrieve the value of O-dimensional / 1-dimensional 2- dimensional persistence diagram from distance matrix with indices provided by the persistence pairings, namely DXX!" A
x((m)
essence, this retrieving procedure equals to how to select retreival indices from O-dimensional / 1-dimensional 2-dimensional persistence pair- ings. Concretely, for O-dimensional topological features we select the "destroyer" simplices in the O-dimensional persistence pairings. For l-dimensional topological features and 1-dimensional topological features, we regard the maximum edge of the "destroyer simplices" in corresponding persistence pairings as retrieval indices.
To
14

---

Under review as a conference paper at ICLR 2023
E EXPERIMENTS.
E. ABLATION STUDY
Based on the CADA-VALS Schonfeld et al. (2019), we conduct ablative experiments on CUB, SUN, and AWA2 datasets verify the effectiveness of our proposed Topology-guice Sampling Strategy and Topology Alignment Module. We first clarify the notations in Tab. 2 TAD denotes our Topology Alignment Module. TAD, TAD represents preserving O-dimensional/ O-dimensional and 1-dimensional topological features. We can see the 4th row with TAD performs a better result than the 2nd row with 7 D and the 3rd row with (especially high dimensional) structure
Topology Alignment Module with
TADo-1. indicating the effectiveness of alignment. Then, with the addition of TGSS, the performance is further enhanced, demonstrating the TGSS can achieve better structure alignment. This experiment result is highly compatible with our provided theoretical analysis on TGSS.
Table 2: Ablation studies TGSS and TAD on CUB, SUN, and AWA2 datasets.
CUB
SUN
Method
CZSL
GZSL
CZSL
GZSL
CZSL
GZSL
acc
$
H
30c
$
acc
CADA-VAE
57.9
51.6
53.5
52.4
61.6
47.2
35.7
40.6
62.6
51.6
33.5
52.4
CADA-VA VAE TAD,
59.2
51.3
58.8
54.8
61.8
48.3
37.2
42.0
67.4
$5.5
71.3
63.6
CADA-VAE TAD...
60.3
52.7
59.2
55.8
61.9
48.5
38.1
42.7
68.2
56.2
77.3
65.1
CADA-VAE + TAD
62.2
53.7
58.7
56.1
62.5
48.8
39.2
43.5
68.8
58.6
76.9
66.6
TopoZero (CADA-VAE TAD TGSS)
64.3
54,9
59.9
$7.3
64.7
49.4
40.9
44.7
70.6
89.1
80.0
68.0
E.2 ANALYSIS
The effectiveness of TAM. To verify the effectiveness of single Topology Alignment Module, we disentangle it from overall TopoZero framework. As reported in Tab. although single TAM can achieve great performance, there exactly exists a distinct performance gap compared with recent sota methods Chen et al. (2021c This is why we introduce an off-the-sheii distribution alignment module into our TopoZero framework.
Table 3: The effectiveness of single TAM.
Method
CUB
SUN
AWA
TAM
58.4
60.7
64.1
TAM + TGSS
60.5
63.1
66.4
Compatibility with sota ZSL. frameworks. To verify the compatibility between our TAM branch with sota 7SI. framework, we implement our proposed TGSS and TAD the sota open-source method TransZero Chen et al. (2022a). The CZSL results are listed in Tab. After applying the proposed TGSS and TAM, the performance of TransZero increases to 77.6%. 67.2% and 72.6% on CUB, SUN and AWA datasets respectively. This improvement indicates the effectiveness of the proposed method on the sota attribute-basse ZSLL framework. Note that the superiority of TransZero benefits from it utilizes semantic attribute vectors of each attribute learned by GloVe Pennington al. (2014) to improve semantic representation. Through this extra knowledge, recent attribute- based ZSL methods Chen et al. (2021a) 2022b) perform better than others without extra knowledge. Thus, we only verify the compatibility between our TopoZero and Transzero rather than comparing performance directly.
Table 4: Compatibility with sota ZSL
framework TransZero.
Method
CUB
SUN
AWA
TransZero
76.8
65.6
70.1
TransZero + Ours
77.6
67.2
72.6
15

---

Under review as a conference paper at ICLR 2023
ONL
Accuracy 180
Accuracy (16)
Accurscy (%6)
0.25
A2
Ag
Figure The coarse effects of Å1, X2 and X3 on the CUB dataset.
Accuracy 196)
Accuracy
Accuracy
Figure 4: The fine effects of Å1, A2 and A3 on the CUB dataset.
Model Complexity Analysis. Our TopoZero has a clear intuition of leveraging parallel structure and distribution for advancing ZSL. Such design thus inevitably leads to the first 5 terms in Eq. 18 structure alignment and the last terms in Eq. för distribution alignment. Although TopoZero has 4 autoencoders in total, the entire training process is simultaneous and loss weights of all terms in Eq. 18 are the same for all datasets. The consistently significant results on all datasets show that our model robust and easy to train. Additionally, several losses are formulated with similar forms, which are cooperated for easy optimization, i.e. LA and La L and 4. P and p- Finally, TAM and DAM are parallel and such disentangle design can make the learning curve smooth and maximize the role of each branch, respectively. Benefiting from this disentangled design, our TopoZero is easy to train compared HSVA, where the latter adopts coupled framework.
for
Sayper-parameter Analysis. In this part, we further verify the sensitivity of hyyeer-paraeed in our TopoZero by conducting experiments on the CUB dataset, including Åς Ag. and Ag- As shown in Fig. 3 the performance of TopoZero is great robustness when varying hyper-parameter from {0.01, 0.05, 0.1, 0.25, 0.5, 1.03. Finally, Å1. X2 and λg are set 0.05, 0.05, and in this paper for the better result.
Although this hyper-panne paremeeareeerr configuration achieves great performance 3 ZSL benchmark datasets, it also raises an interesting question: given these hhper-parametee play distinct role in our TopoZero framework, why their effects most present a consistent trending. The reason for this question that the configuration parameter selection setting unreasonable, where the candidate range of X3 is small. This hides the role of each term in the objective function since the value of 4-th term (controlled by Aς) is far larger than that of 2-nd (controlled by A₂) and 3rd (controlled by Ag) terms, where the value of LV AE and CV in 4-th term is extraordinarily large. Thus, to conduct a detailed hyper-paramett analysis, we extend the range of λ into {0.0001, 0.0005, 0.001, 0. 0.0050.01, 0.05, 0.1, 0.25, 0.5, 1.0). Based on this revision, the individual effects of the three hyper-parameters are expanded remarkably, that illustrated Fig. Simultaneously, our TopoZero achieves higher CZSL accuracy of 64.9 64.9% on the cub dataset via this step. In our opinion, this improvement benefits from this more reason- able hyper-parameter selection procedure, which is conducive to getting rid of parammte overfitting" via mining the role of each item accurately. Considering this step involves some tricks of hyper-parameter tuning, we only discuss this situation rather than adopting this hyper-paramen parameter configuration for better results.
consistent? For instance, the green lines in Fig. 3 al-
the hyper-
Visualization Result. As shown in Fig. I (a) (c), we utilize persistent homology to visualize the multi-dimmensionll topological features of TopoZero and HSVA latent structure space. We can
16

---

Under review as a conference paper at ICLR 2023
that our TopoZero topologiaal latent space presents almost consistent trend in terms of in- put topological space while HSVA fails, indicating that our TopoZero can preserve more geometry information than HSVA when handling with 'uuree of dimensionality' [Wang& & Chen (2017).
17


Reviews:
[{"review_id": "iPKFVVYCI4", "reviewer": "Reviewer_ijnG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a topology-guided sampling strategy (TGSS) to mitigate the gap between sampled data within a mini-batch and global data. The proposed model, which is called TopoZero, consists of a topology alignment module (TAM) and a distribution alignment module. TAM is capable of preserving multidimensional geometric structures in each of the latent visual and semantic spaces. Experimental results show that TopoZero performs well on standard ZSL benchmark datasets.", "summary_of_the_review": "Although the proposed method, called TGSS, has a certain degree of novelty and shows improvement in performance from the baselines, it is not considered to have reached the stage of publishing at this time due to a lack of experiments and insufficient explanation due to the low quality of the writing.", "strength_and_weaknesses": "- Strength\n\n-- A topology guided sampling strategy (TGSS) is proposed to mitigate the gap between sampled data in a mini-batch and global data.\n\n-- A theoretical analysis is presented for TGSS.\n\nTopoZero shows good performance on standard ZSL benchmark datasets.\n\n- Weaknesses\n\n-- The paper is not self-contained, which hinders understanding of the paper.\n\n-- The paper is poorly written, making it difficult to read the paper smoothly.\n\n-- There are some typos.\n\n-- Evaluations of the proposed method are insufficient.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_nVcz8AYU0", "reviewer": "Reviewer_MFWd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a TopoZero framework to improve structure alignment for common space learning methods. Based on two weaknesses of HSVA (trains model with local structure and loses some high-dimensional structure information), the authors propose a Topology-guided sampling strategy and a Topology Alignment Module. Extensive experiments show the effectiveness of the proposed approach.", "summary_of_the_review": "The proposed approach aims to preserve the global topology of training samples to train the models, which seems novel. However,  the framework is too complicated, which ensembles CADA-VAE. It is difficult to judge which part improves the performance. Moreover, the compared methods are not recent state-of-the-arts and there is no qualitative analysis of the proposed approach.", "strength_and_weaknesses": "Strength:\n1. This paper is well-organized and well-written, which is easy to read.\n2. The idea of topology alignment is novel, but it is only reflected in sampling data, where the relations among data are not modeled, which is a bit trivial. \n\nWeakness:\n1. Figure 1 is difficult to understand. What are the lines in figure 1 (a-c) represent? What does ‘0-dim’ represent?\n2. The framework is too completed. It contains 4 autoencoders, which seems an ensemble of the proposed approach and CADA-VAE. There are too many losses in the framework. Is it efficient to learn?\n3. The representations in the formulation are inconsistent.\n(a)Is ‘x’ in Eq 2 ‘x_max’?\n(b)Are ‘Z_v’ in Eq 9 and ‘z_x’ in Eq 11 the same? ‘z_x’ is not mentioned in the paper.\n4. All the compared methods are before 2022 and the performance is not good compared with the recent 2022 papers. \n5. The proposed method is an incremental work based on HSVA, why not use the same baseline with HSVA. It is difficult to judge whether the improvement is made by the proposed TAD or an ensemble of CADA-VAE.\n6. No qualitative analysis is made to show why preserving the global structure is good. What does it benefit? Is it good to generate more effective features? However, no feature visualization is conducted.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Negg0e6N0Pw", "reviewer": "Reviewer_FV59", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a topology-guided sampling strategy as a parallel pipeline to CADA-VAE. Empirical and theoretical analysis of the topology property is provided. The method is evaluated on the three common benchmarks of CUB SUN and AWA and achieves promising performance.", "summary_of_the_review": "Overall, this work meets the standard expected by ICLR. However, the model design contains duplication and redundancy which normally indicates risks in \"hyper-parameter overfitting\". The evaluation of hyperparameters in the appendix is very contradictory and needs further elaboration.", "strength_and_weaknesses": "+ The motivation for topology structure alignment is clear and strong.\n+ The theoretical analysis of TGSS is solid and inspiring.\n+ The paper is well presented and easy to follow.\n- The overall Objective Function (Eq 18 and 19 should be labelled once only) contains duplicates. For example, the reconstruction is considered multiple times in AE, TP and CA.\n- The hyperparameter evaluation is odd. Given AE TP and CA are counting, visual-semantic reconstruction, topologic alignment, and distributional alignment, why the effects of the three hyperparameters are so consistent? It usually needs to find a balance between complementary objective functions. Otherwise, at least one of the terms must be useless.\n- Despite the intuitive rationale in Figure 1 and solid theoretical analysis in 3.1, the idea is lack quantitative analysis supported by empirical examples.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2KTV02l4u3V", "reviewer": "Reviewer_ugcD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposesa Topology-guided Sampling Strategy (TGSS) to mitigate the distribution gap between sampled and global data points for Zero-Shot Learning. In addition, a Topology Alignment Module (TAM) is proposed to perserve multi-dimensional geometry structure in latent visual and semantic space. The proposed method is evaluated on several benchmark datasets for ZSL and generalised ZSL and achieves superior performance.", "summary_of_the_review": "The authors aim to design an approach to facilitating topology preserving in ZSL from a novel perspective. The proposed method is proved to be effective when combined with a classic ZSL framework CADA-VAE which, however, is a relatively old technique for ZSL. The authors fail to compare with more advanced ZSL approaches.", "strength_and_weaknesses": "++ The proposed methods facilitate the idea of topology preserving during representation learning which has been proven important for zero-shot learning. The proposed TGSS has been theoretically and empirically validated in this work.\n\n++ The proposed method is justified theoretically and empirically. The experiments are thorough and convincing.\n\n-- The authors fail to compare with more recent/advanced ZSL approaches in the experiments.\n\n-- I wonder if the TGSS strategy also works for other ZSL frameworks than CADA-VAE.\n\n-- The authors should have discussed the impact of batch sizes on the performance of TGSS. Why is the batch size set as 50 when the number of classes could be as large as 150/645? Is there any simple baseline methods as alternatives to TGSS? e.g., class-balanced sampling for each batch?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject