Fine-grained Few-shot Recognition by Deep Object Parsing
Authors: Ruizhao Zhu
Pengkai Zhu
Samarth Mishra
Venkatesh Saligrama
Venue: Submitted to ICLR 2023 (2023)
Keywords: Few-shot learning
Representation learning

Abstract: We propose a new method for fine-grained few-shot recognition via deep object parsing. In our framework, an object is made up of $K$ distinct parts and for each part, we learn a dictionary of templates, which is shared across all instances and categories. An object is parsed by estimating the locations of these $K$ parts and a set of active templates that can reconstruct the part features.  We recognize test instances by comparing its active templates and the relative geometry of its part locations against those of the presented few-shot instances. Our method is end-to-end trainable to learn part templates on-top of a convolutional backbone. To combat visual distortions such as orientation, pose and size, we learn templates at multiple scales, and at test-time parse and match instances across these scales. We show that our method is competitive with the state-of-the-art, and by virtue of parsing enjoys interpretability as well.

Under review as a conference paper at ICLR 2023
FEW-SHOT RECOGNITION BY DEEP OBJECT PARSING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a new method for fine-grained few-shot recognition via deep object parsing. In our framework, part, we learn a dictionary of templates, which is shared across all instances and categories. An object is parsed by estimating the locations of these K parts and a set of active templates that can reconstruct the part features. We recognize test instances by comparing its active templates and the relative geometry of its part locations against those of the presented few-shot instances. Our method is end- to-end trainable to learn part templates on-top of a convolutional backbone. To combat visual distortions such as orientation, pose and size, we learn templates at multiple scales, and at test-time parse and match instances across these scales. We show that our method is competitive with the state-of-the-art, and by virtue of parsing enjoys interpretability as well.
object is made up of K distinct parts and for each
INTRODUCTION
Deep neural networks (DNN) can be trained to solve visual recognition tasks with large annotated datasets. In contrast, training DNNs for few-shot recognition (Snell et al. 2017: Vinyals et al. (016)), and its fine- mne-grained variant (Sun et al., 2020), where only few examples by way of supervision at test-time, is challenging. Fundamentally, the issue is that few-shots of data often inadequate to learn object model among all of its myriad of variations, which do not impact an object's category, For our solution, we propose to draw upon two key observations from the literature.
provided for each class
(A) There are specific locations bearing distinctive patterns/signattres in the feature space of a convolution neural network (CNN), which correspond salient visual characteristics of an image instance (Zhou et al. Bau et al. (017)).
(B) Attention on only a few specific locations in the feature space, leads to good recognition accuracy (Zhu et al. 2020: Lifchitz et al.) 2021 Tang et al. 2020).
How can we leverage these observations?
Duplication of Traits. In fine-grained classification tasks, we posit that the visual characteristics found in one instance of an object are widely duplicated among other instances, and even among those belonging to other classes. It follows from our proposition that it is the particular collection of visual characteristics arranged in a specific geometric pattern that uniquely determines an object belonging particular class.
Parsing. These assumptions, along with (A) and (B), imply that these shared visual traits can be found in the feature maps of CNNs and only few locations the feature map suffice for object recognition. We call these finitely many latent locations on the feature maps which correspond to salient traits, parts. These parts manifest patterns, where each pattern belongs potentially large) dictionary of templates. This dictionary embodies both the shared vocabulary and the diversity of patterns found across object instances. Our goal is learn the dictionary of templates for different parts using training data, and at test-time, we seek to pars new instances by identifying part locations and the sub-collection of templates that are expressed for the few-shot task. While CNN features distill essential information from images, parsing helps further suppress noisy
finite (but
ww view our dictionary as collection of words, parts phrases that are collection of words from the dictionary, and the geometric relationship between different parts as relationship between phrases.

---

Under review as a conference paper at ICLR 2023
Input Image
Corv Festures
Structure
PArsed Parts
Scale Matching
Small
Smell
Medi
Medium
Large
Large
Figure Motivation: a) In fine-grained few-shot learning, the most discriminating information is embedded in the salient parts (c.g. head and breast of a bird) and the geometry of the parts (relative part locations). Our method parses the object into a structured combination of finite set of dictionaries, such that both finer details and the shape of the object are captured and used in recognition. b) In few shot learning, the same part may be distorted or absent in the support samples due to the perspective and pose changes. We propose to extract features and compare across multiple scales for each part to overcome this.
information, in situations of high-intra class variance such as in few-shot learning. For classification, few-shot instances are parsed and then compared against the parsed query. The best matching class is then predicted as the output.
As an example see Fig (a), where the recognized part locations using the learned dictionary corre- spond to the head, breast and the knee of the birds in their images with corresponding locations in the convolutional feature maps. In matching the images, both the constituent templates and the geo- metric structure of the parts are utilized. Inferring part locations based on part-specific dictionaries a low complexity task, and is analogous to the problem detection of signals in noise radar applications (Van Trees 2004). problem solved by matching the received signal against a known dictionary of transmitted signals.
Challenges. Nevertheless, we do not a-priori have a dictionary, and to learn one, we are only provided class-level annotations by way of supervision. In addition, require that these learnt dictionaries compact (because we must be able to reliably parse any input), and yet sufficiently expressive to account for diversity of visual traits found in different objects and classes.
situation is somewhat more challenging. Unlike the radar situation,
Multi-Scale Dictionaries. Variations in position and orientation relative to the camera lead to dif- ferent appearances of the same object by perspective projections, which means there is variation in the sizes of visual characteristics of parts. To overcome this, we train dictionaries at multiple scales, which leads us a parsing scheme that parses input instances at multiple scales (see Fig. (b)).
Goodness of fit. Besides part sizes, few-shot instances even within the same class may exhibit significant variations in poses, which can in-turn induce variations in parsed outputs. To mitigate their effects we propose novel instance dependen re-eeighting method, for comparison, based on goodness-of to the dictionary.
Contributions. (i) We propose deep object parsing (DOP) method that parses objects into its constituent parts, and each part as collection of activated templates from dictionary, while using the representational power of deep CNNs. Via suitable objectives, we derive simple end-to-end trainable formulation for this method. (ii) We evaluate DOP on the challenging task of fine-grained few shot recognition, where DOP outperforms prior art on multiple benchmarks. Notably, it is better by about 2.5% on Stanford-CC and 10% on the Aircraft dataset. (iii) We provide an analysis of how different components of our method help final performance. We also visualize the part locations recognized by our method, lending interpretability to its decisions.
RELATED WORK
Few-Shot Classification (FSC). Modern FSC methods can be classified into three categories: metric-learning based, optimization-basee data-augmentation methods. Methods in the first cat-
2

---

Under review as a conference paper at ICLR 2023
egory focus on learning effective metrics to match query examples to support. Prototypical Network
(Snell et al. 2017) utilizes euclidean distance on feature space for this purpose. Subsequent ap- proaches built this by improving the image embedding space (Ye et al.] 2020: Das et al.] 2 Afrasiyabi et al. 2021| Zhou et al. 2021 Rizve et al.) (021) or focusing on the metric (Sung et al.
2018
Wang et al. 2019] [Bateni al. [0200] Simon et al. 2020 [i et al. 2020a Wertheimer al.
2021 Fei ct al. 2021 et al. | (021). Some recent methods have also found use of graph based methods, especially in transductive few shot classification (Chen et al.
2021 Yang et al. 2020). Optimization based methods train for fast adaptation using few parameter updates with the
support examples (Finn et al. 2017 Baik et al. |021 Li et al., [017] Lee et al.
2019 Rajeswaran
et al. a 2019 Liu et al. 20. Data aata-augmentttion methods learn a generative model to synthesize ad-
ditional training data for the novel classes to alleviate the issue of insufficient data (Li et al. 2020b
Schwartz et al. 2018 Wang et al. 2018 Xu et al. 2021).
Fine-grained FSC. In fine-grained few-shot classification, different classes differ only in finer vi- sual details. An example of this is to tease apart different species of birds in images. The approaches mentioned above have been applied in this context as well (Li et al. 2019: Sun et al.] 2020: Li et al. 2020ct Xu et al., 2021). (Li et al. 2019) proposes to learn local descriptor and image- to-clss measure to capture the similarity between objects. (Wang et al., (022)) uses a foreground object ex- tractor to exclude the noise from background and synthesize foreground features to remedy the data insufficiency. BSNet (Li et al. 2020c) leverages a bi-similarity module to learn feature maps of di- verse characteristics improve the model's generalization ability. Variational feature disentangling (VFD) (Xu et al. 2021). a tata-augmentation method, is complementary to ours, It disentangles the feature representation into intra-class variance and dlass-disrimiintting information, and gener- ates additional features for novel classes at test-time. TDM (Lee et al. 2022) applies channel-wiss attention to represent different classes with sparse vectors.
Recognition using Object Parts Our method is closely related to recognition based on identifying object components, an approach motivated by how humans learn to recognize object (Biederman) 1987]. It draws inspiration from (Ullman et al.) 2002). who showed that information maximization with respect to classes of images resulted in visual features eyes, mouth, etc. in facial images and tyres, bumper, windows, etc. in images of cars. Along these lines, Deformable Part Models (DPM) (Felzenszwalb et al. 2009: 2010) proposed to learn object models by composing part features and geometries, and utilize it for object detection. Neural Network models for DPMs were proposed in (Savalle et al., 2014; Girshick et al.] 2015). Multi- aallii- based models, which can be viewed as implicitly incorporating parts, have been proposed (Zheng al. (017) in the context of fine-grained recognition problems. Although related, a principle difference is our few-shot setting, where new classes emerge, and we need generate new object models on-the-fly.
Prior works on FSC have also focused on combining parts, albeit with different notions of the con- cept. As such, the term part is overloaded, and is unrelated to our notion. DeepEMD (Zhang al.) 2020) focuses in the amage-distanee metric based on an earth mover's distance between different parts. Here, parts are simply different physical locations in the image and not a compact collection of salient parts for recognition. (Tang et al.) 2020) uses salient object parts for recognition, while (Tokmakov et al. 2019) attempts to encode parts into image features. However, both these methods require additional attribute annotations for training, which may be expensive to gather and not al- ways available. (Hao et al., 2019) and (Wu et al.) 202I) discover salient object parts and use them for recognition via attention maps similar to method. They additionally re- re-weigh their similarity task adaptidapiivlly, which is also a feature of our method. We differ in our use of a finite dictionary of reconstruction
templates for learning a compact representation of parts. Also, sion for accurately localizing salient object parts, and impose a meaningful prior on the geometry of parts, which keeps from degenerate solutions for part locations. For more detailed comparison with prior related work in fine grained FSC, please refer to the supplementary (Appendix B).
supervi-
3 DEEP OBJECT PARSING
Parsing Instances. Each input instance to method is first parsed using learned templates into higher level syntax, in the form of parts. While this term, "parts", is overloaded in prior works, our notion of part is tuple, consisting of part-location and part-expression at that location.
3

---

Under review as a conference paper at ICLR 2023
Part
Part
Magh FeyAl
CNN
feature
Gommanc
Similarie axtric
Poar
oumpent different sciles
Seature extractor
Farures
Milutt Payalk
Mite Supi
Inage
Image
Port
Part K
PARSE
PARSE
Deep Object Parsing. An image x is parsed as collection of salient parts (K in number). Each part is represented by 2D location μ and part expression vector 2. We notate this operation PARSE and describe it in detail in Alg. In our method, we estimate locations and expressions at multiple scales for each part (hence there are more than one μ and & per part) and using these, determine image similarity for few-shot recognition. (Best viewed with zoom)
Figure 2:
Introducing notation, let E .CC be an input instance (in our case, an image), f X. - RGXGXC a convolutional neural network (CNN) backbone and Φ = f(x) features of x, with channels G x G grid. We parse into K distinct part-locations Mp E [G] x [G] and
supported on 2D
part expressions 2p E RC for p [K]. In our method, we also learn dictionary of feature- space templates Dp.e E TR*x* p E [K], E [C] that are used to represent part features of different instances across different categories.
Given an 8 X mask (Pp) centered at μp (with & < G),, the learned templates reconstruct part features o with the Ep acting as codes Ge,Mpp Zp..Dp.c. where the subscript Mμ,, denotes projection onto the support of M ((a) (or simply
window cut-out of &. centered at /4p). Note that instead of using multiple, we used one dictionary atom per part per channel. While more atoms can reconstruct features better, we found experimentally that they did not benefit few-shot learning performance.
Part Expression LASSO Regression. Given instance x, its feature output, o, and candidate part-location, μ4, we can estimate sparse part-expression coefficients (p(11) E RC by optimizing the & regularized reconstruction error, the location μ Up (A being the regularization constant).
Zp (1) = arg min
Σ [1), De. A111111
CEC
Non-negativity. Part expressions žp.e signify presence or absence of part templates in the observed feature vectors, and such can be expected to take on non-negative values. This fact turns out be useful later for DNN implementation.
Part Location Estimation. Note that part expression z, is function of location μ., while the part location Up can be estimated by plugging in the optimal eart-exppesssoos for each candidate location value, namely, [Ap arg min
Σ (1) (p)) All
HE[G] X CEC
This couples the two estimation problems, and difficult to implement with DNNs, motivating our approach below.
Feedforward DNNs for Parsing. To make the proposed approach amenable to DNN implemen- tation, approximate the solution to Eq. [ by optimizing the reconstruction error followed by thresholding, namely, we compute 2/) arg ming Eesc |1 - Dp.c.ell' 2|? and we thresh- old the resulting output by deleting entries smaller than Ç: (u) = ul 2<- This is closely related to thresholding methods employed in LASSO (Hastie al. 2001).
The quadratic component of the loss allows for an explicit solution, and the solution reduces to tem- plate matching per channel, which further be expressed convolution (Gonzalez Woods
4

---

Under review as a conference paper at ICLR 2023
2008 Using this insight, we derive estimate of M4p as
Pp = arg max μ |E(G)x((C x
Σ ((Op.e f (14 - Xa)²
(3)
GEC
where * is convolution, P.cc Dp.e///ppelll and d = punll becomes channel dependent
constant. With the above estimate of Up, we get the estimate of 2p (recall Sc(u) = ju|2c):
p.c @cpp))) zp.c(H) (z(p))
Zp.c
For full derivation of the above estimates, please refer to the supplementary materials (App. A).
Estimates differentiable in parameters. Since argmax is non-differentiable function, using Eq. ( for estimating part-locations does not allow us to use gradient based learning for the parameters of the DNN. We can cireumvent this by approximating the argmax as the expectation of softmax distribution Up over [G] x [G] with a low temperature T.
(10p.c (o)(m) - >o?); Mpp Buwey
(n) softmax
T
Σ CEC
(5)
2/,0 (Dpr )pmm) De
2p.c = Sc(Ep.c)
p.ppeel?
where 5y,, is differentiable approximation of dirac delta centered at μu using a narrow normal dis- tribution and !:' is the double-dot product the sum of all elements of product. Our derivation come from template matching (or convolving the templates) with the CNN features and pooling the product of location indices and Up. Part-expressions are then found via simple convolution and dot product (Eq. (6).
A) hence leads to very simple expressions, where part-locations Hp
Multi-Scale Extension. We extend our approach to incorporate parsing parts at multiple scales. This often required because of significant difference in orientation and pose between query and support examples. To do so we simply consider masks M (2) and templates D at varying mask sizes E S, each leading to independent part location and expression estimates (Jus,p- 2s.p) for part p. Alg. specifies the parse of input instance and Fig. shows overview of object parsing.
Algorithm 1: PARSE (Object Parsing using DNNs)
Given: Convolutional backbone f, templates (Dappcl threshold Ç, l regularization constant X, temperature T
Input: Image x
Compute convolutional features Φ =f(x)
for p E [K],
do
Estimate Mx (through Vx.p)) using Eq. 5
Estimate zs.p [zs.p. using Eq. (6)
end for Output: Part locations and expressions
(Zephee(K),es))
3.1 FEW-SHOT RECOGNITION
At test-time we are given query instance, q, and by way of supervision, M support examples each for N classes, and the goal is predict the query class label y(ι) [N]. We first run PARSE (Alg. I on each of these. PARSE(g) = (Lu1)), {2(g)} and for the ith support example of class y, ((.g) (f.w) (A = ((( ). {=x.p }). For comparing query and support examples need notion of distance/similarriy. which we define next.
Goodness-offH reweighting. The entropy of the distribution Ps, is an important indicator of goodness-of-fii of the dictionary templates (lower entropy meaning a more precise and confident (g)
part-location prediction result of better fit). Let hagp and h (t.g) ha,p be the entropies of v and
(g)
5

---

Under review as a conference paper at ICLR 2023
(6v) da,p respectively. To use these as weights for computing distance (as below), we learn a simple parametric function a IRM++ - R.
(i.2) 24. 8 S.p p E [K]
Additionally, with 2z, over all support examples in class y. With these, define the following distance measure between the query example and the support examples of class
EiEM
represent the mean part expression
d(g.v)
Σ ΣV KE(K]
E
a(h (1 kE(M))
2(1) -011> =2iP
(v)
[expression distance]
Ed #268
ΣEΣ
| (i (ll (ppps)))ppp)))
Igeometric distance] (7)
iE[M] sn E& $2E8
where ([Ms. (((ssp)le)))) is vector of pairwise distances between all part locations scale 8, malized to unit sum. The distance function consists of an expression term, and a geometric term with y acting as a tunable weight to control the proportion of the two. Each term is sum all combinations of part scales over query and support. Note that the geometric term simply attempts find if two polygons with vertices at part locations similar (i.e. have proportional sides), with the distance being 0 if they are. Finally, the class prediction is made 7( arg min d(g, v)-
Training. We train in episodes following convention. For each episode, we sample N classes at random, and additionally sample support and query examples belonging to these classes from training data (details in Sec. 4. Using softmax over the negative distance function above as the class distribution query q, we define the cross-cntropy loss as
(CE(g) = - log (xp(-d(g,09))
(8)
EyE[N] exp( exp(-d(g,1)) v))
Additionally, while training, we impose geometric prior to get diverse instance parts in PARSE by maximizing the Hellinger distance (Everitt 1998) H....) between part distributions. The corre- sponding criterion for minimization is
Edit (I) = Σ
Σ SE p-P'E(K]
H(Vs.:
pfp"
Alg. _ outtines the loss computation a single query q. Line 6 trolling the weight of the prior. In each episode, we use average of the loss output over multiple query examples, which results in an end-to-end differentiable criterion in all trainable parameters, allowing us optimize using gradient descent.
1) is a tunable parameter con-
Algorithm 2: Training loss for Few-shot recognition with DOP (single episode, single query)
Given: requirements for Alg. PARSE, weighting function tunable parameter
Input: Query example with ground truth label 9, ( X [N]. Support examples I UyU [4,
Trainable parameters: Convolutional backbone f, part templates (esme)essppellleeee weighting function
Compute parses PARSE(q) = (Hsp), (2(1))
(AR))( (qu2,)
(1 }. (2a.p }); E 8, E [K]
(1.))
Compute distances d(g. v) for [N] using Eq.
Output: loss E(g) (g)(1) + " LHT E ErelUia (x) using Eqs. 8) and 9
3.2 IMPLEMENTATION DETAILS
We use two Resnet (He et al. 2016) backbones (Resnet- 12 and Resnet-18) as feature extractors. We
use the 12 model used in prior works (Lee et al. 2019). which has more output channels and more parameters (12.4M) compared to Resnet-13 18 (11.2M (1.2M). The input image is resized to 84 x 84 for Resnet- 12 and 224 224 for Resnet-18. In the output features, for Resnet-12, C 640 and G = 5, and for Resnet-18 18, C = 512 and G = 7. The number of parts K is set to for most experiments (see Sec. 4.2 for experiments with different number of parts). There three scales

---

Under review as a conference paper at ICLR 2023
8 = (1.3.5) considered for each part. The temperature T in Eq. (5) is set to 0.01 and the threshold Ç in Eq. (6) is set to 0.05. y is set to 0.01. For a we used a linear layer, and subsequently normalized [a<hhaspp ( (ieee) ppejk] (a K-dimensional vector; see Eq. 7 with softmax.
(4) (6.3)
4 EXPERIMENTS
4.1 FINE-GRAINED FEW-SHOT CLASSIFICATION
We compare DOP on four fine- -en-grained datasets: Caltech- Brrech-UCSD--irdee (CUB) (Wah et al., 201
Stanfor-DDog (Dog) (Khosla et al. (011)) Stanford-Cat (Car) (Krause et al. 2013) and Aireraft MMaji et al. 2013) against state-of-the-an methods.
Caltech-UCSD-BiR Birds (CUB) (Wah et al. 22111) is fine- fine-grained classification dataset with 11,788 images of 200 bird species. Following convention split into 100 base, 50 validation and 50 test classes.
et al. 2018), the 200 classes are randomly
Aircraft contains 100 classes of aircrafts and 10,000 images in total. Following recent benchmark (Lee et al. 2022, Wertheimer et al. (222), we processes all images based on bounding box. And the 100 classes are split into 50, 25 and 25 classes for training, validation and test.
Stanford-Dog/Car (Khosla al. 2011 Krause al. 2013) are two datasets for fineggrained clas- sification. Dog contains 120 dog breeds with a total number of 20,580 images, while Car consists of 16,185 images from 196 different car models. For few-shot learning evaluation, we follow the benchmark protocol proposed in eLi al.) 2019). Specifically, 120 classes of Dog are split into 70, 20, and 30 classes, for training, validation, and test, respectively. Similarly, Car is split into 130 train, 17 validation and 49 test classes.
Experiments Setup. We conducted 5-way (5 classes episode) 1-shot and 5-way 5-shot classifica- tion tasks on all datasets. Following the episodic evaluation protocol in (Vinyals et al., al.] 2016)), at test time, we sample 600 episodes and report the averaged Top-1 accuracy. In each episode, 5 classes from the test set are randomly selected. 1 or 5 samples for each class are sampled as support data, and another 15 examples are sampled for each class as the query data. The model is trained on train split and the validation split is used to select the prpper-prrmmters.. We compare our DOP to state-of-the-as art FSC and fine-graine mee-grinned FSC methods. More details in the supplementary (App. B).
Training Details. Our model is trained with 10,000 episodes on CUB and 30,000 episodes on Stanford-Dog/Car for experiments with both ResNet12 and ResNet18. In each episode, we randomly select 10 classes and sample 5 and 10 samples as support and query data. The weight on the geometric prior 7 is set to 1.0 respec-
Table 1: Few-shot classification accuracy in % on Aircraft dataset. Unless specified, methods use ResNet-12 backhone.
Methods
I-shot
5-shot
ProtoNet et al. 2017
67.28
83.21
et al. 2020
70.23
83.03
on CUB and 0.1 on Stanford-Dog/CAe tively. We train from scratch with Adam op- timizer (Kingma & Ba 2014). The learning rate starts from Se-4 on CUB and 1e-3 on Stanford-Car/Do and decays to 0.1x every 3,000 episodes on CUB and 9,000 episodes on Dog/Car. On CUB, objects are cropped using the annotated bounding box before resizing to the input size. On Stanford- Stanford-Car/Dog. we use the resized raw image as the input. We employed standard data augmentations, including horizontal flip and perspective distortion, to the input images.
CTX ct al. 2020
71.57
79.31
FRN Wertheimer al.
2021
69.58
83,98
Helix
2022
74.01
83.11
TDM et al. 2022
71.57
84.77
DOP (ResNet18) DOP (ResNet12)
84.26
93.41
85.50
94.35
Results. Our results along with comparisons against
on CUB, Aireraft and Stanford-
Dog/Car are reported in Tabs. 2 and respectively.
DOP is competitive with or outperforms recent works on fine-grained FSC. On CUB (Tab. 2 DOP outperforms all compared approaches with 83. 33.39% 1-shot accuracy and 93.01 % for 5-shot with Resnet- 12 backbone. Same is the case for Aircraft (Tab. where DOP outperforms a recent method in TDM (Lee et al. 2022) by large margin using ResNet- 12 backbone, performing at 85.50% 1-shot and 94.35% % 5-hhot accuracy. While on Stanford-Car (Tab. B) we outperform compared approaches by 3.06% and 1.85% on 1-shot and 5-shot, respectively, on Stanford- outperform
7

---

Under review as a conference paper at ICLR 2023
Table 2: Few-shot accuracy in % on CUB compared to most recent state-of-the-arr methods (along with 95% confidence intervals). If not specified, the results are those reported in the original paper. t: results are obtained by running the public implementation released by authors using ResNet18 backbone. A more comprehensive comparison to all related methods can be found in App. C.
Methods
Backhone
I-shot
5-shol
PnotoNet(Snell et al.
2017
ResNet18
71.88+0.91
87.42++0.48
DeepEMD Zhang al.
2020
ResNet12
75.65++0.83
88.6941.50
FOT Wang et al.
2021
ResNet18
72.56++0.77
87.22++0.46
VFD
Xu et al. 2021)
ResNet12
79.12+t0.83
9..48
Wertheerer et al.
2021
ResNet12
83.16
92.59
RENet Kang et al.) 202
ResNet12
79.49=t0.4
91.11#:0.24
ODAN(uuan ct al. [2021
ResNet12
67.17 0.81
82.09:00.56
RAP(Hong et al. 2021)
ResNet18
83,59+0.18
90.77=0.10
Lee et al. 2022)
ResNet12
83.36
92.80
HelixFormer al. 2022)
ResNet12
81.66+0.30
91.83++.117
DOP
ResNet18
82.62:0..55
92.61=10.38
DOP
ResNet12
83.39
93.01=10443
all methods but VFD. We note here that VFD generates additional features at test-time for novel classes, and as such, complementary DOP.
Table 3: Few-shot classification accuracy in % on Stanford-Car/Dog benchmarks (along with 95% confidence intervals). f: results are obtained by running the codes released by authors using ResNet18 backbone.
Car
Dog
Methods
Backbones
I-shot
S-shot
1-shot
S-shot
ProtoNeti Snell al.
2017
ResNet18
60.67:0.87
75.56=10:45
6..06=0.67
74.31=0.51
MetaOpiNeti LLe al.
2019
ResNet18
60.56+0.78
76.35-1.52
65.48=0.56
79.39++0.43
BSNet et al 2020c
ResNet12
60.36:10.98
85.28
69.09 0.90
82.45:10.58
VFD et al. 2021
ResNet12
0.824:1087
88.00:0.47
Honne al. 2021
ResNet12
76.62:10.70
89.57:1.40
0.883300.80
89.6600666
Folix et al.
2022
ResNet12
79.40++0.43
92.26-
65.92-+0.49
80.65+00.36
DOP
ResNet18
81.41:10.71
93.48-10.88
84.75+0.411 7..5
DOP
ResNet12
81,83+0078
93,84+0045
70.10+00.99
15.
4.2 ANALYSES
From Prototypical Networks to Deep Object Parsing. As an overview, DOP combines object parsing, dictionaries at multiple template sizes, use of part geometry for distance computation and instance-dependent distance reweighting based on goodness-of-fit. These can be seen as method- ological developments over ProtoNet (Snell et al. 2017), a simple CNN feature-space distance based few shot classification approach.
ProtoNet to single part DOP. Let us consider simplified DOP method with a single part (K 1) and parsing done at a single scale (template size 5) 8 = {5}. There is thus one template D 5 consisting of learned parameters. In estimating the part location (Eq. (5). we con- volve the template over the CNN features, and perform some additional operations (Eq. with no additional learnable parameters to find part expressions. A ProtoNet with the same number of learnalle parameters can use D as final convolutional layer and perform global pooling over its outputs. On 5-way 5-shot classification on CUB this ProtoNet has accuracy 88.38%, while the simplified DOP has accuracy 90.56%. Thus, this improvement in performance be attributed to learning template shareable across parts that can be used for reconstructing features (using the part expressions 2). This reconstruction objective allows the part expressions to have less noise and
8

---

Under review as a conference paper at ICLR 2023
thus lower intra-class variance. Next, analyze what happens when we parse objects into multiple parts and at multiple scales.
Using multiple parts. Tab. 4 shows the effect of adding CUB. We see more parts up to a certain point (K = 4) allows DOP to learn better representations consequently improving performance, but with even more parts performance drops the model can start learning irrelevant or background signatures.
parts 5-way 5-shot accuracy
Using templates multiple scales. In Tab. [ussing the Stanford-Dog dataset, studied the effect of parsing parts at multiple scales. Learning dictionaries at multiple scales improves performance, since this allows DOP to parse the object parts even when their scale may vary (due to different positions and orientations relative to the camera).
Table 4: Effect of using different number of parts on 5-way 5-shot accuracy on CUB.
Table 5: Effect of using templates at different scales on 5-way 5-shot accuracy on Dog.
Num parts
Scales
[3]
[5]
[3,5]
[1,3,5]
Accuracy 90.56
92.10 92.61 92.21
92.06
Accuracy 81.56
81.38
83.04
84.75
In the supplementary, we perform additional analysis ablating the use of part geometress for distance computation and the use instance dependent reweighting
What parts does DOP detect? We visualize the locations H4p learned by DOP Fig. DOP able to detect consistent parts for the same task and often finds semantically meaningful parts like head and torso/breast in birds and dogs and wheels and doors/windows on cars. Fig. 3 also shows some failure cases of DOP, where it might fail to locate parts on the object if similar visual signatures appear in the background. A visualization of part templates learned and part expressions can be found in the supplementary (App. D).
Aireraft
Dog
CUB
Car
Failure
We WiBl Miokk ANV savortion Prinet
Figure 3: Exemplar part locations learned by DOP when K 3. From left to right: Aircraft, CUB, Dog, Car, and failure cases. DOP can fail and locate parts the background if it has visual signatures similar to an object.
5 CONCLUSIONS
We presented DOP, a deep object-parsing method for fine-grained few-shot recognition. Our fun- damental concept that, while different object classes exhibit novel visual appearance, ficiently small scale, visual patterns are duplicated. Hence, by leveraging training data to learn dictionary of templates distributed across different relative locations, an object can be recognized simply by identifying which of the templates in the dictionary are expressed, and how these patterns are geometrically distributed. We build a statistical model for parsing that takes the output convolutional backbone input to produce parsed output. We then post-hoc learn to re- re-weight query and support instances to identify the best matching class, and as such this procedure allows for mitigating visual distortions. Our proposed method is end-to-end deep neural network training method, and we show that our performance is not only competitive but also the outputs generated interpretable.
suf-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arman Afrasiyabi, ing for few-shot image classification, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9041-9051, 2021.
Lalonde, and Christian Gagne. Mixture-basct feature space learn-
Sungyong Baik, Janghoon Choi, Heewon Kim, Dohee Cho, Jaesik Min, and Kyoung Mu Lee. Meta- learning with task-adaptive loss function for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9465-9474, 2021.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few- shot visual classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14493-14502, 2020.
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings of IEEE conference computer vision and pattern recognition, pp. 6541-6549 6549, 2017.
Irving Biederman.
a theory of human image understanding. Psycho-
logical review, 94(2):115, 1987.
Chaofan Chen, Xiaoshan Yang. Changsheng Xu, Xuhui Huang, and Zhe Ma. Eckpn: Explicit class knowledge propagation network for transductive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5605-6605, 2021.
Rajshekhar Das, Yu-Xiong Wang, and Jose MF Moura. On the importance of distractors for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9030-9040, 2021.
Carl Doersch, Ankush Gupta, and Andrew Zisserman. Crosstransformers: spatially-aware few-shot transfer. Advances in Neural Information Processing Systems, 33:21981-21993, 2020.
Brian Everitt. Cambridge dictionary of statistics. 1998.
Nanyi Fei, Yizhao Gao, Zhiwu Lu, and Tao Xiang. Z-score normalization, hubness, and few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 142-151, 2021.
Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained bast- models. IEEE transactions on pattern analysis and ma- chine intelligence, 32(9): 2009.
Pedro FF Felzenszwalb, Ross Girshick, and David McAllester. Cascade object detection with deformable part models. 2010 IEEE Computer society conference on computer vision and pattern recognition, pp. 2241-2245 leee, 2010.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135. PMLR, 2017.
Ross Girshick, Forrest landola, Trevor Darrell, and Jitendra Malik. Deformable part models are convolutional neural networks. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 437-446, 2015.
Rafael C. Gonzalez and Richard Woods. Digital image processing. Prentice Hall, Upper Saddle River, N.J. 2008
Fusheng Hao, Fengxiang He, Jun Cheng, Lei Wang, Jianzhong Cao, and Dacheng Tao. Collect and select: Semantic alignment metric learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 8460-8469, 2019.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artêm Yankov, Courtney D Corley, and Nathan O Hodas. Few-shot learning with metric-agnostic conditional embeddings. arXiv preprint arXiv: /8f2..8800.4444 04376, 2018.
Jie Hong, Pengfei Fang, Weihao Li, Tong Zhang, Christian Simon, Mehrtash Harandi, and Lars Pe- tersson. Reinforced attention for few-shot learning and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 913 923, 2021.
Huaxi Huang, Junjie Zhang, Litao Yu, Jian Zhang, Qiang Wu, and Chang Xu. Toan: Target-oriented alignment network for fine-graines image categorization with few labeled samples. IEEE Trans- actions on Circuits and Systems for Video Technology, 32(2):853-88) 866, 2021.
Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho. Relational embedding for few-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8822,-833, 2021.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141. 2014.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR- 13), Sydney, Australia, 2013.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
SuBeen Lee, WonJun Moon, and Jae-Pi Heo. Task discrepancy maximization for fine-grained few- shot classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5331-5340, 2022.
Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. Boosting few-shot learning with adaptive margin loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12584,, 2020a.
Kai Li, Yulun Zhang, Kunpeng Li, and Yun Fu. Adversarial feature hallucination networks for few-shot learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 13470-13479, 2020b.
Wenbin Li, Lei Wang, Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-clan measure for few-shot learning. Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7260-7268, 2019.
Xiaoxu Li, Jijie Wu, Zhuo Sun, Zhanyu Ma, Jie Cao, and Jing- Hao Xue. Bsnet: Bi-similarity network for fho-shot fine-grained image classification. IEEE Transactions on Image Processing, 30:1318-1331, 2020c.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv:1707.09833 2017.
Yann Lifchitz, Yannis Avrithis, and Sylvaine Picard. Few-shot few-shot learning and the role of spatial attention. In 2020 25th International Conference Pattern Recognition (ICPR), pp- 2693-2700. IEEE, 2021.
Q Sun Y Liu, TS Chua, and B Schiele. Meta-transfed learning for few-shot learning. In 2018 JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.

---

Under review as a conference paper at ICLR 2023
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- tine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 2013.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. 2019.
Mamshad Nayeem Rizve, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Exploring com- plementary strengths of invariant and equivariant representations for few-shot learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10836- 10846, 2021.
Pierre-André Savalle, Stavros Tsogkas, George Papandreou, and lasonas Kokkinos. Deformable part models with cnn features. In European Conference on Computer Vision, Parts and Attributes Workshop, 2014.
Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Rogerio Feris, Ab- hishek Kumar, Raja Giryes, and Alex M Bronstein. Delta-encoder: an effective sample synthesis method for few-shot object recognition. arXiv preprint 1rXiv:1806.04770 2018.
Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi. Adaptive subspaces for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4136-4144 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. arXiv preprint ar.iv::103.00//111 2017.
Xin Sun, Hongwei Xv, Junyu Dong, Huiyu Zhou, Changrui Chen, and Qiong Li. Few-shot learning for domain-specific fine-grained image classification. IEEE Transactions on Industrial Electron- ics, 68(4):3588-3598 2020.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018.
Luming Tang, Davis Wertheimer, and Bharath Hariharan. Revisiting pose-normalizzaion for fine- grained few-shot recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14352-14361, 2020.
Pavel Tokmakov, Yu-Xiong Wang, and Martial Hebert. Learning compositional representations for few-shot recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6372-6381, 2019.
Shimon Ullman, Michel Vidal-Naquet, and Erez Sali. Visual features of intermediate complexity and their use in classification. Nature neuroscience. 5(7):682-687, 2002.
Harry L Van Trees. Detection, estimation, and modulation theory, part I: detection, estimation, and linear modulation theory. John Wiley & Sons, 2004.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 19:3630-3635 2016.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds 200-2011 Dataset. Technical Report CNS-TR-201 20-220100000 1-001, California Institute of Technology, 2011.
Chaofei Wang, Shiji Song. Qisen Yang. Xiang Li, and Gao Huang. Fine-grained few shot learning with foreground object transformation. Neurocomputing, 466:16-26 2021.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Re- visiting nearest-neighhor classification for few-shot learning. arXiv preprint arXiv:1911.044 2019.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE conference on computer vision and pattern recogni- fion, pp. 7278-7286, 2018.
12

---

Under review as a conference paper at ICLR 2023
Davis Wertheimer, Luming Tang, and Bharath Hariharan. Few-shot classification with feature map reconstruction networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8012-8021, 2021.
Jiamin Wu, Tianzhu Zhang, Yongdong Zhang, and Feng Wu. Task-aware part mining network for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8433- 8442, 2021.
Jingyi Xu, Hieu Le, Mingzhen Huang, ShahRukh Athar, and Dimitris Samaras. Variational feature disentangling for fine-grained few-shot classification. In Proceedings of the IEEE/CVE Interna- tional Conference on Computer Vision, pp. 8812-8821, 2021.
Ling Yang, Liangliang Li, Zilun Zhang, Xinyu Zhou, Erjin Zhou, and Yu Liu. Dpgn: Distribution propagation graph network for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13390-13399, 2020.
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8808 8817 2020.
Baoquan Zhang, Xutao Li, Yunming Ye, Zhichao Huang, and Lisai Zhang. Prototype completion with primitive knowledge for few-shot learning. In Proceedings of the IEEE/CVE Conference on 2021.
Computer Vision and Pattern Recognition, pp-
Bo Zhang, Jiakang Yuan, Baopu Li, Tao Chen, Jiayuan Fan, and Botian Shi. Learning cross-image object semantic relation in transformer for few-shot fine- aeee-graied image classification. In Proceed- ings of the 30th ACM International Conference on Multimedia, pp. 2135-2144, 2022.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's is distance and structured classifiers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12203-12213, 2020.
Heliang Zheng, Jianlong Fu, Tao Mei, and Jiebo Luo. Learning multi-attention convolutional neural network for fine-grained image recognition. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 5209-5217, 2017. doi: 10.1 109/CCCV.001..557.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene enns. arXiv preprint 14126664....866. 2014.
Ziqi Zhou, Xi Qiu, Jiangtao Xie, Jianan Wu, and Chi Zhang. Binocular mutual learning for im- proving few-shot classification. In Proceedings of the IEED/CV International Conference Computer Vision, pp. 8402-8411 2021.
Yaohui Zhu, Chenlong Liu, and Shuqiang Jiang. Multi-attention meta learning for few-shot fine- grained image recognition. In IJCAI, pp. 1090-1096, 2020.
13


Reviews:
[{"review_id": "zjiDebe_l4", "reviewer": "Reviewer_SUHw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for fine-grained few-shot recognition by finding a dictionary of K parts to represent objects. The dictionary is shared across all instances and categories. An object is represented by where are these K parts and a set of templates that reconstruct the part features. The method achieves competitive performance compared to SoTA methods.", "summary_of_the_review": "My main concern is the novelty and unclear significance of the overall contribution. ", "strength_and_weaknesses": "Strength:\n- The method is end-to-end trainable.\n- An object instance is represented as a collection of parts with locations, which is somewhat interpretable.\n- The method achieves SoTA performance on some datasets.\n\nWeaknesses:\n- The novelty is on the weak side since there are previous methods that also parse objects into different salient parts for object recognition [2] [3]. The introduction doesn't discuss these methods and it's hard to clearly see the main contribution of the work. The authors write the following at the end of the related work: \n\"We differ in our use of a finite dictionary of templates used to learn a compact representation of parts. Also, we use reconstruction as supervision for accurately localizing salient object parts, and impose a meaningful prior on the geometry of parts, which keeps us from degenerate solutions for part locations.\" From there, I see there are two main points: - a finite dictionary of templates and - using feature reconstruction. The justification here is weak and there is no analysis of these two points.\n[2] Task-aware part mining network for few-shot learning\n[3]  Collect and select: Semantic alignment metric learning for few-shot learning\n\n- The motivation is not very convincing to me. I think a CNN automatically parses object instances into different parts [1] where each feature might respond to a different discriminative object part. In fact, here the authors also use a pre-trained network. Thus the main contribution here is to explicitly represent them as a fixed collection of K parts rather than directly using the implicit collection of parts learned from the deep network. I am not entirely sure why this representation is more robust than just using the CNN features. Does it come from the explicit locations of parts or is it because the method can extract the top K most informative and discriminative parts? In general, I think I didn't get the answer to \"why does this method work?\" from a high-level perspective. The connection between motivation and fine-grained few-shot learning is also not very obvious. Should this method work also on the standard setting of fine-grained classification? \n\n[1] Visualizing and Understanding Convolutional Networks\n\n- Parsing objects into a fixed set of parts seems very challenging and requires several technical components involved, e.g., setting the right value of K, using multiple scales, or instance-dependent re-weighting. These components only address the challenges specifically posed in this particular modeling. The solution is quite heavily engineered.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lWljQfw3c5", "reviewer": "Reviewer_qRQa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an end-to-end part-based deep object parsing model for fine-grained few-shot classification. It learns a compact dictionary of salient part templates and performs multi-scale template matching during testing. Different from traditional part-based works, this paper uses feature map reconstruction as supervision to localize object parts and imposes a geometric prior on part locations during matching. This work provides good model interpretability and is competitive with very recent state-of-the-arts on several popular few-shot classification datasets. \n", "summary_of_the_review": "I prefer to give a borderline accept score to the paper given that it clearly presents and provides a state-of-the-art part-based few-shot object classification method with good interpretability though some of its modules exist in previous works and some typos exist in the paper. It would be nice for the authors to make their code public to increase the potential impact of the work.  ", "strength_and_weaknesses": "Strength: \n* The proposed model is technically sound and effective, and it's end-to-end trainable. It successfully combines the advantages of good interpretability in traditional part-based models and nice performance improvement brought by deep networks and modules.\n* Generally speaking, the presentation is clear and the experiments are sound.\n\nWeakness:\n* The math formulas in Sec 3 are a bit difficult to understand. It's better to explain the variables more clearly immediately before each formula.\n* Possible typos in math formulas:\n  * In the 3rd paragraph of sec 3.1, z_{s,p}^y should be divided by M because it represents the mean part expression?\n  * In the alpha(...) function in Formula (7), s1 and s2 should be swapped.  \n* It would be nice to add more detailed comparison with most recent deep part-based models (e.g. [1], [2]) and show this work's novelty and strength.\n* It's better to explain why the proposed method surpasses other methods considerably in some datasets (e.g. the Aircraft dataset) while only on part with other methods on some other datasets (e.g. the CUB dataset).\n\n[1]: Few-Shot Classification with Feature Map Reconstruction Networks. CVPR 2021\n\n[2]: Task Discrepancy Maximization for Fine-grained Few-Shot Classification. CVPR 2022 \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "eORAfrdYmt", "reviewer": "Reviewer_cctR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied fine-grained image recognition under few-shot settings. The authors proposed to learn a dictionary of parts using object parsing first and then train a few-shot learner based on the parsed features. Learning a structured aligned representation is the right way for fine-grained few-shot learning. The high intra-class variations enlarge with the limited labeled samples in each category, which is more challenging than traditional fine-grained image recognition. It is interesting that the paper proposes to learn parsed object features from the perspective of dictionary learning.", "summary_of_the_review": "This paper proposed to solve the fine-grained few-shot image classification using deep object parsing. The idea is reasonable and interesting. However, the motivation of the paper is not well illustrated, and many other concerns exist, as discussed in the Strength And Weaknesses section. Therefore, the reviewer leans to reject the submission and encourages the authors to improve the paper and resubmit to other venues.", "strength_and_weaknesses": "Strength:\n1. The idea of learning a dictionary of parts for fine-grained few-shot image classification is correct and reasonable. \n2. The experimental results reported validate the effectiveness of the proposed DOP, and DOP achieved state-of-the-art performance on CUB, Car, and Aircraft datasets.\n\nWeaknesses:\n1. The motivation of the paper is somewhat weak. The connections between fine-grained few-shot image classification and deep object parsing are not well illustrated, especially for the intra-class variations that will enlarge under the few-shot settings.\n2. Some formulations in the paper are hard to follow. For example, on page 4, \"The location corresponds to an $s \\times s$ attention mask'', \"$D_{p,c} \\in R^{s \\times s}$ '', what's the attention mask used for? what's the connection between the $G \\times G$ 2D feature map slice and the $s \\times s$ attention mask? \"$ M(\\mu) $ centered round the location $\\mu$\", what's the function of $M(\\mu)$? \n3. As presented in the paper, the DOP needs to use DNNs to parse the input objects and generate the aligned-structured features. Is the DNNs in Algorithm 1 a pre-trained model or not? How to obtain these DNNs? If it is a pre-trained one, does the pre-trained dataset contains testing images or directly grab from pre-trained models on ImageNet?\n4. As the DOP consists of two steps, it imports additional operations of object parsing using DNNs. The model complexity, total parameters and running time comparisons with other single models should be investigated.\n5. In the experimental comparisons, the compared methods seem to vary a lot. On CUB, there are 15 baseline models. On Aircraft, Car, and Dog, there are only 5 to 7 baseline models. Moreover, equipped with complex deep object parsing, the DOP model achieves slight improvement on CUB, with only 0.03 and 0.21 boosts on 1-shot and 5-shot image classification over TDM. In Table 3, the accuracy of DOP lags behind the VFD by a wide margin on Dog, and more explanations or clarifications are encouraged.\n6. There exist previous works [C1,C2,C3] that proposed aligning and learning the fine-grained features in an end-to-end way, which are also inspired by the template learning in fine-grained image classification. Comprehensive comparisons are suggested.\n7. The visualization of the learned templated parts seems inaccurate in some correct cases in Figure 3. For example, the learned parts of Dog are not matched. In the top image, the locations are head, ear and body, while in the bottom image, the locations are nose, body and body.\n\nC1. Local spatial alignment network for few-shot learning\nC2. TOAN: Target-Oriented Alignment Network for Fine-Grained Image Categorization with Few Labeled Samples\nC3. Learning Cross-Image Object Semantic Relation in Transformer for Few-Shot Fine-Grained Image Classification", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6DO4K6CEzR", "reviewer": "Reviewer_47uq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of few-shot fine-grained recognition. The work is based on the generic assumption that while different object classes exhibit novel appearance, visual patterns are duplicated. Thus, the proposed method is forced to learn a dictionary of templates distributed across different relative locations. As a result an object can be recognised by identifying its corresponding parts in term of visual templates and their relation.  The whole process is expressed as a statistical model for parsing. Given a query image, the model re-weights the query and supports instances to find out the best matching class. The method is validated on 4 public datasets giving the results in terms of 5-way top-1 and 5-way top-5 accuracy.", "summary_of_the_review": "The assumption is intuitive, the statistical model is technically sound but it is not clear that the method outperforms the state of the art and in which of the datasets (as stated the authors mentioned that only the Stanford-Car dataset is improved).", "strength_and_weaknesses": "The strengths of the paper are:\n1. The main assumption is logic and intuitive.\n2. The statistical method is straightforward and technically sound.\n3. The method also can facilitate its interpretability and explainability since express explicitly the visual dictionary and the relevant parts of the object.\n4. The paper is well organised.\n5. Validation is done on 4 public datasets.\n6. Results are promising. \n7. Ablation study is presented.\n\nWeaknesses:\n1. State of the art should be completed and the method should be compared to the following works:\n- Huang, S., Wang, X., & Tao, D. (2021). Stochastic partial swap: Enhanced model generalization and interpretability for fine-grained recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 620-629).\n- Baffour, A. A., Qin, Z., Wang, Y., Qin, Z., & Choo, K. K. R. (2021). Spatial self-attention network with self-attention distillation for fine-grained image recognition. Journal of Visual Communication and Image Representation, 81, 103368.\n- Zhu, H., & Koniusz, P. (2022). EASE: Unsupervised Discriminant Subspace Learning for Transductive Few-Shot Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9078-9088).\n- Lazarou, M., Stathaki, T., & Avrithis, Y. (2021). Iterative label cleaning for transductive and semi-supervised few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 8751-8760).\n- Hong, J., Fang, P., Li, W., Zhang, T., Simon, C., Harandi, M., & Petersson, L. (2021). Reinforced attention for few-shot learning and beyond. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 913-923).\n- Fei, N., Gao, Y., Lu, Z., & Xiang, T. (2021). Z-score normalization, hubness, and few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 142-151), etc.\n2. Looking at the papers mentioned above, it is not clear that the final results are outperforming the state of the art as claimed in the paper.\n3.  Also, it is not clear why the authors didn't use more popular datasets for few-shot learning like miniImageNet, tieredImageNet, CIFAR-FS and FC100.\n4. Curiously, the paper defines the contributions as a method that outperforms the Stanford-Car few shot fine-grained learning as well as the ablation study and the visualisation aspect of the method. It is recommended to define better the theoretical contributions.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject