BEYOND SUPERVISED LEARNING: RECOGNIZING UNSEEN ATTRIBUTE-OBJECT PAIRS WITH VISION-LANGUAGE FUSION AND ATTRACTOR NETWORKS
Authors: Hui Chen
Zhixiong Nan
Nanning Zheng
Venue:  (2020)
Keywords: image understanding

Abstract: This paper handles a challenging problem, unseen attribute-object pair recognition, which asks a model to simultaneously recognize the attribute type and the object type of a given image while this attribute-object pair is not included in the training set. In the past years, the conventional classifier-based methods, which recognize unseen attribute-object pairs by composing separately-trained attribute classifiers and object classifiers, are strongly frustrated. Different from conventional methods, we propose a generative model with a visual pathway and a linguistic pathway. In each pathway, the attractor network is involved to learn the intrinsic feature representation to explore the inner relationship between the attribute and the object. With the learned features in both pathways, the unseen attribute-object pair is recognized by finding out the pair whose linguistic feature closely matches the visual feature of the given image. On two public datasets, our model achieves impressive experiment results, notably outperforming the state-of-the-art methods.

Under review as a conference paper at ICLR 2020
BEYOND SUPERVISED LEARNING: RECOGNIZING UNSEEN APTRIBUTE-OBJECT PAIRS WITH VISION- LANGUAGE FUSION AND ATTRACTOR NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
This paper handles a challenging problem, unseen attribute-object pair recogni- tion, which asks model to simultaneously recognize the attribute type and the object type of given image while this attribute-object pair not included in the training set. In the past years, the conventional classifier- based methods, which recognize unseen attribute-object pairs by composing separately-tranne attribute classifiers and object classifiers, are strongly frustrated. Different from conven- tional methods, we propose a generative model with a visual pathway and a lin- guistic pathway. In each pathway, the attractor network involved to learn the intrinsic feature representation to explore the inner relationship between the at- tribute and the object. With the learned features in both pathways, the unseen cutribute-object pair is recognized by finding out the pair whose linguistic feature closely matches the visual feature of the given image. On two public dataset- s, our model achieves impressive experiment results, notably outperforming the state-of-thee ffthheear art methods.
INTRODUCTION
Inferring the unknown from the known signals the advanced intelligence. This paper targets to teach the machine to recognize unseen attribute-objeet pairs based on seen attribute-object pairs. For example, teach the machine to recognize the "sliced apple" (unseen pairs) after letting the machine observe the samples of "green apple" and "sliced tomato" (seen pairs), as shown in Fig. (upper). This problem is termed as unseen attribute-object pair recognition.
Unseen attribute-object pair recognition meaningful and challenging problem. Deep neural net- work techniques (Krizhevsky et al.) 2012: Lecun et al. 2015; Simonyan & Zisserman 2015 He et al. 2015), especially supervised learning techniques, have achieved impressive successes on var- ious tasks. However, due to the large number of possible objjbut pairs, supervised learning methods, asking for massive data annotations for each attribute-object pair, will confront the com- position explosion disaster. Therefore, it is meaningful based on seen pairs. The main challenge of the unseen attribute-object pair recognition that the testing attribute-object pairs are included in the training set. The abstractness of attributes further increases the challenge. Another challenge results from the similarity between some attributes such as "huge" and "big".
recognize unseen attribute-objed pairs
To recognize unseen attribute-object pairs, conventional methods (Chen & Grauman 2014| Misra al., 2017) typically learn attribute classifiers and object classifiers the first, and then recognize unseen pairs by composing these separately-traiin -rained classifiers, which ignore the inner relationship between attributes and objects. The conventional dlassifier-basee methods also ignore the abstract- ness of attributes. It is the fact that objects can be accurately classified since the same type of objects share similar appearance. As shown in Fig. | (middle), different cars present similar characteristic. However, the attribute is abstract, and the same type of attribute varies significantly when describing different types of objects. For example, as shown in Fig. I (lower), the visual characteristics of the attribute "beautiful" are notably different from each other when describing sunset, aurora, mountain. Due to the abstractness of attributes, the classifier- bassed methods achieve lower perfor- mance attribute recognition than that the object recognition, which further leads the low

---

Under review as a conference paper at ICLR 2020
green apple
slioed tomato
sliced apple
beautiful
beautiful
beautiful
Figure 1: Problem definition and analysis.
accuracy of attribute-object pair recognition. In addition, though an image conveys rich informa- tion in internal structure, the conventional classifier-based methods are designed to learn feature representation to simply infer a label, and any information not required to infer the label is omitted. Unseen attribute-object pair recognition is high-level and complex vision problem relying the deep and comprehensive understanding of an image, which asks a model to learn an intrinsic feature representation conveying the rich information of an image.
Based on the above observations, we propose dual pathway generative model. The dual pathways the visual pathway and the linguistic pathway, respectively. In each pathway, taking the initial feature representation as the input, we employ the attractor networks to iteratively learn the intrinsic feature representation. We design several loss functions optimize the model. Given the optimized model and all possible attribute-object pairs, the recognition is realized using voting inference method. In the experiments, we compare our model with several state-of-the-art models on two challenging public datasets. Experiment results demonstrate that our model outperforms the state- of-the-an art models by large margins.
The contributions of this paper are as follows: (1) We for the first time introduce the attractor net- works recognize unseen attribute-object pairs; (2) Our method exhibits much better performance than conventional methods and state-of-the-as methods two challenging public datasets.
2 RELATED WORK
Unseen Attribute-Ohjec Pair Recognition The intuitive idea recognize unseen attribute- object pairs is combining attribute classifiers and object classifiers (Chen & Grauman] 2014: Misra 2017). However, these classifier-based methods separately process the attribute and the object, ignoring the inner relation between the attribute and the object. As a result, these methods do not achieve satisfied performance and heavily suffer from the "domain shift" problem (Fu et al. 2015) the distribution of the testing data is different from that of the training data. To overcome the "domain shift" problem, Nagarajan & Grauman (2018) creatively proposes to model an attribute as an operator and an attribute-object pair as an object vector that is "operated" by the operutor, and the model presents competitive results. Recently, Nan al. (2019) introduces generative model with the encoder-decoder mechanism and obtains ttate-of-the-ane results. Motivated by these works, we propose a generative model with attractor networks and languaee fusion mechanism to recognize unseen attribute-object pairs.
The attractor network a recurrent neural network, which is origi- nally proposed memory model to represent concept (Hinton & Shallice 1991 Herrmann al.) 1993). The model implies that a concept in the semantic memory is represented by a set nodes that are mutually connected according their semantic relatedness (Lerner al. 2010). Given the
Semantic Attractor Network
2

---

Under review as a conference paper at ICLR 2020
initial nodes to represent concept, the value of each node evolves over time to reach stable state,
which is called an attractor (Plaut & Shallice 1993 Masson 1995; Cree et al. 1999). A set of states
surrounding attractor is termed as the basin of attraction (Zemel & Mozer 2001 ichi Asakawa, 2013). The basin of attraction plays key role in the evolution of an attractor network, and the states surrounding attractor represent similar concepts.
For unseen attribute-object pair recognition, a model is required to learn an intrinsic representation of the given image. The attractor network is able recurrentyy learn a set of mutually connected nodes to represent the input image, and this representation is stable and intrinsic. In addition, the basin of attraction allows the model to associate similar unseen attribute-object pairs with the seen pairs, which is significant for unseen attribute-object pair recognition. Therefore, we involve attrac- tor networks in our model to further improve the stability and generalization of the initial feature representations in both visual pathway and linguistic pathway.
Zero Shot Learning The supervised learning is one of the most important techniques in artificial intelligence, presenting striking advantages a series of tasks in object detection (Sermanet al.) 2013), machine translation (Bahdanau et al. 2014) and speech recognition (Graves et al.] 2013). However, the supervised learning heavily relies on massive data annotations. "The large scale data annotation is time-consumingg and some data are difficult to annotate. Therefore, zero shot learning (ZSL) gradually draws researchers' interest (Larochelle et al. 2008 Palatucci al. 2009). ZSL aims to recognize unseen objects (i.e., object types in the testing set are not included in the training set). To recognize unseen objects, early works including Lampert et al. (2009) 2014) typically learn projection from the input visual space to semantic space where the attribute descriptions of unseen objects are known. Then, given an image, the model extracts visual feature in the visual space and projects it into the semantic space to obtain the attribute prediction. The recognition is realized by finding the unseen object whose attribute description closest with the attribute prediction. For these methods, attribute classifiers are trained separately and the relation between attributes are ignored. In order to mitigate this issue, some works seek to embed the visual features and the attribute descriptions into a common latent space (Ba et al. 2015 Wang et al. 2019). However, the input visual space and the latent space have their own manifold structure, which usually leads to significant variation in recognition performance. Therefore, Li et al. (2017) proposes alternate optimization mechanism to make the latent space consistent with the input space. However, the above methods lack either the ability to learn the bi-directional mappings between the visual space and the semantic/latent space or a flexible metric to evaluate the similarity between different kinds of features, thus Huang et al. (2018) leverages the Generative Adversarial Network (GAN) generate various visual features conditioned on class labels and maps each visual feature to its corresponding semantic feature. then recognizes unseen objects by measuring the similarity between the visual feature and textual feature.
pncluddiglLampeette
3 APPROACH
3.1 PROBLEM FORMULATION
For unseen attribute-object pair recognition, the training set (seen pairs) is defined as {zt, (af, o where x is the 4th image, and (af, of) E L, is its corresponding attribute la- bel and object label. The testing set (unseen pairs) defined as U = {x)), (af, of)) where 2² is the jth image, and oa) E f. is its corresponding attribute label and object label. The seen pairs and unseen pairs are disjoint, i.e., L, n Lu = O. During the testing, we identify the attribute-object pair label of given image from all possible pairs possible pairs.
where n,, is the number of all
3.2 NETWORK ARCHITECTURE AND DATA TRANSITIONS
The overview of our network architecture is illustrated in Fig. 2 The model consists of the visual pathway to process the visual data and the linguistic pathway to process the linguistic data. For the visual pathway, given an input image, Convolution Neural Network (CNN) is used to extract the initial visual feature v which is further processed by the visual encoding module, obtaining the visual encoder feature e e serves as the input of an attractor network, which outputs the
3

---

Under review as a conference paper at ICLR 2020
decoding module
atbrador network
72
Visut pathoey
CNN
000001
512
2048
512
1024
2
Visudl uncoding module
attractor network
word
emlesidings
Linguste pathoay
clean
concar
highwlay
2040
512
512
apple
sloodd
300
1024
2
liguustic encodiag module
Figure 2: The overview of our network architecture. The network is composed of the visual pathway and the linguistic pathway. In the visual pathway, an input image is processed by the Convolution Neural Network (CNN), the visual encoding module and attractor network, obtaining the visual attractor feature AV. In the linguistic pathway, the data flows are different for the training procedure and testing procedure. During the training, the linguistic attribute feature af and the linguistic object feature of that correspond to the input image are firstly extracted, and then processed by the linguistic encoding module and attractor network, obtaining the linguistic attractor feature A The loss measuring the similarity between A and A as well as other losses are computed to train the neural network. During the testing, the linguistic attractor features of all possible unseen attribute-ojject pairs are computed (for simplicity, only one pair is illustrated in the figure) and compared with the visual attractor feature of the input image predict the most likely attribute- object pair.
visual attractor feature A eV also serves as the input of the decoding module, which outputs the reconstruction visual feature "².
For the linguistic pathway, the attribute word and the object word are embedded as the linguistic attribute feature at and the linguistic object feature of, respectively. at and of are processed by the linguistic encoding module, obtaining the linguistic encoder feature et. ef serves as the input of attractor network, which outputs the linguistic attractor feature A
Visual Encoding Module computed as:
Given the initial visual feature xV, the visual encoder feature ex
e = Sparr (x) ( (z))I)
where concat represents the concatenation operation, f (-), f8 (+). and (.air() are three linear functions.
Visual Attractor Network As mentioned above, an attractor network consists of a set of mutually- connected nodes. Given the initial value of all nodes, the attractor network evolves over time to update the value of each node and finally reaches stable state. We use the visual encoder feature to initialize all nodes, and follow the same update equations described in the work of Devereux al. 2018) to iteratively update their values.
In detail, for node let g, be the input of the node at time i. and (t) be the activation value of the node at time a, (t) computed by sigmoid function a(-) which takes q,, (t) as the input: a..(1) = a (gm(t)). (t)). (t) is composed of two parts, the inner input and the external input. The inner input is the input of the node n at time - 1, denoted as qn (t 1). The external input time is weighted sum of the activation values of other nodes connected to the node n, denoted as (t).
The external input y (t) of the node n at time t in the visual pathway is defined as:
p = Σ a a (t 1) + bn
(2)
4

---

Under review as a conference paper at ICLR 2020
where Win represents the connection weight from node i to node n, a. 1) represents activation value node at time t in the visual pathway, and b, represents the bias of node n.
The input 9 (t) of the node at time t in the visual pathway is calculated linear combination:
(t) (t) (1 cu (t 1) - 1) represents the input of the node n at time proportion parameter.
(3)
where (t-
the visual pathway, and cy is
The activation value (t) of the node time in the visual pathway is calculated as sigmoid function (.)) of its input 9, (t):
= o
(4)
Using (0) and (0) to represent the initial input and the initial activation value of all nodes of the attractor network in the visual pathway, we initialize the attractor network's s states by setting Q² (0) e² AA (0) aee"). Once initialized, the attractor network iterates recurrently according to the update equations defined in Eq. 2 Eq and Eq to the time-tick T,, and outputs the final
activation value, which is denoted as A
Visual Decoding Module
For the decoding module, we reconstruct the original visual feature x
by linear function:
8 =
=== f
(5)
Linguistic Encoding Module Given the linguistic attribute feature af and the linguistic object feature of, the linguistic encoder feature ef is computed as:
Spa (concat ff 8
(6)
where concat represents the concatenation operation, fa (-), fo), and (fuer(() are three linear functions.
Linguistic Attractor Network Similar with the visual attractor network, initialize the linguis- tie attractor network by setting (0) = ( = ((ef) where (0). A represent the initial input and the initial activation value of all nodes of the linguistic attractor network. For node n, its external input (t) at time is weighted sum:
(t)
Σ Win a 1) + bn inf
where represents the connect weight from node to node af 1) represents the activation value node at time t 1, and b,, represents the bias of node n.
The input of (t) of the node at time t is calculated as:
(t) c
(t) + (1 -c)
(8)
where of (t- - 1) represents the input of node n at time and e is proportion parameter. The activation value af(() of the node at time / is calculated as:
a = (t.
(9)
Given the initial values of all nodes, the attractor network recurrently iterates according update equations defined in Eq. Eq 8 and to the time-tick Ti, and outputs the final activation value, which is denoted as A
3.3 Loss FUNCTIONS
Attractor Loss Let W be the matrix that represents the connection weights of all nodes in an attractor network and Wij be the connection weight from the node to the node j. The connection weight from the node to the node is required be the same with the weight from j to i (i.e.,
5

---

Under review as a conference paper at ICLR 2020
Wij Wy:). In addition, any node in attractor network does not connect with itself (i.e., Waa 0). Therefore, the attractor loss is defined as:
Lot = I| w 1/2 + [[diag(W)llz
(10)
where diag(.) is a function to extract the diagonal elements in the matrix W, and || ||2 is the L2 norm.
The attractor network recurrently iterates over time, and small differences in initial conditions may yield notable fluctuation of the output. Hence, define the conditional loss Led to constrain e² and ef to be close to each other:
Conditional Loss
Led = ||e - 1l
(11)
Fusion Loss To fuse the information in the visual pathway and linguistic pathway, we design the fusion loss to minimize the L2 distance between the visual attractor feature AAY and the linguistic attractor feature -
f = ||A4 A
(12)
Decoding Loss Inspired by recent works for unseen attribute-object pair recognition (Nan et al, 2019) and zero-shot learning (Kodirov et al. 2017), introduce the decoding loss minimize the 1.2 distance between the initial visual feature x² and the reconstruction visual feature i defined in Eq.
Ld || S
(13)
The motivation of designing the decoding loss is to learn better visual encoder feature ex
Discriminative Loss For some pairs, either the attribute or the object feature is dominant to represent the whole pair. Therefore, preserve the individual property of the attribute and object, we design the discriminative loss, which defined as a cross entropy loss:
Lais - log(y Y log(u.))
(14)
where ya denotes the attribute ground truth, va denotes the attribute prediction, y denotes object ground truth, and 2a denotes the object prediction, va and yb are computed as:
(15)
Vo = (o (x²))
(16)
where (-a(() and (.) are two softmax functions, and (-) and (C(() linear functions.
3.4 LEARNING AND INFERENCE
Let W be all parameters of our proposed model. During the training, we learn the parameters by minimizing the loss functions:
W arg min a + nL LLde + ALdis + BLeedd
(17)
We use the ADAM algorithm (Kingma Ba 2014) learn the parameters.
During the inference, we propose a voting inference method. For an input image. its visual attractor feature A computed, at the same time, the linguistic attractor features of all n., possible unseen attribute-object pairs are computed, which are denoted as {Af)"= )ー By computing the 1.2 distance between A' and each Af, the pair that corresponds to the minimum distance is taken the initial attribute-object pair recognition for the input image. For the images belonging to the same pair, we use the recognitions for all images to vote a pair label to be the final autribute-object pair recognition.
EXPERIMENTS
4.1 DATASETS
We evaluate the model on two public challenging datasets, the MIT-States dataset (Isola et al. 2015) and the UT-7appos50K dataset Yuu & Grauman 2014). The MIT-States dataset is composed of

---

Under review as a conference paper at ICLR 2020
63,440 images, covering 115 attribute classes, 245 object classes, and 1,962 attribute-object pairs. Each image is annotated with an attribute-object pair label like "dirty kitchen". Same with the setting in previous works (Misra et al. 2017 Nagarajan & Grauman 2018 Nan et al. 2019). 1262 pairs are used for the training and 700 pairs for the testing. UT--aapos550K is a fine-grained shoes dataset including 16 attribute classes and 12 object classes, totally with 50,025 images. Same with the setting in previous works (Misra et al. 2017; Nagarajan & Grauman 2018 Nan et al. 2019). 83 pairs are used for the training and 33 pairs for the testing.
4.2 BASELINES AND METRIC
Five baseline methods, including two conventional methods and three state-of-the-art methods, are compared with our method. We briefly introduce the baselines as follows:
-ANALOG (Chen & Grauman 2014) is conventional method that recognizes unseen attribute- object pairs using a set of seen object-specific attribute classifiers;
REDWINE (Misra et al. 2017) is a conventional method that recognizes unseen attribute-object pairs by composing attribute and object classifiers;
SAE (Kodirov al. 2017) is state-of-the-art ZSL method. It firstly projects the input feature into a semantic space where the auxiliary information unseen pairs is known, and the recognition is realized by finding the pair whose auxiliary information is closest with the input feature;
ATTOPERATOR (Nagarajan & Grauman 2018) is a state-of-the-arr method that predicts unseen pairs by comparing the visual feature of the given image with all possible attribute-object features that are modeled as the object vectors transformed by attribute operators;
GENERATE (Nan al. 2019) is state-of-the-ar method that predicts unseen pairs by compar- ing the visual feature of the given image with the linguistic features of all possible pairs in a latent space.
We use the top-1 accuracy as evaluation metric, which is widely adopted by the state-of-the thetthrrrtt methods (Nagarajan & Grauman 2018 Nan al. 2019)
4.3 IMPLEMENTATION DETAILS
We use the ResNet- 18 (He et al. 2015) network pretrained on the ImageNet dataset (Deng et al, 2009) to extract the 5 512-dimension visual feature V.V. For fair comparison, neither the fine-tuning operation nor data augmentation is applied to our method and baseline methods. We use the Ge model (Pennington et al., 2014) to extract the 300-dimension linguistic attribute feature af and linguistic object feature o=. In the visual encoding module and linguistic encoding module, the attribute feature and object feature are enlarged as 1024-dimension features to improve the repre- sentation potentiality, which are then concatenated as 2048-dimension features to represent attribute- object pairs. To relieve the computation burden of attractor networks, ex and ef are transformed as 51 12-dimension features to serve as the input of attractor networks, which output the 312-dimension attractor features A and AA.
Our model is implemented using TensorFlow (Abadi et al. 2016). Every linear function involved in Eq. D Eq. 5 Eq. 6 and Eq. [6) is implemented by one fully connected layer. During the training. the initial learning rate is 0.002, which decays by 0.95 every epoch. The batch size is 128 using a Nvidia 2080 GPU. Using the grid-search method, we set parameters a,, 7), X, B in 17 be 1.0, 2.0, 6.0, 2.0 for the MIT-States dataset and 1.0, 3.0, 6.0, 1.0, 2.0 for the UT- Zappos50K dataset. Attractor parameters C,, Ct Tu, Ti are set as 0.1, 0.2, 20, 10 for the MIT-States dataset and 0.08, 0.16, 25, 15 for the UT-ZZppos550K dataset. The training is terminated (80 epochs on the MIT-States dataset and 120 epochs on the UT-Zappos50K dataset) when the sum of losses slightly decreases. The weight connections and biases of all nodes in an attractor are randomly initialized.
4.4 EXPERIMENT DESIGN AND RESULT ANALYSIS
Compare with Baselines The comparative results with baselines are summarized in Tab. [ We observe that our method outperforms the baseline methods by large margins, achieving 114.6%
7

---

+dis Under review as a conference paper at ICLR 2020
Table 1: Top-1 unseen attribute-ohject recognition accuracies of baseline methods and method on the MIT-States dataset and UT-Zappos50K dataset.
Methods
MIT-States(%)
UT-Zappos(%% (%)
CHANCE
0.14
3.0
ANALOG & Grauman 2014)
1.4
18.3
SAE (Kodrov et al. 2017
14.0
31.0
REDWINE Missa et a.. 2017)
12.5
40.3
OPERATOR (Nagarajan & Grauman
2018
14.2
46.2
GENERATE (Nan al. 2019
17.8
48.3
38.2
73.7
Table 2: Top-1 unseen attribute-object recognition accuracies of the "Base" model and "Base Attractor" model on the MIT--ttates dataset and UTTZZappos50K dataset. "Base" denotes the model without attractor networks, and "Base + Attractor" denotes the model with attractor networks.
Models
MIT--States(%)
UT-Zappos(*))
Base
36.1
51.8
Base Attractor (our)
38.2
73.7
Table 3: Top-1 accuracies of different loss function compositions on the MIT--ttates dataset and UT--Zappos50k dataset. "at", "fus", "de", "dis", and "cd" represent the attractor loss Tuat fusion loss L fus, decoding loss Lde, discriminative loss Ldis. and conditional loss Led, respectively.
Losses
MIT-States(%)
UT-ZZppps("0)
at+fus (basic loss)
0.14
7.8
0.20
11.2
+ed
0.21
12.3
+de
35.0
68.0
+dis+ed
0.3
15.1
+dis+de
36.4
68.9
+cd+de
36.8
72.4
+dis+tc++de (our)
38.2
73.7
and 52.6% accuracy improvements over the second best method on the MIT--States dataset and on the UT-Zappos50KK dataset. The reasons are two-fold: I) the attractor networks recurrently learn the intrinsic and stable feature representations, allowing the deep understanding of an image; 2) the encoder-decoder mechanism improves the robustness of feature representations, which is significant for alleviating the "domain shift" problem.
Attractor Significance This experiment is targeted to validate the effectiveness attractor net- works by comparing the recognition accuracies of the base model (without attractor networks) and our model (with attractor networks). The results are shown in Tab. 2 we can observe that the attrac- network vital for the accuracy improvements. Our model achieves 5.8% accuracy improve- ment over the base model on the MIT-States dataset and 42.3% improvement on the UT-Zappos50K dataset, demonstrating the significance of attractor networks.
Loss Compositions In this experiment, we evaluate the effects different loss compositions, and the results are shown in Tab, 3 Since the attractor loss Lat is indispensable for the attractor net- work and the fusion loss L Lus indispensable for the vision-language fusion mechanism, the sum of these two losses (al+fus) is taken as the basic loss. If only the basic loss used, the accuracy only 0.14% on the MIT-States dataset and 7.8% on the UT-Zappos50K dataset. When the discrimi- native loss (+dis) or conditional loss (+cd) is added to the basic loss, the accuracy slightly improves. When the decoding loss (+de) is added to the basic loss, model achieves the striking accuracy improvements on both datasets, demonstrating that the encoder-decoder decoder mechanism in our model effective. We can also observe that the model tends to achieve higher accuracy when adding more losses, and the model obtains the highest accuracy when all losses are used, which further validates the effectiveness of the individual loss.
8

---

Under review as a conference paper at ICLR 2020
Table 4: Top-1 accuracies of three parameter sharing settings and setting on the MIT--States dataset and UT-Zappos50k dataset. fø = f represents the parameter sharing in the visual en- coding module, fa for represents the parameter sharing the linguistic encoding module, and Att Att represents the parameter sharing in the visual attractor network and linguistic attractor network.
Parameter Sharing
MIT-States(%)
UT-Zappos(%o)
fx fo SC f6
37.8
60.1
36.1
50.6
Att Att
2.4
1.6
no sharing (our)
38.2
73.7
sheepslin, boots-ankle
broken drum Inflated boat
light shirt
scratched phone
satin, sundals
shrepskin, boots-aklee
rubber, sandals
syathetic, sandals
rusly truck andest Jewelry
Durn es, shoes-lott flats
suede, slipers
sylon, bosts-ankle sy
barren lake
thick mocillaee
bage kitchen
cotton, shoes- flats
Jeathrr slippers
rubbir hostsonkle Bylon, bosts-anke
Recggitiinn results on MIT-Stales
Recogrition results on UTT-apposSkk
Figure 3: Samples of unseen attribute-object pair recognition the MIT-States dataset (left) and UTT-aappos0Ok dataset (right).
Parameter Sharing The attribute and object are fundamentally different entities, thus they should be equally processed. Motivated by this idea, illustrated in Fig. we use different projection functions (f2 and f.') to obtain the attribute feature and the object feature in the visual encoding module, and we also use different projection functions (fe and fo process the attribute feature and the object feature in the linguistic encoding module. In addition, the vision information differs from the language information in data structure, thus we use two attractor networks to separately process the visual feature and the linguistic feature. To validate our ideas, three parameter sharing experiments are conducted: 1) the attribute projection function fe' in the visual encoding module shares parameter with the object projection function fy in the visual encoding module (i.e., let ft = ==f") ), 2) the attribute projection function fe. in the linguistic encoding module shares parameter with the object projection function fe in the linguistic encoding module (i.e., let fc f6. and 3) the attractor network in the visual pathway Att shares the same parameter with the attractor network in the linguistic pathway Att (i.e., let Att AttS). The experiment results are shown in Tab, 4 and we can observe that our setting corresponds to the highest accuracies on two datasets, validating our ideas.
Qualitative Result Analysis Fig. 3 shows some samples of recognitions on the MIT-States dataset (left) and UT-Zappos50k dataset (right). In the figure, the black texts on the top of images represent the ground truth, the green texts represent true recognitions, and the red texts represent false recogni- tions. We can observe that our model can correctly recognize some pairs with abstract attributes such as "tight shirt" and "satin sandal". However, as shown in the second row, some images are falsely recognized. One reason for false recognitions that some attributes or objects present similar visual features. For example, the "rusty truck" is recognized "old car" and "barren creek" is recognized as "barren lake". Actually, "rusty" and "old" exhibit the similar visual feature and "creek" shares the similar visual feature with "lake". Another reason is that one object type may be subclass of another object type. For example, "ancient jewelry" is recognized as "thick necklace", Actually, the necklace is kind of jewelry. In addition, object having multiple attributes may also leads

---

Under review as a conference paper at ICLR 2020
to false recognitions. For example, "clean kitchen" is recognized "huge kitchen". Actually, the kitchen indeed is huge and clean.
5 CONCLUSION
This paper studies challenging and meaningful problem termed as unseen attribute-object pair recognition. To handle the problem, we propose a vision-language fusion generative model that involves attractor networks and the encoder-decoder mechanism. The proposed model presents impressive performance.
Our main conclusions are follows: 1) Unseen attribute-object pair recognition is a complex prob- lem that asks a model to learn the intrinsic feature representation and overcome the "domain shift" problem. 2) The attractor network presents the notable potentiality to learn an intrinsic feature representation and the encoder-decoder mechanism is significant for alleviating the "domain shift" problem. 3) During the testing, the voting inference method contributes to the accuracy improve- ments.
Unseen attribute-object pair recognition is high-level vision problem, asking model to be smart as humans. Therefore, we will further explore this problem using cognition-inspired methods the future.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint 1603:v:660304400 2016.
Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, and Ruslan Salakhutdinov. Predicting deep zero-shot convolutional neural networks using textual descriptions. In IEEE International Conference on Computer Vision, pp. 4247-4255 2015.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. Computer Science, 2014.
Chao-Yel Yeh Chen and Kristen Grauman. Inferring analogous attributes. In IEEE Conference Computer Vision and Pattern Recognition, pp. 200-207, 2014.
George S. Cree, Ken McRae, and Chris McNorgan. An attractor model of lexical conceptual pro- cessing: Simulating semantic priming. Cognitive Science, 23(3):371-4144 1999.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Barry J. Devereux, Alex Clarke, and Lorraine K. Tyler. Integrated deep visual and semantic attractor neural networks predict fmri pattern-information along the ventral object processing pathway. bioRxiv, pp. 302406, 2018.
Yanwei Fu, Timothy M Hospedales, Tao Xiang, and Shaogang Gong, Transductive multi-view zero-shot learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(11), 1): 2332-2345, 2015.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recur- rent neural networks. In IEEE international conference on acoustics, speech and signal process- ing, pp. 6645-6644 JEEE, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. arXiv preprint arXiv: 2015.
M. Herrmann, Eytan Ruppin, and Marius Usher. neural model of the dynamic activation of memory. Biological Cybernetics, (8(5):4554463 1993.
10

---

Under review as a conference paper at ICLR 2020
Geoffrey E Hinton and Tim Shallice. Lesioning an attractor network: Investigations of acquired dyslexia. Psychological review, 98(1):74, 1991.
He Huang, Changhu Wang, Philip S. Yu, and Chang-Dong Wang. Generative dual adversarial network for generalized zero-shot learning. In IEEE Conference Computer Vision and Pattern Recognition, pp. 801-810, 2018.
Shin ichi Asakawa. Re-evaluation of attractor neural network model to explain double dissociation in semantic memory disorder. Psychology, 4(3):363-375 2013.
Phillip John Isola, Joseph Jaewhan Lim, and Edward H Adelson. Discovering states and transfor- mations in image collections. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1383-1391, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning. arXiv preprint /rX4v:1704.088 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. Neural Information Processing Systems, 141(5):107--1005, 2012.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Conference Computer Vision and Pattern Recognition, pp. 951-955 2009.
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-ased classification for zero-shot visual object categorization. IEEE Transactions Pattern Analysis and Machine In- telligence, 36(3):533 465, 2014.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. Zero-data learning of new tasks. In AAAI, volume 1, pp. 3, 2008
Yann Lecun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436, 2015.
Itamar Lerner, Shlomo Bentin, and Oren Shriki. Automatic and controlled processes in semantic priming: an attractor neural network model with latching dynamics. Proceedings of the Annual Meeting of the Cognitive Science Society, 32(32), 2010.
Yan Li, Junge Zhang, Jianguo Zhang, and Kaiqi Huang. Discriminative learning of latent features for zero-shot recognition. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7463-74777 2018.
Yanan Li, Donghui Wang, Huanhang Hu, Yuetan Lin, and Yueting Zhuang. Zero-shot recognition using dual visual-semantic mapping paths. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5207-5215, 2017.
Michael E. J. Masson. A distributed memory model of semantic priming. Journal of Experimental Psychology: Learning. Memory and Cognition, 21(1):3-23, 1995.
Ishan Misra, Abhinav Gupta, and Martial Hebert. From red wine to red tomato: Composition with context. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 1160-1100 1169, 2017.
Tushar Nagarajan and Kristen Grauman. Attributes as operators: Factorizing unseen attribute-object compositions. In Proceedings of the European Conference on Computer Vision, pp. 172-190, 2018.
Zhixiong Nan, Yang Liu, Nanning Zheng, and Song-Chun Chun Zhu. Recognizing unseen attribute-object pair with generative model. In Thirty- Third AAAI Conference on Artificial Intelligence, pp. 8811- 8818, 2019.
Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. Zero-shot learning with semantic output codes. In Advances in neural information processing systems, pp. 1410-1418, 2009.
11

---

Under review as a conference paper at ICLR 2020
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recog- nizing scene attributes. In IEEE Conference on Computer Vision and Pattern Recognition. pp- 2751-2758, 2012.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1532-1543, 2014.
Global vectors for word
David Plaut and Tim Shallice. Deep dyslexia: A case study of connectionist neuropsychology. Cognitive newropsychology, 1065):377-500, 1993.
Mareus Rohrbach, Sandra Ebert, and Bernt Schiele. Transfer learning in transductive setting. In Advances in Neural Information Processing Systems 26, pp. 46-54, 2013.
Pierre Sermanet, David Eigen, Xiang Zhang, Michaël Mathicu, Rob Fergus, and Yann LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229, 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations 2015, 2015.
Xinsheng Wang, Shanmin Pang, Jihua Zhu, Zhongyu Li, Zhiqiang Tian, and Yaochen Li. Visual space optimization for zero-shot learning. arXiv preprint arXiv:1907.00330, 2019.
Aron Yu and Kristen L Grauman. Fine-grained visual comparisons with local learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 192-199, 2014.
Richard S. Zemel and Michael C. Mozer. Localist attractor networks. Neural Computation, 13(5): 1045-1064, 2001.
12


Reviews:
[{"review_id": "SylOEzmTqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to handle the unseen attribute-object pairs recognition, which asks a model to simultaneously recognize the attribute type and the object type of a given image while this attribute-object pair is not included in the training set. The claimed contribution includes: (1) they for the first time introduce the attractor networks to recognize unseen attribute-object pairs; (2) their method exhibits much better performance than conventional methods and state-of-the-art methods on two challenging public datasets.  \n\nI found this paper poorly written. First, the introduction of attractor networks lacks intuition, explanation, and experiment support. The only support of attractor network is Table2, which show marginal improvement on MIT-States and significant improvement on UT-Zappos. Even for the two picked dataset, the improvement is not consistent, this is not enough to show the effectiveness of attractor network. The authors could design some visualization/metrics to show what the attractor networks have learned, i.e., the visualization of the nodes. The authors should also provide more ablation studies on the attractor network, i.e., the number of the nodes, the time-tick Tv, etc. After all, the experiment results in this paper shows little evidence of how the attractor network works, and the insight of how it works if so.\n\nSecond, the ablation study and conclusion are confusing. Combining Table 1,2&3, we can see that the decoding loss is the magic. Therefore, I should say that the decoding loss is the main reason for the ''much better performance'' in this paper, instead of the attractor network. However, the baseline method GENERATE already has the reconstruction/decoder loss. Why is its number low? \n\nI did not get the meaning of this sentence in Section 3.4. \"For the images belonging to the same pair, we\nuse the recognitions for all images to vote a pair label to be the final attribute-object pair recognition.\"", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SylMYma6YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper attempts to tackle unseen object-attribute recognition in still images. This follows a line of works that addresses such a problem using embedding (with operators), classification and generative approaches. The paper makes use of the visual attractor network, extracting different visual representations for objects and attributes. A collection of losses (some regularisation and some distance in embedding space) are proposed. I believe the paper (except A below) is technically correct.\n\nThis paper was particularly tricky to read. While convincing at parts, there are two worrying statements. One I currently believe (pending the authors' response) is a scientific flaw (A), the other causes the results to be potentially incomparable to published works (B). I detail these next.\n\n(A) Scientific Flaw: Quoting from the paper, at inference (i.e. for a test image), \"we propose a voting inference method. For an input image, its visual attractor feature AV is computed, ... By computing the L2 distance ... the pair that corresponds to the minimum distance is taken as the initial attribute-object pair recognition for the input image. For the images belonging to the same pair, we use the recognitions for all images to vote a pair label to be the final attribute-object pair recognition.\"\nIn my interpretation, the authors are taking all test images that are labeled using the same pair (e.g. 'sliced-apple' - assuming this is an unseen pair). They then use all these to make a final decision for all these images. \nI believe my interpretation is correct because their statement \"images belonging to the same pair\" can only exist in testing for their target scenario (unseen object-attribute pair).\nIf my interpretation is correct, this is A MAJOR FLAW in the approach. How do the authors know that a collection of test images \"belong to the same pair\"?? This knowledge can only be acquired from labels of the test set and cannot be used as part of any algorithm.\nI would very much like for my interpretation to be naive, but I could not find a different explanation for this statement in page 6. The authors note in the conclusion that this voting approach is part of their contribution.\n\n(B) In reporting the results, the authors state that: \"Using the grid-search method, we set parameters \\alpha\u000b; ... [5 parameters] \f in Eq.17 to be 1:0; 2:0; 6:0; 3:0; 2:0 for the MIT-States dataset and 1:0; 3:0; 6:0; 1:0; 2:0 for the UTZappos50K\ndataset. Attractor parameters cv; cl; Tv; Tl are set as 0:1; 0:2; 20; 10 for the MIT-States dataset and 0:08; 0:16; 25; 15 for the UT-Zappos50K dataset.\"\nThis assumes that the authors are grid-searching all parameters, and evaluating the test set performance for each case, then reporting the maximum possible test performance per dataset. The authors do not reference a validation set they use to set these parameters instead. Importantly, the values are SIGNIFICANTLY different for each test set and there is no explanation of why the parameters vary largely. It is not clear how the performance on an new dataset would be, \nUp to my knowledge, the other methods that they compare to in Table 1, have not reported a grid-search over the parameter space for maximum performance of the test set. This IMO makes the comparative evaluation, which the authors deem to be 'impressive' quite unfair. It is not possible to assess the method's performance using these results.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkl7Aytg5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the so-called “unseen attribute-object pair recognition”,\nwhich asks a model to simultaneously recognize the attribute type and the\nobject type of a given image. \n\n1) Despite the authours claimed this is a novel task, it has been thoroughly studied long time ago. For example, NEIL [1], and comparative attributes [2]. So it would be advisable to thoroughly review those related literature.\n\n2) the whole framework is a two pathway encoder-decoder networks. The framework is well explained. It would be great if there are more words about the motivations why the network is designed in such a manner. In general, there is lack of discussion why the attractor networks are novel, and significant. \n\n3) In term of network structure, the loss functions introduced are pretty straight-forward. Please claim novelty about loss functins, if there is anything special.\n\n4) The experiments are pretty weak: Only on two small datasets. It would be great if there are more experiments on larger, or challenging datasets. Considering it’s almost ten pages; I would prefer more persuasive experiments to validate the novelties (but please better summarize the novelty again).\n\n\n\n[1] NEIL: Extracting Visual Knowledge from Web Data. 2012\n\n[2] Constrained Semi-Supervised Learning using Attributes and Comparative Attributes. ECCV 2012\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focuses on attribute-object pairs image recognition, leveraging some novel "attractor network".

At this stage, all reviewers agree the paper needs a lot of improvements in the writing. There are also concerns regarding (i) novelty: the proposed approach being two encoder-decoder networks; (ii) lack of motivation for such architecture (iii) possible flow in the approach (are the authors using test labels?) and (iv) weak experiments.
Decision: Reject