Title: OBJECT-CONTRASTIVE NETWORKS: UNSUPERVISED OBJECT REPRESENTATIONS

Abstract: Discovering objects and their attributes is of great importance for autonomous agents to effectively operate in human environments. This task is particularly challenging due to the ubiquitousness of objects and all their nuances in perceptual and semantic detail. In this paper we present an unsupervised approach for learning disentangled representations of objects entirely from unlabeled monocular videos. These continuous representations are not biased by or limited to a

:

discrete set of labels determined by human labelers. The proposed representation is trained with a metric learning loss, where nearest neighbors in embedding space are pulled together while being pushed against other objects. We show these unsupervised embeddings allow robots to discover object attributes that generalize to previously unseen environments. We quantitatively evaluate performance on a large-scale synthetic dataset with 12k object models, as well as on a real dataset collected by a robot and show that our unsupervised object understanding generalizes to previously unseen objects. Specifically, we demonstrate the effectiveness of our approach on robotic manipulation tasks, such as pointing at and grasping of objects. An interesting and perhaps surprising finding in this approach is that given a limited set of objects, object correspondences will naturally emerge when using metric learning without requiring explicit positive pairs. Videos of robotic experiments are available at sites.google.com/view/object-contrastive-networks Figure 1 : Object-Contrastive Networks (OCN): by attracting embedding nearest neighbors and repulsing others using metric learning, continuous object representations naturally emerge. In a video collected by a robot looking at a table from different viewpoints, objects are extracted from random pairs of frames. Given two lists of objects, each object is attracted to its closest neighbor while being pushed against all other objects. Noisy repulsion may occur when the same object across viewpoint is not matched against itself. However the learning still converges towards disentangled and semantically meaningful object representations which can be useful in autonomous robotics applications.

1. INTRODUCTION:

The ability to autonomously train to recognize and differentiate previously unseen objects as well as infer general properties and attributes is an important skill for robotic agents. Increased autonomy leads to robustness, one of the main challenges real-world robotics faces. It also renders scaling up data collection practical. Additionally, removing human supervision from the loop has the potential to enable learning richer and less biased continuous representations than ones supervised by a limited set of discrete labels. Unbiased representations can prove useful in unknown future environments different from the ones seen during supervision, a typical challenge for robotics.

In this work we present an unsupervised method that learns representations that disentangle perceptual and semantic object attributes such as class, function, and color. We automatically acquire training data by capturing videos with a real robot; a robot base moves around a table to capture objects in various arrangements. Assuming a pre-existing objectness detector, we extract objects from random frames within a same scene containing the same objects, and let the metric learning system decide how to assign positive and negative pairs of embeddings. Representations that generalize across objects naturally emerge despite not being given groundtruth matches. Unlike previous methods, we abstain from employing additional self-supervisory training signals such as tracking or depth. The only inputs to the system are monocular videos. This simplifies data collection and allows our embedding to integrate into existing end-to-end learning pipelines. We demonstrate that a trained Object-Contrastive Network (OCN) embedding allows us to reliably identify object instances based on their visual features such as color and shape. Moreover, we show that objects are also organized along their semantic or functional properties. For example, a cup might not only be associated with other cups, but also with other containers like bowls or vases.

The key contributions of this work are: (1) an unsupervised algorithm for learning representations of objects (naturally encoding attributes like class, color, texture and function) which generalize to previously unseen objects;

(2) showing monocular videos are sufficient to contrast similar and dissimilar objects pairs naturally without requiring explicit correspondences; (3) demonstrating the autonomy of the system, using a robot from data collection to tasks such as pointing and grasping similar objects to ones presented to it.

2. RELATED WORK:

Object discovery from visual media. Identifying objects and their attributes has a long history in computer vision and robotics (Tuytelaars et al., 2009) . Traditionally, approaches focus on identifying regions in unlabeled images to locate and identify objects (Sivic et al., 2005; Russell et al., 2006; Arora et al., 2007; Fritz & Schiele, 2008; Kim et al., 2008) . Discovering objects based on the notion of 'objectness' instead of specific categories enables more principled strategies for object recognition (Uijlings et al., 2013; Romea et al., 2011) . Several methods address the challenge to discover, track, and segment objects in videos based on supervised (Wang et al., 2014) or unsupervised (Kwak et al., 2015; Schulter et al., 2013; Haller & Leordeanu, 2017) techniques. The spatio-temporal signal present in videos can also help to reveal additional cues that allow to identify objects (Wang & Gupta, 2015; Jain et al., 2017) . In the context of robotics, methods also focus on exploiting depth to discover objects and their properties (Mishra et al., 2012; Karpathy et al., 2013) .

Many recent approaches exploit the effectiveness of convolutional deep neural networks to detect objects (Ren et al., 2015; Liu et al., 2016; Lin et al., 2017) and to even provide pixel-precise segmentations (He et al., 2017) . While the detection efficiency of these methods is unparalleled, they rely on supervised training procedures and therefore require large amounts of labeled data. Self-supervised methods for the discovery of object attributes mostly focus on learning representations by identifying features in multi-view imagery (DeTone et al., 2017; Lin et al., 2015) and videos (Wang & Gupta, 2015) , or by stabilizing the training signal through domain randomization (Doersch et al., 2015; Zhang et al., 2018) .

Some methods not only operate on RGB images but also employ additional signals, such as depth (Florence et al., 2018; Pot et al., 2018) or egomotion (Agrawal et al., 2015) to self-supervise the learning process. It has been recognized, that contrasting observations from multiple views can provide a view-invariant training signal allowing to even differentiate subtle cues as relevant features that can be leveraged for instance categorization and imitation learning tasks (Sermanet et al., 2018) .

Unsupervised representation learning. Unlike supervised learning techniques, unsupervised methods focus on learning representations directly from data to enable image retrieval (Paulin et al., 2015) , transfer learning (Zhang et al., 2017a) , image denoising (Vincent et al., 2008) , and other tasks (Dumoulin et al., 2016; Kumar et al., 2015) . Using data from multiple modalities, such as imagery of multiple views (Sermanet et al., 2018) , sound (Owens et al., 2016; Aytar et al., 2016) , or other sensory inputs (Dehzangi et al., 2017) , along with the often inherent spatio-temporal coherence (Doersch et al., 2015; Radford et al., 2015) , can facilitate the unsupervised learning of representations and embeddings. For example, Zagoruyko & Komodakis (2015) explore multiple architectures to compare image patches and Pathak et al. (2017b) exploit temporal coherence to learn object-centric features. Gao et al. (2016) rely of spatial proximity of detected objects to determine attraction in metric learning, OCN operates similarly but does not require spatial proximity for positive matches, it does however take advantage of the likely presence of a same object in any pair of frames within a video. Zhang et al. (2017b) also take a similar unsupervised metric learning approach for tracking specific faces, using tracking trajectories and heuristics for matching trajectories and obtain richer positive matches. While our approach is simpler in that it does not require tracking or 3D matching, it could be augmented with extra matching signals.

In robotics and other real-world scenarios where agents are often only able obtain sparse signals from their environment, self-learned embeddings can serve as an efficient representation to optimize learning objectives. Pathak et al. (2017a) introduce a curiosity-driven approach to obtain a reward signal from visual inputs; other methods use similar strategies to enable grasping (Pinto & Gupta, 2016) and manipulation tasks (Sermanet et al., 2018) , or to be pose and background agnostic (Held et al., 2015) . Mitash et al. (2017) jointly uses 3D synthetic and real data to learn a representation to detect objects and estimate their pose, even for cluttered configurations. Hickson et al. (2018) learn semantic classes of objects in videos by integrating clustering into a convolutional neural network.

3. UNSUPERVISED LEARNING OF OBJECT REPRESENTATIONS:

We propose an unsupervised approach to the problem of object understanding for multiple reasons:

(1) make data collection simple and scalable, (2) increase autonomy in robotics by continuously learning about new objects without assistance, (3) discover continuous representations that are richer and more subtle than the discrete set of attributes that humans might provide as supervision which may not match future new environments. All these objectives require a method that can learn about objects and differentiate them without supervision. To bootstrap our learning signal we leverage two assumptions: (1) we are provided with a general objectness model so that we can attend to individual objects in a scene, (2) during an observation sequence the same objects will be present in most frames (this can later be relaxed by using an approximate estimation of ego-motion). Given a video sequence around a scene containing multiple objects, we randomly select two frames I and Î in the sequence and detect the objects present in each image. Let us assume N and M objects are detected in image I and Î, respectively. Each of the n-th and m-th cropped object images are embedded in a low dimensional space, organized by a metric learning objective. Unlike traditional methods which rely on human-provided similarity labels to drive metric learning, we use a selfsupervised approach to mine synthetic synthetic similarity labels.

3.1. OBJECTNESS DETECTION:

To detect objects, we use Faster- RCNN (Ren et al., 2015) trained on the COCO object detection dataset (Lin et al., 2014) . Faster-RCNN detects objects in two stages: first generate class-agnostic bounding box proposals all objects present in an image (Fig. 1 ), second associate detected objects with class labels. We use OCN to discover object attributes, and only rely on the first objectness stage of Faster-R-CNN to detect object candidates. Examples of detected objects are illustrated in Fig. 1 .

3.2. METRIC LOSS FOR OBJECT ATTRIBUTE DISENTANGLEMENT:

We denote a cropped object image by x ∈ X and compute its embedding via a convolutional neural network f (x) ∶ X → K. Note that for simplicity we may omit x from f (x) while f inherits all superscripts and subscripts. Let us consider two pairs of images I and Î that are taken at random from the same contiguous observation sequence. Let us also assume there are n and m objects detected in I and Î respectively. We denote the n-th and m-th objects in the images I and Î as x

I n

and x Î m , respectively. We compute the distance matrix

D n,m = (f I n -f Î m ) 2 , n ∈ 1..N, m ∈ 1..M .

For every embedded anchor f I n , n ∈ 1..N , we select a positive embedding f Î m with minimum distance as positive: f Î n+ = argmin(D n,m ). Given a batch of (anchor, positive) pairs {(x i , x

+ i )} N i=1

, the n-pair loss is defined as follows (Sohn, 2016) :

L N -pair {(x i , x + i )} N i=1 ; f = 1 N N i=1 log 1 + i≠j exp(f ⊺ i f + j -f ⊺ i f + i )

The loss learns embeddings that identify ground truth anchor-positive pairs from all other anchornegative pairs in the same batch. It is formulated as a sum of softmax multi-class cross-entropy losses over a batch, encouraging the inner product of each anchor-positive pair (f i , f + i ) to be larger than all anchor-negative pairs (f i , f + j≠i ). The final OCN training objective over an observation sequence is the sum of npairs losses over all pairs of individual frames:

L OCN = L N -pair {(x I n , x Î n+ )} N n=1 ; f + L N -pair {(x Î m , x I m+ )} M m=1 ; f 3.3 ARCHITECTURE

OCN takes a standard ResNet50 architecture until layer global pool and initializes it with ImageNet pre-trained weights. We then add three additional ResNet convolutional layers and a fully connected layer to produce the final embedding. The network is trained with the n-pairs metric learning loss as discussed in Sec. 3.2. Our architecture is depicted in Fig. 1 and Fig. 2 . for comparison purposes all models evaluated in Sec. 5 share the same architecture of a standard ResNet50 model followed by additional layers. While the architectures are shared, the weights are not across models. While the unsupervised model (left) does not require supervision labels, the 'softmax' baseline as well as the supervised evaluations (right) use attributes labels provided with each object.

We evaluate the quality of the embeddings with two types of classifiers: linear and nearest neighbor.

3.4. OBJECT-CENTRIC EMBEDDING SPACE:

By using multiple views of the same scene and by attending to individual objects, our architecture allows us to differentiate subtle variations of object attributes. Observing the same object across different views facilitates learning invariance to scene-specific properties, such as scale, occlusion, lighting, and background, as each frame exhibits variations of these factors. The network solves the metric loss by representing object-centric attributes, such as shape, function, texture, or color, as these are consistent for (anchor, positive)-pairs, and dissimilar for (anchor, negative)-pairs.

3.5. WHY SHOULD THIS WORK?:

One might expect that this approach may only work if it is given a good enough initialization so that matching the same object across multiple frames is more likely than random chance. While ImageNet pretraining certainly helps convergence as shown in Table 1 , it is not a requirement to learn meaningful representations as shown in Sec. 8. When all weights are random and no labels are provided, what can drive the network to consistently converge to meaningful embeddings? We estimate that the co-occurrence of the following hypotheses drives this convergence: (1) objects often remains visually similar to themselves across multiple viewpoints, (2) limiting the possible object matches within a scene increases the likelihood of a positive match, (3) the low-dimensionality of the embedding space forces the model to generalize by sharing abstract features across objects, (4) the smoothness of embeddings learned with metric learning facilitates convergence when supervision signals are weak, and (5) occasional true-positive matches (even by chance) yield more coherent gradients than false-positive matches which produce inconsistent gradients and dissipate as noise, leading over time to an acceleration of consistent gradients and stronger initial supervision signal.

4. DATA COLLECTION, HYPERPARAMETERS, AND TRAINING:

To evaluate the effectiveness of OCN embeddings we generated two datasets of real and synthetic objects. For the (unlabeled) real data we arrange objects in table-top configurations and capture frames from continuous camera trajectories. The (labeled) synthetic data is generated from renderings of 3D objects in a similar configuration. Details about the datasets are reported in Table 4 .

4.1. SYNTHETIC DATA GENERATION:

To generate diverse object configurations we use 12 categories (airplane, car, chair, cup, bottle, bowl, guitars, keyboard, lamp, monitor, radio, vase) from ModelNet (Wu et al., 2015) . The selected categories cover around 8k models of the 12k models available in the entire dataset. ModelNet provides the object models in a 80-20 split for training and testing. We further split the testing data into models for test and validation, resulting in a 80-10-10 split for training, validation, and test. For validation purposes, we manually assign each model labels describing the semantic and functional properties of the object, including the labels 'class', 'has lid', 'has wheels', 'has buttons', 'has flat surface', 'has legs', 'is container', 'is sittable', 'is device'. Fig. 9 shows an example scene.

We randomly define the number of objects (up to 20) in a scene and select half of the objects from two randomly selected categories. The other half is selected from the remaining object categories. We further randomly define the positions of the objects and vary their sizes, both so that they do not intersect. Additionally, each object is assigned one of eight predefined colors. We use this setup to generate 100K scenes for training, and 50K scenes for each, validation and testing. For each scene we generate a number (n = 10) of views and select random combination of two views for detecting objects. In total we produce 400K views (200 pairs) for training and 50K views (25K pairs) for each, validation and testing.

4.2. AUTOMATIC REAL DATA COLLECTION:

Our real object data set consists of 187 unique object instances spread across six categories including 'balls', 'bottles & cans', 'bowls', 'cups & mugs', 'glasses', and 'plates'. Table 5 provides details about the number of objects in each category and how they are split between training, test, and validation. Note that we distinguish between cups & mugs and glasses categories based on whether it contains a handle. Fig. 3 provides a snapshot of our entire object dataset.

We automated the real world data collection through using a mobile robot equipped with an HD camera (Fig. 8 ). At each run, we place about 10 objects on the table and then trigger the capturing process by having the robot rotate around the table by 90 degrees (see Fig. 8 ). In average 130 images are captured at each run. We select random pairs of frames for each trajectory during training of the OCN. We performed 345, 109, and 122 runs of data collection for training, test, and validation dataset, respectively. In total 43084 images were captured for OCN training and 15061 and 16385 were used for test and validation, respectively.

4.3. TRAINING:

An OCN is trained based on two views of the same synthetic or real scene. We randomly pick two frames of a camera trajectory around the scene to ensure the same objects are present; the frames are selected based on their time stamps so that they are as far apart as possible. We set the npairs regularization to λ = 0.002. The distance matrix D n,m (Sec. 3.2) is constructed based on the individually detected objects for each of the two frames. The object detector was not specifically trained on any of our datasets. Furthermore, we only used scenes where at least 5 objects were detected in each frame. Operating on less objects results in a more noisy training signal as the n-pairs loss cannot create enough meaningful (anchor, negative)-pairs for contrasting them with the (anchor, positive)-pair. As the number of detected objects per view varies, we reciprocally use both frames to find anchors and their corresponding positives as discussed in Sec. 3.2. Across our experiments, the OCN training converged after 600k-1.2M iterations.

5. EXPERIMENTAL RESULTS:

To evaluate the effectiveness of an OCN embedding as representation for object attribute disentanglement, we performed experiments on a large-scale synthetic dataset and two robotic tasks of pointing and grasping in a real-world environment. Moreover, the experiments are designed in a way to directly showcase the usefulness of OCN on real robotics applications.

5.1. ATTRIBUTES CLASSIFICATION:

One way to evaluate the quality of unsupervised embeddings is to train attribute classifiers on top of the embedding using labeled data. Note however this may not entirely reflect the quality of an embedding because it is only measuring a discrete and small number of attributes while an embedding may capture more continuous and larger number of abstract concepts.

Classifiers::

We consider two types of classifiers to be applied on top of existing embeddings in this experiment: linear and nearest-neighbor classifiers. The linear classifier consists of a single linear layer going from embedding space to the 1-hot encoding of the target label for each attribute.

It is trained with a range of learning rates and the best model is retained for each attribute. The nearest-neighbor classifier consists of embedding an entire 'training' set, and for each embedding of the evaluation set, assigning to it the labels of the nearest sample from the training set. Nearestneighbor classification is not a perfect approach because it does not necessarily measure generalization as linear classification does and results may vary significantly depending on how many nearest neighbors are available. It is also less subject to data imbalances. We still report this metric to get a sense of its performance because in an unsupervised inference context, the models might be used in a nearest-neighbor fashion (e.g. as in Sec. 5.3).

Baselines: we compare multiple baselines in Table 1 and Table 6 . The 'Softmax' baseline refers to the model described in Fig. 2 , i.e. the exact same architecture as for OCN except that the model is trained with a supervised cross-entropy/softmax loss. The 'ResNet50' baseline refers to using the unmodified outputs of the ResNet50 model (He et al., 2016 (He et al., ) (2048-dimensional vectors) -dimensional vectors) as embeddings and training a nearest-neighbor classifier as defined above. We consider 'Softmax' and 'ResNet50' baselines as the lower and upper error-bounds for standard approaches to a classification task. The 'OCN supervised' baseline refers to the exact same OCN training described in Fig. 2 , except that the positive matches are provided rather than discovered automatically. 'OCN supervised' represents the metric learning upper bound for classification. Finally we indicate as a reference the error rates for random classification.

Results: we quantitatively evaluate our unsupervised models against supervised baselines on the labeled synthetic datasets (train and test) introduced in Sec. 4. Note that there is no overlap in object instances between the training and the evaluation set. The first take-away is that unsupervised performance closely follows its supervised baseline when trained with metric learning. As expected the cross-entropy/softmax approach performs best and establishes the error lower bound while the ResNet50 baseline are upper-bound results. Note that the dataset is heavily imbalanced for the Table 1 : Attributes classification errors: using attribute labels, we train either a linear or nearest-neighbor classifier on top of existing fixed embeddings. The supervised OCN is trained using labeled positive matches, while the unsupervised one decides on positive matches on its own. All models here are initialized and frozen with ImageNet-pretrained weights for the ResNet50 part of the architecture (see Fig. 2 ), while the additional layers above are random and trainable. Attributes are defined in Sec. 4.1. binary attributes reported in Table 1 and Table 6 and require balancing for linear classification. In Fig. 4 and Sec. 9, 11, we show qualitative results of nearest neighbor objects discovered by OCN.

5.2. INSTANCE DETECTION AND TRACKING:

An OCN embedding can be used to match instances of the same object across multiple views and over time. This is illustrated in Fig. 5 , where objects of one view (anchors) are matched against the objects of another view. We can find the nearest neighbors (positives) in the scene through the OCN embedding space as well as the closest matching objects with descending similarity (negatives). We report the quality of finding corresponding objects in Table 2 and differentiate between attribute errors, that indicate a mismatch of specific attributes (e.g. a blue cup is associated with a red cup), and object matching errors, which measure when objects are not of the same instance. An OCN embedding significantly improves detecting object instances across multiple views. 

5.3. ROBOT EXPERIMENTS:

Pointing: We evaluate performance of OCN on a pointing robotic task (Fig. 6 ). The robot has to point to an object that it deems most similar to the object directly in front of him on the small table. The objects on the big table are randomly selected from each of the six object categories (Table 5 ).

We consider two sets of these target objects. The quantitative experiment in Table 3 uses three query objects per category and is ran three times for each combination of query and target objects (3 × 2 × 18 = 108 experiments performed). The full set of experiments for one of the three runs is illustrated in Fig. 15 .

Table 3 quantifies OCN performance of this experiment. We report on errors related to 'class' and 'container' attributes (note that the other ten attributes described in Sec. 4.1 are not relevant to the real object data set). While the trained OCN model is performing well on the most categories, it has particularly some difficulty on the object classes 'cups & mugs' and 'glasses'. These categories are generally mistaken with the category 'bowls'. As a result the network performs much better in the attribute 'container' since all the three categories 'bowls', 'bottles & cans', and 'glasses' refer to the same attribute.

Grasping: We qualitatively evaluate the OCN performance on a grasping task in an environment unseen during training. First, a person holds and shows an object to the robot, then the robot picks up the most similar object from a set of objects on a table (see Fig. 7 ). In this experiment, we focus on evaluating OCN with objects that have either similar shape or color attribute. Using OCN the robot can successfully identify and grasp the object that has the closest color and shape attributes to the query object. Note training data did not contain objects held by hand.

Figure 6 : The robot experiment of pointing to the best match to a query object (placed in front of the robot on the small table). The closest match is selected from two sets of target object list, which are placed on the long table behind the query object. The first and the second row respectively correspond to the experiment for the first and second target object lists. Each column also illustrates the query objects for each object category. Image snapshots with green frame correspond to cases where both the 'class' and 'container' attributes are matched correctly. Image snapshots with blue frame refer to the cases where only 'container' attribute is matched correctly. Images with red frames indicates neither of attributes are matched. 

6. CONCLUSION:

We introduced a novel unsupervised representation learning algorithm that allows us to differentiate object attributes, such as color, shape, and function. An OCN embedding is learned by contrasting the features of objects captured from two frames of single view camera trajectories of table-top indoor environments. We specifically attend to individual objects by detecting object bounding boxes and leverage a metric learning loss to disentangle subtle variations of object attributes. The resulting embedding space allows to organize objects along multiple dimensions and serves as representation for robotic learning. We show that an OCN embedding can be used on real robotic tasks such as grasping and pointing, where it is important to differentiate visual and semantic attributes of individual object instances. Finally, we show that an OCN can be trained efficiently from RGB videos that are automatically obtained from a real robotic agent.

APPENDIX 7 DATASET:

Figure 8 : Consecutive frames captured with our robotic setup. At each run we randomly select 10 objects and place them on the table. Then a robot moves around the table and take snapshots of the table at different angles.

We collect in average 80-120 images per scene. We select pairs of two frames of the captured trajectory and train the OCN on the detected objects. 

10. ADDITIONAL ROBOTIC POINTING RESULTS:

Figure 15 : The robot experiment of pointing to the best match to a query object (placed in front of the robot on the small table). The closest match is selected from two sets of target object list, which are placed on the long table behind the query object. The first and the last three rows respectively correspond to the experiment for the first and second target object lists. Each column also illustrates the query objects for each object category. Image snapshots with green frame correspond to cases where both the 'class' and 'container' attributes are matched correctly. Image snapshots with blue frame refer to the cases where only 'container' attribute is matched correctly. Images with red frames indicates neither of attributes are matched.

Figure 16 : An OCN embedding organizes objects along their visual and semantic features. For example, a red bowl as query object is associated with other similarly colored objects and other containers. The leftmost object (black border) is the query object and its nearest neighbors are listed in descending order. The top row shows renderings of our synthetic dataset, while the bottom row shows real objects. For real objects we removed the same instance manually.



Figures and Tables:

Figure2: Models and baselines: for comparison purposes all models evaluated in Sec. 5 share the same architecture of a standard ResNet50 model followed by additional layers. While the architectures are shared, the weights are not across models. While the unsupervised model (left) does not require supervision labels, the 'softmax' baseline as well as the supervised evaluations (right) use attributes labels provided with each object. We evaluate the quality of the embeddings with two types of classifiers: linear and nearest neighbor.

Figure 3: We use 187 unique object instance in the real world experiments: 110 object for training (left), 43 objects for test (center), and 34 objects for evaluation (right).

Figure 4: An OCN embedding organizes objects along their visual and semantic features. For example, a red bowl as query object is associated with other similarly colored objects and other containers. The leftmost object (black border) is the query object and its nearest neighbors are listed in descending order. The top row shows renderings of our synthetic dataset, while the bottom row shows real objects.

Figure 5: View-to-view object correspondences: the first column shows all objects detected in one frame (anchors). Each object is associated to the objects found in the other view, objects in the second column are the nearest neighbors. The third column shows the distances of all objects, all other objects are shown from left to right in descending order according to their distances to the anchor.

Figure7: Robot experiment of grasping the object that is closest to the query object (held by hand). Images on the left are captured by the robot camera, and the images on the right are the video frames from a third person view camera. The leftmost object (black border) is the query object and its nearest neighbors are listed in descending order. The top row and the bottom row show the robot successfully identifies and grasps the object with similar color and shape attribute respectively.

Figure9: Synthetic data: two frames of a synthetically generated scene of table-top objects (a) and a subset of the detected objects (c). To validate our method against a supervised baseline, we additionally render color masks (b) that allow us to identify objects across the views and to associate them with their semantic attributes after object detection. Note that objects have the same color id across different views. The color id's allow us to supervise the OCN during training.

Figure 11: A result showing the organization of real bowls based on OCN embeddings. The query object (black border, top left) was taken from the validation all others from the training data. As the same object is used in multiple scenes the same object is shown multiple times.

Figure 12: A result showing the organization of bottles from synthetic data based on OCN embeddings. The query object (black border, top left) was taken from the validation all others from the training data.

Figure 13: A result showing the organization of vases from synthetic data based on OCN embeddings. The query object (black border, top left) was taken from the validation all others from the training data.

Figure 14: A result showing the organization of chairs from synthetic data based on OCN embeddings. The query object (black border, top left) was taken from the validation all others from the training data.

Object correspondences errors: attribute error indicates a mismatch of a particular attribute of an object, while an object matching error is measured when the matched objects are not the same instance.

Table content: <table><tr><td>Method</td><td colspan="2">Attribute Error Object Matching Error</td></tr><tr><td>OCN supervised</td><td>4.53%</td><td>16.28%</td></tr><tr><td>OCN unsupervised</td><td>5.27%</td><td>18.15%</td></tr><tr><td>Resnet50 embeddings</td><td>19.27%</td><td>57.04%</td></tr></table>

Quantitative evaluation on the robot pointing experiment. We report on two attribute errors: 'class' and 'container'. See Sec. 5.3 for more information about the experiment.

Table content: <table><tr><td>Attributes</td><td>Balls</td><td>Bottles &amp; Cans</td><td>Bowls</td><td>Cups &amp; Mugs</td><td>Glasses</td><td>Plates</td><td>Total</td></tr><tr><td>Class error</td><td>11.1 ±7.9%</td><td>0.0 ±0.0%</td><td>22.2 ±15.7%</td><td>88.9 ±7.9%</td><td colspan="3">38.9 ±7.9% 5.6 ±7.9% 27.8 ±3.9%</td></tr><tr><td colspan="2">Container error 11.1 ±7.9%</td><td>0 ±0.0%</td><td>16.7 ±13.6%</td><td>16.7 ±0.0%</td><td colspan="3">16.7 ±13.6% 5.6 ±7.9% 11.1 ±2.3%</td></tr></table>

Datasets overview: to train an OCN we use real and synthetic data.

Table content: <table><tr><td>Synthetic</td><td>12</td><td>4k</td><td>250k</td><td>2</td><td>250K</td></tr><tr><td>Real</td><td>6</td><td>187</td><td>576</td><td>115-230</td><td>400K</td></tr></table>

Real object dataset: we use 187 unique object instances spread across six categories.

Table content: <table><tr><td/><td colspan="6">Balls Bottles &amp; Cans Bowls Cups &amp; Mugs Glasses Plates</td></tr><tr><td>Training</td><td>14</td><td>13</td><td>19</td><td>19</td><td>22</td><td>23</td></tr><tr><td>Validation</td><td>5</td><td>4</td><td>8</td><td>6</td><td>5</td><td>6</td></tr><tr><td>Test</td><td>6</td><td>6</td><td>10</td><td>6</td><td>6</td><td>9</td></tr><tr><td>Total</td><td>25</td><td>23</td><td>37</td><td>31</td><td>33</td><td>38</td></tr></table>

Peer review (iclr2019):
 In this paper, an unsupervised representation learning method for visual inputs is proposed. The proposed method incorporates a metric learning approach that pulls nearest neighbor pairs of image patches close to each other in the embedding space while pushing apart other pairs. The train and test scenarios are captured based on a table top scenario with multiple objects of different colors such as cups and bowls. Standard datasets are not used for benchmarking. Pros: - Unsupervised feature learning is an active and interesting area of research. - The proposed method and network architecture is simple. Cons: -The paper does not present technical novelty. The proposed network architecture is a ResNet50 that uses the object proposals obtained by Faster RCNN (Ren et al. 2015) and incorporates the n-pair loss function proposed by Sohn 2016. -Technical details are missing. >When using object proposals obtained by Faster RCNN (Ren et al. 2015) to generate pairs, do you use all of the object proposals or do you select a subset of them by some threshold? >How is the robotic control performed in the robot grasping and pointing tasks? It seems that grasping and pointing tasks are not learned and are based on conventional motion planning and state estimation. These details are not included in the manuscript. >Section 4.2 mentions that a mobile robot was needed for collecting the data. What kind of mobile robot was used? Why was a mobile robot needed? There is no details about the robot platform in the manuscript and no visualization is provided for the robot setup. >Why is a ResNet pretrained with ImageNet is used throughout the experiments (and not pretrained with COCO?) while object proposals are obtained by Faster RCNN which is pretrained by COCO. How would that affect the results? -The paper uses imprecise, confusing and sometimes incorrect phrases and wordings throughout the draft. Here are a few examples: > It is unclear what it is exactly meant by “object correspondences” that naturally emerge (e.g. in the last four lines of abstract). Would a correct object correspondence refer to “similar instances”, “similar colors”, “similar object categories”, “similar object poses”, “similar functionality”? For example, the first column of Fig. 1 shows an example of two cups with “similar green color”, “similar tall shapes” and “similar functionality” and “similar background” that are considered as “negative pairs (with red arrows)” while in the same Fig 1. Two cups with drastically different appearances one of which is “yellow with handle” (second column) and the another is “green without handle” (third column) are considered to be positive pairs (blue arrows). Similar confusing examples and arguments can repeatedly be found in the experiments and embedding visualizations: Fig. 4, Fig.5, Fig. 10- Fig. 15. > Throughout the draft and in Figures (e.g. Fig. 1) it is emphasized the the data collection is done by “robot” or it is “robotic”. Why was a robot needed to capture images or videos of around a table? A hand held moving camera could also be simply used. Capturing images and videos around a space is a very well-known and simple practice done in many previous works in computer vision and machine learning. It does not need a robot and it is not robotic. > First sentence of the second paragraph of the introduction is not a technically correct statement based on the well-known computer vision literature. Specifically, “class” is not considered as an “attribute”. What does it mean to disentangle “perceptual” and “semantic” object attributes? This is a very vague statement. “color” is neither a “semantic” attribute nor a “perceptual” attribute. Also, in section 3.2 and Fig. 2, it is confusing to consider “class” label as an attribute. Please refer to the well-known prior work of “Farhadi A, Endres I, Hoiem D, Forsyth D. Describing objects by their attributes. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on 2009 Jun 20 (pp. 1778-1785). IEEE.” for a clear and well-known explanation of object attributes vs categories. If your definition of “attributes” is technically different than that of this aforementioned prior work, it should be clarified to avoid confusion. -Unconvincing usefulness for robotics applications: The paper explains that the proposed method is effective and useful for robotic tasks however tis claim is not backed up with convincing robotic tasks or scenarios. Also, the performance in the robotic grasping and pointing tasks are not compared with any baselines. > While usefulness of the method for improving robotic grasping task is listed in the contributions (number 3 in the last paragraph of Section 1) there are only two qualitative grasping scenarios shown in the paper. It is not explained how the robot control is done. In both grasping scenarios the target object is the one in the middle. It seems that the grasping control policies are not learned and are motion planned or are a scripted policy and only the visual matching of the target object with the objects of the scene are done with the proposed method. For evaluating such scenario no robot is needed and only similarity score could be reported. It would had been interesting to see how representation learning can be seamlessly incorporated for robotic grasping which involves control as a tangible contribution however, the presented scenario does is not contributing to that problem and is only doing visual matching. No baseline comparison is provided here. > The details of the robot control for the robot pointing scenario is also not provided. The objects are places on two rows in all scenarios and it looks like the the actual pointing is done by motion planning after the visual matching step done by the proposed method. The presented method in this paper tries to find the closest visual match to the target object amongst the objects on the table and does not integrate it with any policy learning for improving the “robotic pointing task” so there is no robotic contribution here. No baseline is provided performance comparison in this robotic task as well. - The experimental results are weak and unconvincing: > In the experiments corresponding to Table 6 and Table 1, what is the performance of ResNet 50 embedding (linear)? Can you provide these results? > Table 6 shows that the performance gain of the proposed method compared to the supervised and unsupervised baselines is marginal. > What is the performance of a baseline network with similar architecture that uses “contrastive loss” (based on object pairs with similar attributes). This baseline is missed. > Qualitatively, the visualizations in Fig. 1, Fig. 4-5, Fig. 10-15 show that OCN embedding is noisy. In all these figures, there are many less similar instances that are retrieved on the top and many similar instances that are ranked down in the list. > The paper refers to ad-hoc experimental protocols which makes the evaluations unclear and invalid: what is meant to report “class” and “container” attributes in Table 3 and section 5.3? Why are “bottles and cans” are considered in a same group while there were referred to as different objects in all previous explanations of the object types and categories used in the training and testing? What is the difference between “cups and Mugs” and “glasses” that are separated? How are “Balls” , “Bowls”, “Plates”, etc listed as *attributes*? Aren’t these previously referred to as object categories? > The paper has some inconsistent experimental setups. Why in Fig 16, the same instances of were removed for the real objects and not for the synthetic objects? -The presentation of the paper can be improved. Here are a few examples: >Grammatical or unclear wordings and typos (e.g. Caption of Fig. 2 “attracting embedding nearest neighbors”; is not a readable. Repetitive words, Last line of section 3. , etc) >In Fig. 10-15, using t-SNE (Maaten LV, Hinton G. Visualizing data using t-SNE. Journal of machine learning research. 2008;9(Nov):2579-605.) instead of a sorted list provides much more expressive visualization of the effectiveness of the approach. It is highly recommended that t-SNE be used instead of sorted lists if image patches.