Tailoring Visual Object Representations to Human Requirements: A Case Study with a Recycling Robot
Authors: Debasmita Ghose
Michal Adam Lewkowicz
Kaleb Gezahegn
Julian Lee
Timothy Adamson
Marynel Vazquez
Brian Scassellati
Venue: CoRL 2022 Poster (2022)
Keywords: Representation Learning
Contrastive Learning
Human-in-the-loop Robot Learning

Abstract: Robots are well-suited to alleviate the burden of repetitive and tedious manipulation tasks. In many applications though, a robot may be asked to interact with a wide variety of objects, making it hard or even impossible to pre-program visual object classifiers suitable for the task of interest. In this work, we study the problem of learning a classifier for visual objects based on a few examples provided by humans. We frame this problem from the perspective of learning a suitable visual object representation that allows us to distinguish the desired object category from others. Our proposed approach integrates human supervision into the representation learning process by combining contrastive learning with an additional loss function that brings the representations of human examples close to each other in the latent space. Our experiments show that our proposed method performs better than self-supervised and fully supervised learning methods in offline evaluations and can also be used in real-time by a robot in a simplified recycling domain, where recycling streams contain a variety of objects.

Tailoring Visual Object Representations to Human Requirements: A Case Study with a Recycling Robot
Debasmita Ghose, Michal Adam Lewkowicz, Kaleb Gezahegn, Julian Lee," Timothy Adamson," Marynel Vázquez, Brian Scassellati Yale University (first name). {last -. name name) @yyale
Abstract: Robots are well-suited to alleviate the burden of repetitive and tedious manipulation tasks. In many applications, though, robot may be asked to interact with a wide variety of objects, making it hard or even impossible to pre- proogrm visual object classifiers suitable for the task of interest. In this work, we study the problem of learning a classifier for visual objects based on a few examples provided by humans. We frame this problem from the perspective of learning a suitable visual object representation that allows us to distinguish the desired ob- ject category from others. Our proposed approach integrates human supervision into the representation learning process by combining contrastive learning with additional loss function that brings the representations of human examples close each other in the latent space. Our experiments show that our proposed method performs better than supersupee and fully supervised learning methods in of- fline evaluations and can also be used in real-time by a robot in a simplified recy- cling domain, where recycling streams contain a variety of objects.
Keywords: Contrastive Representation Learning, Human Centric Robot Learning
1 Introduction
Robots are well-suited to alleviate the burden of repetitive and tedious manipulation tasks in homes and in the public and private sectors. For example, robots might help humans with housekeeping, jamitorial and custodial work, or sorting recyclables. In many of these applications, a robot may be asked to interact with a wide variety of objects, making it hard or even impossible to pre- rrepprgram visual object classifiers suitable for the task of interest. For example, imagine a home robot meant to help a person clean her room. One might think of defining object categories for this task, such as "clothes", "shoes", "books", etc. But what if the human wants the robot to pick up only "folded clothes" from the floor? These kinds of tasks require learning from the human what object category is relevant. Since these categories can be defined arbitrarily, the robot needs to learn the object properties that distinguish the seleetee objects from others. Importantly, another challenge in learning object category from a human is that the robot cannot assume the objects not selected by the human do not belong in the category of interest. That is because humans are unwilling to give many examples and may only provide examples of objects they are interested in.
In this work, we study the problem of learning classifier for visual objects based on (a few) ex- amples provided by humans. We frame this problem from the perspective of learning suitable visual object representation that allows the robot to distinguish the desired object category from others. This perspective is motivated by recent work in self-supervised learning, which has led to significant advances in learning useful representations from high-dimensional data [1]. partic- ular, contrastive learning [2] learns representations by pulling similar samples close to each other while pushing dissimilar ones far apart in the latent space. Recent contrastive learning methods have been successful in learning strong representations for various downstream tasks in computer vision, such as image classification, semantic segmentation, and object detection [1]. The challenge with applying existing contrastive learning techniques to the problem of learning representation that distinguishes an arbitrary object category based on human examples is that contrastive learning
Authors contributed equally
Code and data available at http://www...ogoooooooooooooooo.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

will capture properties of the data, but those properties may not necessarily lead to good object classifier according human requirements.
We propose a new approach that integrates human supervision into the representation learning pro- cess to compute visual object representations in accordance with human requirements for the task of interest. Our approach combines an existing contrastive learning method that maximizes the agree- ment between different instances in the same cluster with our proposed loss function that brings the representations of objects selected by a human close to each other in the latent space. This combination results in representations that encode the human desired object characteristics.
We evaluate our approach to incorporate human supervision into the representation learning process in simplified recycling setup derived from standard Material Recovery Facility (MRF). We focus on this challenging application domain for three reasons. First, recycling streams can contain a wide variety of diverse objects, ranging in size, weight, color, cleanliness, and form [3]. Second, constraints on what needs to be recycled can vary dynamically because the requirements of the MRF's customers may vary [4], Third, because of the many objects that quickly move by, a human demonstrator cannot pick out all the items that need to be removed from the stream. This results in limited positive examples and no negative examples for the robot.
To conduct offline and online evaluations, we set up conveyor belt that humans and robots can work on simultancously and feeder system to provide a steady stream of recyclables. We demonstrate that our method outperforms a fulll--upervised learning method and some self-supervised learning techniques on offline dataset. Finally, we deploy our proposed learning method in real-time on a robot working alongside a human in the simplified recycling setup. Our results show how incremental supervision from the human helps the robot learn visual object representations tailored to the human's 's requirements.
2 Related Work
Self-Supervised Learning: Self-supervised learning obtains supervisory signals from the data by leveraging its underlying structure [1. 5]. One broad category of techniques in self-supervised learn- ing is contrastive learning, which has been used in wide range of computer vision applications [6, 7, 8]. Contrastive learning methods [2] typically Iearn a latent space by pushing similar samples (c.g., two adjacent frames of video [9]) close each other while pulling dissimilar data samples (c.g., frames from two different videos) apart in the latent space. More recently, instance discrimi- nation [10, 11, 12, 13], variant of contrastive learning, has shown remarkable success in improving performance on a variety of downstream tasks. To learn latent-space representations, most instance discrimination techniques
two augmented views of the same image as positive data pair in con- junction with a variety of contrastive learning techniques, such as the contrastive loss [10, 11, 14), triplet loss [15, 16), momentum encoding [17, 18, 19], or online clustering [20].
Recently, there have been some efforts to learn more robust latent-space representations by lever- aging properties of non-trivial positive pairs that correlated in the latent space. For example, these non-trivial positive pairs can either be selected from a support set [12, 21] or from associating multiple data instances with clusters [6. 13, 22, 23]. In this work, we leverage a clustering technique called Contrastive Clustering [13] to enable robots to learn visual object representations. Contrastive Clustering extends instance-level discrimination to distinguish between objects belonging to differ- ent clusters while maximizing agreement between objects belonging to a cluster.
Adding Supervision to Contrastive Learning: Prior work has guided representation learning pro- cesses by leveraging labeled data [24, 25). Khosla et al. [25] proposed a method to train a contrastive learner in fully-supervised fashion to improve performance on multiple downstream tasks. How- ever, since their method is label intensive, more recent work has explored using cemi-supervised [26, 27] and weakly supervised [28] learning techniques to compute strong representations. Wilber et al. [29] introduced an algorithm called SNaCK that proposes adding a loss function to visual rep- resentation learning that reflects a human's 's preferences. However, SNaCK requires human to pro- vide a large number of examples of objects that the human thinks are similar and dissimilar, which would be infeasible in real human-robo interactions. To the best of our knowledge, method is the first to show how human feedback can be leveraged to learn visual object representations tailored to dynamic human-defined requirements with limited examples of a single category.
Human Centric Robot Learning: There are many ways in which a robot can learn from a human. The three most common types of demonstrations used for robot learning are kinesthetic teaching,

---

INSTANCE PRODECIION HEAD
WEICHT SHARING
x
/1-
M/
WEIGHT SHARING
CLUSTER PROTECTION HEAD
51
ENCODER
Figure 1: Our Approach. Data pairs constructed using randomly applied data augmentations. The extracted features are fed into three jointly trained heads. The Instance Projection Head projects the features into space where each row denotes an augmentation of image and uses the instance loss to minimize the distance between two augmentations of the same image. The Cluster Projection Head projects feature vectors into a space where each column denotes each instance's cluster assign- ments, and the cluster loss minimizes the distance between instances belonging to a cluster. Finally, the Human- Supervise Head minimizes the distance between each selann-elleced example.
teleoperation, and passive observation [30]. Our work uses passive observations because of the ease at which they allow the human to teach the robot and the hardware-agnosti nature of this demonstration type. Much of the recent work in the field of learning from passive observations uses reinforcement learning train a robot. For example, the work of Mukherjee et al. [31] uses learned goal proximity function as a dense reward for policy training, and Karnan et al. [32] propose to learn navigation policies from single video demonstration. Our work complements these efforts by proposing a novel approach for learning visual object representations that a robot can use for manipulation tasks, like sorting recycling streams.
Typically within the area of human preference learning, a robot learns the preferences of its human collaborators to perform a task through various reward shaping techniques [33, 34j. Our work is related to this thread of research because we want a robot to learn the preferences of its human collaborator for sorting task. However, want the robot to use passive observation rather than more traditional active querying techniques [35, 36, 37] because, in the recycling domain, humans are already busy picking objects from recycling streams. Preference learning in our work then entails having a robot learn the visual characteristics of the categories of objects that the human collaborator is interested in. We propose approach this problem from the perspective of self- supervised representation learning and incorporate human supervision a novel manner to ensure that the learned representation aligns well with the human's preferences.
3 Learning Visual Object Representations Tailored to Human Requirements
We propose a method to guide the representation learning process of a contrastive learner using human supervision, as shown in Fig. 1. To achieve this goal, we optimize feature extractor jointly with three "heads." These heads maximize the agreement between 1) multiple views of the same image, 2) multiple instances of the same cluster, and 3) human selected examples.
Let X' be the set of all objects relevant to a manipulation task and C be the total number of human- defined categories such that given category c [1,... C). As a property of our problem, a human selects a set of items H such that H C X' and all objects in H belong to a given human-defined category c. The paragraphs below use this notation to describe the components of our approach.
The base encoder is a feature extraction network that obtains representations from two stochastically augmented views of the same image [10]. For given image æ, the base encoder outputs a feature vector y, such that y = fx) Therefore, two stochastic augmentations of an image denoted by x; and zy, their feature representations
Base Encoder:
be denoted by y: f(x.) and y; = f(x,). Additionally, if 2₁ is an image in H. then its feature representation is 3/,. = f(xx).
We feed the image representations output by the base encoder to the three heads described next.
Instance Projection Head: The Instance Projection Head Multi-Layer Perceptron (MIP) with one hidden layer to map the features of two augmented views of the same image to a latent space
3

---

[10]. For example, for the representation of an image's first augmentation y:, its representation after passing through the Instance Projection Head is 2; = h(y:) (Similarly, for the second augmentation, 2; ) As in Chen et al. [10], we apply the NTT-Xent loss on the representations from the Instance Projection Head. Therefore, for positive pair of examples (i,j) constructed from two augmented views of the same image, the instance loss w.r.t. the first augmentation is formulated as:
- log exp (sim (Z,, 2,) (2,))/To)) /Fns)
p ins
(1)
[[yli]]] exp (sim (z,, Zi) / T(ns)
where N is the number of all positive pairs in mini-batch, sim(u, v) similarity between the feature vectors, and [ * x] E (0, 1}, which evaluates to I iff h + i. We use Ei (1... + ( to compute the instance loss over all positive pairs in a mini-batch.
"² is the cosine |
Lins
2N
2. Cluster Projection Head: The Cluster Projection Head [13] learns representations by maximiz- ing agreement between two representations (under different augmentations) belonging to a given cluster. This head is an MLP with one hidden layer followed by softmax operation. It projects data instance into a latent space whose dimensionality equals the total number of pre-defined clus- ters (K)) which should be approximately equal to C. Intuitively, the cluster projection head tries to partition the embedding space into a pre-specified number of clusters based on the inter-instance similarity between object features. For an encoder representation of the first augmentation of an image yi. the latent representation after passing through the cluster projection head is C; = g (yy) (Similarly, for second augmentation, ej- g(y;)). Let C, RN be the output of the Cluster Projection Head for mini-batch of N images under the first augmentation i. Then, the nhth element in the output CC can be interpreted as its probability of belonging to the htth cluster, where k E [1 can be denoted by CPc Thus, ē E R- can be interpreted as the column vector of C; under the first augmentation. The cluster loss can be formulated as:
exp (sim (č;, C5) /Tatu)
nd - log
(2)
Σ k=1 [[zti exp (sim (č., Gx) /Taa)
where Telu is the cluster-level temperature parameter. Finally, the cluster loss computed by traversing I Σx - (P-) + ( - HY)), where H(Y)= = (P(E
all K. clusters is Let = 2K P(āj)log
is the entropy of cluster assignment probabilities and P(ă) = Σ m- || The entropy term helps avoid the trivial solution where most instances get assigned to the same cluster as shown by [13, 23].
CT
Our proposed nupan-supervised head aims to guide the representa- tion learning process towards human requirements. Intuitively, it tries to force the representations of the objects selected by the human close to each other in the latent space to serve as a mechanism to help inform the formation of clusters according to the properties of objects important to the human. This loss is applied only to the representations
3. Human-SSpervised Head:
objects selected by the human Yh Formally, let YA E R xx D be the representation learned by the base encoder across all objects selected by the human (H C X)) and D be the latent dimension of the base encoder. Additionally, let (4) E R be the mean of the representations of all the objects selected by the human, which is considered the cluster-centroid of the human pool.
As new objects are selected by the human, we pass their image (æx)) through the base encoder to get their feature representation (y), = fxx)). If previously there were s numan-selected objects and the human selects É new objects, we re-calculate the mean of the cluster (2/)) to include the features of the new objects by 1n// = p.. 2i The human-supervised loss then minimizes the distance between the new centroid of the human-selected objects and every object selected by the human:
s-t
Lhuman = -dist(ph. kn)
Theoretically, any distance metric (disl) can be used to maximize agreement between the centroid of the human-selected pool and all seleeee objects. However, we empirically found cosine- similarity to work better compared to LI and L2 distances since it is bounded between -1 and 1.
Objective Function: The Instance Projection Head, Cluster Projection Head, and Human Super- vised Head are combined in the final objective function that we use to train our model:
L = NinLinee + Aetu + Ahuman Chamon
(4)
where Aina > 0, Aetu > 0. and Akuman > 0 are hyperparameters.

---

4
Experimental Setup
HOFFER
We use the Stretch RE-1 robot [38] with a static push plate mounted on a telescopic arm to extract items from moving conveyor belt (as shown in the sup- plementary video). The robot can extract items by either extending its telescopic arm to push objects off the conveyor belt or pulling objects towards it- self. As shown in Fig. 2, the setup consists of a 60-gallon hopper, a small feeder belt, and the main conveyor belt (I0 feet long and 1.5 feet wide). We position three cameras above the belt to monitor the recycling stream.
RECYCLE
TIASH
RSCYCLE
Figure 2: Experimental Setup. The human and the robot work together on the recycling conveyor belt to sort the recyclables.
We use over 500 unique recyclable items of various sizes, shapes, colors, and forms in our ex- periments. The recyclables belong to ten categories: crushed metal cans, un-crushed metal cans, crushed plastic bottles, un-crushed plastic bottles, un-crushed colored plastic bottles, brown card- board boxes, coated cardboard boxes, cardboard trays, malf-gallon milk jugs, and one-gallon milk jugs. For randomizing the stream, the hopper dispenses a random selection of recyclables onto the feeder belt, which then transports the items to the main conveyor belt. A human can pick up some items of given category by standing next to the conveyor belt. The robot is expected to sort items similar to the human-selected objects and extract them from the stream by pushing or pulling them off the conveyor belt.
5 Offline Evaluation
Data Collection: For evaluating the system offline, we created a dataset by having selection of recyclables pass through the setup described in Section 4. We used YOLACT [39], real-time in- stance segmentation model with a 10s [40] backbone, to predict rotated bounding box over each item on the conveyor belt. We trained this instance segmentation model with 411 conveyor-bell images containing 4419 instances of recyclables, which were manually annotated with instance mask. Using an 80% train, 20% validation split, we obtained an Fl-score of 0.89 after training the YOLACT instance segmentation model for 100 epochs using the Torch. Each predicted mask was converted to a rotated bounding box whose crops were extracted for evaluation. Finally, we created dataset containing 1502 variable-sized crops of recyclables, where each crop belongs to one of the ten categories described in Section 4.
[41] library in Py-
For training, we select three random sets of 40 examples from each of our 10 categories to serve as the human-selected pool for a given category (resulting in 30 human pools in total). We use each human-selected pool to train our proposed model. During inference, we compute the pairwise cosine similarity of every object not selected by the human with every object in given human-sclected pool. We then compute the average cosine similarity of the top-5 most similar objects and assign it as the similarity score of the candidate object. We compute an Fl-score per category by thresholding each similarity score between [0.1, 0.95] and assigning candidate object to a given category if the similarity score is above the threshold [42, 433. We report F1-scores for the best threshold averaged over three human-selected pools per category.
Evaluation Protocol:
Quantitative Results: Experiment in Fig. 3 shows the performance of our method on the ten categories specified in Section 4 using the evaluation protocol explained above. We compared our method with a supervised learning method comprised of only our human supervised head in iso- lation. This head effectively learns to classify instances when human-selecter data from only one positive class is available during training [44]. Experiment in Fig. 3 shows the Fl-score of this supervised learning method to be significantly lower than our method.
We conduct an ablation study of our method evaluate how each component of the loss in eq. (4) contributes to the overall performance. Experiment I in Fig. 3 shows the F1 -c-rsc of only the human- -unerviee head across all categories. The suman-supervised head performs poorly on its own likely because the data is heavily imbalanced between human- selected and non-human- selected objects. Experiment in Fig. 3 shows the F1-scores of the instance projection head, which on its own is identical to SimCLR [10]. The high Fl-scores across three human pools per category show that this head learns a robust visual representation for all categories. Experiment in Fig. shows that the cluster projection head performs poorer than the instance projection head.
Ablation Study:
5

---

Bottles
Hal Callon
inact Cars
Contre Carcboard
1.0
0.8
Score
12
0.0
Experiment
Human Supervision
Supervision Supervision
Contrastive Clustering Cluster Loas! [13)
Loss Only (20) Cluster Losa Only
Supervised Contrastive Clustering
Figure 3: Results of our Offline Evaluation: Comparison with Baselines and Ablation Study aggregated over human-sclected pools per category
Evalluation Set
Evaluation
Evaluation
8
10
$
10
10
Training Se5
Training
10
a) Impect of Human Supervision Instance Loss
Impect of Human Supervision on Cluster Loss
c)
Impact Human Supervision on Constrastive Clustering
<-10%
1-10%, -5%e
(65%
0%
10%
15% 10%)
>10%
juge
Thay
Figure 4: Impact of augmenting human supervision to contrastive learning: Difference in performance between self-supervised learning augmented with human supervision and the self- supervised learning model trained independently for instance loss, cluster loss, and contrastive clus- tering, respectively across categories. Specifically, w.r.t. the experiment numbers in Figure 3, a) represents Experiment 4 Experiment 2, b) represents Experiment 5 Experiment c) represents Experiment 7 Experiment
To assess the impact of human supervision each of our loss components, we jointly trained each of our self-supervised loss components (instance loss, cluster loss, and contrastive clustering) with our proposed loss that adds human supervision (eq. 3). For these experiments, models across three human pools per category (resulting in 30 human pools total). As explained in Section 5, each human pool consists of 40 randomly selected images belonging Fig. 4a), 4b) and 4c) summarize the results. Each column in these figures denotes the human pool the method trained with, and each row denotes the human pool the method was evaluated Each cell corresponds to the difference in F1-scores between the respective self supe-vuue method trained with human supervision and the sele-supervised learning method independently.
trained and evaluated
given category.
Fig. 4a) and Experiment the performance of the instance projection head across all categories. We attribute this result to conflicts between the two objectives. The instance loss maximizes the agreement between two augmentations of the same image while optimizing for pushing other instances away. In contrast, the human supervision head optimizes for pulling human- human-providee providerovddd instances together. Fig. 4b) and Experiment 5 in Fig. 3 show that human supervision helps improve the performance of the model trained with the cluster loss for most categories. In this case, the objective functions align with human supervision supporting the formation of meaningful clusters. It is worth noting that when the human pools used for training and evaluation are the same (diagonal elements in Fig. 4b)), human supervision often helps improve the performance of cluster loss.
Fig. show that human-supervised head (eq. 3) does not improve
Finally, we show that our method slightly outperforms the self-supervised contrastive clustering method [13] (Experiment 6 in Fig. 3), showing an average improvement of 4.93% over all cate-

---

gories. Additionally, Fig, 4c) shows the performance of our method compared to contrastive clus- tering, which combines the instance and cluster loss. Our method performs well across categories, even when the human pools for training and evaluation belong to different categories (off- (off-diagonal elements in Fig. 4c)). These results show that combining the losses that maximize agreement be- tween both different views of the same instance (instance loss) and different instances belonging to the same cluster (cluster loss) benefit from even limited human supervision. Our human supervision loss facilitates learning representation with more homogeneous clusters, as seen in the I-SNE plot in the supplementary material.
Realsense Cameras
6 A Case Study with a Recycling Robot
This section describes in more detail the system we used to demonstrate our approach in a recyclable sorting task, as shown
BIll Tmaagee
NGB Intageo
in Fig. 5. The robot learns the characteristics of objects that human is sorting, working alongside this person in real-time. Afterward, we detail our experimental design, evaluation protocol, and results.
Hand Landmark Detection
Real-time
Object Detection
Beanding Bayed
Hand Landmarka
Bounding Busss
6.1 System Architecture
Pick Event Detection
Hit Planner
We predict instance segmentation masks over each object the conveyor belt using the overhead cam- eras' video streams as described in Section 5. We convert these masks tightly-fitteeh rotated bounding boxes.
1. Object Detection:
Postibve Esampiis
Similarites
Push / Pull Commrnnd
Samilarity Estimation (Our Method)
Stretch Robot
This module determines which objects the human is picking in real-time. This is done by iden- tifying hand movements associated with an object being picked and then obtaining un-occluded image of the object from pre- vious frames in the video. The system uses the MediaPipe real- time hand tracking API and ResNet- [40] image classifier to identify between three events: picking event, no picking, and no hand present over the conveyor belt. To determine which item the human is picking in real-time, the model compares the placement of the person's 's hand with the position of objects on the belt in few prior frames.
2. Detection of Picked Objects:
Controller
Figure 5: System Architecture
3. Similarity Estimation (Our Method): Our approach allows the robot to learn the features of the objects picked by the person and to remove similar objects out of the items remaining on the conveyor belt. Our human- supervise contrastive clustering model is trained in real-time for 5 epochs on all the objects on the conveyor belt and the numan-selected objects by optimizing for eq. 4. The robot then uses the trained learner to identify objects similar the objects selected by the human. We run inference on every remaining object on the conveyor belt that was not picked by the human and calculate the pairwise cosine similarity between a given object and every object in the human-selected pool. We assign the average pairwise cosine similarity with the positive dataset to be the similarity score of the given object.
4. Hit Planner: A hit score is calculated for each object on the belt based on the similarity score of that object and the number of other objects that would likely be removed unintentionally should the robot try to remove the desired object from the conveyor belt. These so-called "casualties" are consequence of our robot's hardware only being able push and pull objects off the belt rather than performing more complicated manipulation actions. The object with the highest hit score is targeted by the robot and removed from the belt.
6.2 Experimental Protocol and Results
One of the researchers removed recyclables of a given category from the moving conveyor belt. The robot system recorded the items that the human picked from overhead camera. Once a certain number of items were removed from the stream, the robot trained on the data it had collected. The robot then began remove items it deemed were similar to human-selected objects from the belt. The human and robot worked together on the same belt, and the robot continued to update its model as it watched the human pick more objects. When the human picked 30 objects from the recycling stream, the robot's 's model was cleared, and the researcher trained the robot's model for a different category of objects.
7

---

We perform inference on images of all objects on the conveyor belt (in- cluding the objects selected by the human using our trained model) to obtain their representation in the em- bedding space. We then compute the pairwise cosine similarity between each human-selected object and ev- ery remaining object on the con- veyor belt, which are "candidate ob- jects.' For each candidate object, we average the cosine similarities be- tween itself and the top-5 most simi- lar human-selected objects and assign
0.85
Brown Cardboard
Coated Cardbbard
Colored Bottles
F1 5cooro
0.80
Crushed Bottles
Crushed Cans
0.75
Half Gallon
Intact Bott es
0.70
Intact Cans
One Gallon
0.65
Trays
Number of Items Selected by the Human
Figure 6: Model performance in real-time. the average the similarity score of that candidate object. These similarity scores for the candi- date objects on the conveyor belt are leveraged by the hit-planner module to decide what objects are possible to remove with minimal casualties. We manually labeled the video recordings of the robot removing items of a given category to count the number of items correctly/inoorrectyy removed or missed calculate the Fl-score of the robot.
For sample efficiency in real-time, we initialize the robot's model with a pre-trained contrastive clustering model trained our offline dataset (described in Section 5) with no human supervision. Fig. 6 shows the F1-score of the robot system across all ten categories after the researcher picks 30 items in increments of 10. On average, there were 90 + 16 items the belt for each round of evaluation (when the human picked ten items). For every 10 human human-provided examples, the robot's Fl-score improved by points on average across categories.
7 Limitations and Future Work
Our evaluation assumes that the human selects objects that belong to the same category. Future experiments will include inter-category object selection and wider participant pool outside the research team. Hardware constraints also impacted our evaluation. The robot's limited range of mo- tion combined with its relatively slow speed resulted in missing target items and accidental removals. Also, our system assumed the belt's velocity was the velocity of every object. When this assump- tion breaks, the robot miscalculates objects' distances. These hardware constraints accounted for 36% of all missed items and 32% of all accidental removals across all rounds of evaluation. In the future, the system would benefit from using object tracking algorithms to help the robot identify the movement patterns of each object to make object removal more accurate. Additionally, we evaluated our approach on a relatively small diversity of objects compared what is expected in a Materi- als Recovery Facility (MRF) on relatively sparse and slow belt. Future work should evaluate the performance on more diverse stream of objects on a fast- fast-moving belt.
Aggregating human feedback at scale is an interesting extension of our work. For example, we imagine situation where humans observe objects near robot through a camera feed and provide supervision for which objects to pick remotely. While we don't think that this approach currently makes sense for the recycling domain given economic constraints, future work could gather labels in this manner to accelerate robot learning with our approach. Finally, even though our work focuses on recycling, our approach could also be applied in industrial quality control or the fruit sorting industry. Broadly speaking, our approach could potentially benefit applications where humans and robots work jointly and systematically to manipulate objects. However, since we did not run explicit tests on any of these application domains, we tried not to speculate too broadly in the paper.
8 Conclusion
We proposed novel approach to incorporate human supervision into a contrastive clustering model to better align robot's 's visual object representation with humans" requirements. Our method demon- strated improved performance over a supervised learning method and self-supervised learning tech- niques in simplified recycling setup. In our case study, we had a human and a robot work alongside each other to sort out recyclables, with the robot learning from the human. We showed that the robot learns representations well suited to the human's 's requirements with few examples.
8

---

Acknowledgments
This work was funded by the National Science Foundation (NSF) under grants No. 1955653, 1928448, 2106690, and 1813651. We would like to thank Jake Brawer, Shasvat Desai, Deep Chakraborty, Nicole Salomons, and Nathan Tsoi, who helped improve the paper through their feed- back. We would also like to thank Cameron Adams and David Stanley for their help with the supplemental video.
References
[1] A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1::2, 2020.
[2] S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Puttern Recognition (CVPR'05), volume I, pages 539-546. IEEE, 2005.
[3] F. Tanguay-Rioux, M. Héroux, and R. Legros. Physical properties of recyclable materials and implications for resource recovery. Waste managemens, 136:195-203, 2021.
[4] H. Damgacioglu, M. Hornilla, O. Bafail, and N. Celik. Recovering value from single stream material recovery facilities-an outbound contamination analysis in florida. Waste Management, 102:804-814, 2020.
[5] Y. Bastanlar and S. Orhan. Self-supervised contrastive representation learning in computer vision. 2022.
[6] T. Panambur, D. Chakraborty, M. Meyer, R. Milliken, E. Learned- Millerr and M. Parente. Self- supervised learning to guide scientifically relevant categorization martian terrain images. arXiv preprint arXiv:2204 09854, 2022.
[7] V. Sharma, M. Tapaswi, M. S. Sarfraz, and R. Stiefelhagen. Clustering based contrastive learning for improving face representations. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pages 109-116. IEEE, 2020.
[8] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020.
[9] H. Wu and X. Wang. Contrastive learning of image representations with cross-video cycle- consistency. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10149, October 2021.
[10] T. Chen, S. Korbblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[11] X. Chen and K. He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[12] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. With liule help from my friends: Nearest-neighbor contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9588-9597, 2021.
[13] Y. Li, P. Hu, Z. Liu, D. Peng, J. T. Zhou, and X. Peng. Contrastive clustering. In 2021 AAAI Conference on Artificial Intelligence (AAAI), 2021.
[14] J. Zbontar, L. Jing. I. Misra, LeCun, and S. Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pages 12310- 12320. PMLR, 2021.
[15] M. Schultz and T. Joachims. Learning distance metric from relative comparisons. Advances in neural information processing systems, 16, 2003.

---

[16] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: self-super learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141. IEEE, 2018.
[17] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and paltern recognition, pages 9729-9738, 2020.
[18] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422) 04297, 2020.
[19] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-
21284, 2020.
[20] M. Caron, I. Mísra, J. Mairal, P. Goyal, P. Bojanowski, and Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
[21] K. Ayush, B. Uzkent, C. Meng, K. Tanmay, M. Burke, D. Lobell, and S. Ermon. Geography- aware self-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1018]-10190, 2021.
[22] D. Zhang, F. Nan, X. Wei, S. Li, H. Zhu, K. McKeown, R. Nallapati, A. Arnold, and B. Xiang. Supporting clustering with contrastive learning, arXiv preprint arXiv:2103. 12953, 2021.
[23] H. Zhong, J. Wu, C. Chen, Huang, M. Deng, Nie, Z. Lin, and X.-S. Hua. Graph contrastive clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9224-9233, 2021.
[24] O. Majumder, A. Ravichandran, Maji, M. Polito, R. Bhotika, and S. Soatto. Revisiting contrastive learning for few-shot classification. CoRR, abs/2101.11058, 2021. URL https: l/arxiv.org/ aaaaaaas//100 11058.
[25] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Kr- ishnan. Supervised contrastive learning, Advances in Neural Information Processing Systems, 33:18661-18673, 2020.
[26] Y. Zhang, X. Zhang. J. Li, R. Qiu, H. Xu, and Q Tian. Semi-supervised contrastive learning with similarity co-calibration. IEEE Transactions on Multimedia, pages 1-1, 2022. doi: 10. 1 1109/TMM.2022.3155000
[27] Alonso, A. Sabater, D. Ferstl, L. Montesano, and A. Murillo. Semi- suni-superrvied se- mantic segmentation with pixel-level contrastive learning from a class-wise memory bank. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8219-88 8228, October 2021.
(28) M. Zheng. F. Wang, S. You, C. Qian, C. Zhang, X. Waang and C. Xu. Weakly supervised contrastive learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 10042-10051, 2021.
[29] M. Wilher, S. Kwak, D. Kriegman, and S. Belongie. Learning concept embeddings with combined human-machine expertise. In Proceedings of the IEEE International Conference on Computer Vision, pages 981 981-989, 2015.
[30] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual review of control, roboties, and autonomous systems, 3: 297-330, 2020.
[31] D. Mukherjee, K. Gupta, L. H. Chang, and H. Najjaran. A survey of robot learning strate- gies for human-robot collaboration in industrial settings. Robotics and Computer-Integrane Manufacturing, 23:102231,2000
10

---

[32] H. Karnan, G. Warnell, X. Xiao, and P. Stone. Voila: Visual- coaaal--osevvaooooneeee imitation learn- ing for autonomous navigation. In 2022 International Conference Robotics and Automation (ICRA), pages 2497-2503. IEEE, 2022.
[33] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. 2017.
[34] K. Lee, L.. Smith, and P. Abbeel. Pebble: Feedback- interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091, 2021.
[35] N. Wilde, D. Kulić, and S. L. Smith. Active preference learning using maximum regret. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10952-10959. IEEE, 2020.
[36] N. Wilde, A. Blidaru, S. L. Smith, and D. Kulić. Improving user specifications for robot behavior through active preference learning: Framework and evaluation. The International Journal of Robotics Research, 39(6):651-66) 2020.
[37] S. van Waveren, C. Pek, J. Tumova, and I. Leite. Correct me if i' m wrong: Using non-experts to repair reinforcement learning policies. Proceedings of the 17th ACM/IEEE International Conference Human-Rood Robot Interaction, pages 1-9, 2022.
[38] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. arXiv preprint arXiv:2109. 10N.2200..0892 2021.
[39] D. Bolya, C. Zhou, F. Xiao, and Y. Lee. Yolact: Real-time instance segmentation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 9157- 9166, 2019.
[40] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[41] K. Chen, J. Wang, J. Pang. Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, et al. Mmde- tection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: arXiv: 1906.0716.. 2019.
[42] Z. C. Lipton, C. Elkan, and B. Naryanaswamy. Optimal thresholding of classifiers to maximize measure. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages
Springer, 2014.
[43] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D) Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects context. In European conference on computer vision, pages 740-755. Springer, 2014.
[44] P. Perera, P. Oza, and V. M. Patel. One-class classification: A survey, arXiv preprint arXiv:2101.03064 2021.


Reviews:
[{"review_id": "u-Adq6fh0M2", "reviewer": "Reviewer_nRzh", "summary": "This paper adapts modern visual object representations based on contrastive losses to better align with human visual categories. At a technical level, the problem is posed as a joint optimization of three objectives that maximize the agreement between: 1) multiple augmented views of the same image, 2) multiple instances from the same category of objects and 3) human-selected examples from a single category. This means that the contrastive component learns a latent space based purely on the visual features of the examples, while the human supervision loss is meant to align that latent space with more “human-like” features. Simulated experiments with perfect “human” input show that adding the supervised loss component does help the contrastive losses recover a representation that better classifies 10 object categories. A conveyor belt scenario with the same objects shows that (with pre-training) the method can correctly classify each category after 30 (expert) examples (per category). ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I thought the paper was well-written and clear, as was the supplementary video (which I really appreciated as an overview explanation of the paper before digging into the details). The paper is overall well-motivated in the introduction, and the experimental results and ablation well investigated. I also liked the fact that the experiments have object categories that are more interesting than the typical computer vision ones (crushed cans or bottles, intact cans or bottles, etc.). Lastly, I appreciated having a real robot demo on one of the more interesting applications I’ve seen in robotics recently. Despite these strengths, however, I still do have some issues with the paper that I’d like to detail.\n\nPerhaps my biggest gripe with the paper is that the technical novelty seems relatively limited: add a supervised loss to two existing losses. The experimental results do help make up for it, in that they demonstrate the usefulness of all 3 losses together in the ablation, but I would’ve appreciated more discussion and insight to make up for the rather simple contribution: 1. What led to choosing the architecture from Fig. 1? Instead of sharing the weights have you tried training the network iteratively with each loss? What’s your intuition behind why the current architecture would work best? How do you tune the lambda terms in Eq. (4)? 2. Does this method apply more broadly to more than just object properties or robotics? It seems like the proposed pipeline could work for more than just object classification or the conveyor belt example; perhaps a discussion expanding on this was left out to artificially fit into the CoRL robotics theme more strongly, but I would appreciate having it (maybe in the Conclusion). 3. In what settings would the assumption that the human always selects objects from the same category would apply? The conveyor belt application does seem a bit contrived with this assumption in place (I’d normally pick objects from any category and just place them in different bins, rather than sequentially clear them out one category after the other).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the technical novelty and originality of the work is weak, but the motivation is compelling and (I believe) well-timed, the results are compelling, and the application is interesting. As a result, I am leaning weak accept, but I look forward to more discussion as my opinion can be swayed in either direction.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please add more discussion addressing the comments above (architecture choices and loss optimization, broad applications beyond object categorization and robotics, feasibility of assumptions).\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "qpnvsbu09dU", "reviewer": "Reviewer_F5P5", "summary": "The work integrates human supervision into representation learning process to compute visual object representations inline with human requirements for interest tasks. The work combines existing contrastive learning method with a loss function that brings the representations of human-selected objects close to each other in the latent space. They evaluate the work on a simplified recycling setup that allows human-robot interaction.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths : (1) adapts to dynamic human-defined requirements by leveraging human-feedback (2) performs both online and offline evaluations, ablations, and outperforms some self-supervised learning baselines. (3) the case study has immediate relevance to real-world, though would like further explanation on the novelty of the architecture.\n\nweaknesses : (1) requires active human supervision which is expensive  (2) the objects used for the experiments are limited in variations to fully evaluate the performance of the system. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I read the paper, and used good judgement in assessing the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "(1) would like to see if authors considered any approaches to alleviate the need of active human feedback by instead relying on approaches that incorporate scalable human feedback ( example : using intermittent oversight or meta-imitation learning agents). (2) would also like to see experiments with more diverse set of objects"}}, {"review_id": "Xc6PvY8ln13", "reviewer": "Reviewer_2cDu", "summary": "The authors introduce a method for training a visual classifier in a few-shot manner from human preferences, for use in real-time robotics scenarios. In particular, they build upon existing methods from contrastive learning (e.g. contrastive clustering), but introduce an additional loss that tries to make the embeddings of all human-picked objects similar. They show through both offline and online experiments that their method shows small, but statistically significant, improvement over the baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The experiments are a clear strength here: the authors have successfully applied their method for the proposed task of object classification on a conveyor belt, where they set up an experiment where they performed 5 epochs of real-time training on a human subject's preferences for which objects to pick. This experiment validates empirically their claims over the usefulness of their model in their proposed use case, and the paper deserves recognition for this non-toy experiment alone.\n\nThe major weaknesses of this paper are indeed the limitations discussed in the limitations section. There is still a reliance on this idea of predefined, fixed-count clusters (which by the way is not explained in detail, see issues section). The human preference is only used as an additional \"cluster\". The main problem with this kind of approach is that it is not clear what will happen when the human chooses some objects from within a cluster but not others, e.g. the human cluster does not fully intersect or exclude any of the existing clusters. This use case is relevant in that it is not hard to imagine there would be other criteria besides the cluster assignment for how the human picks these objects (otherwise we could just classify based on the cluster label alone), and it is likely that in these cases the similarity score-based approach for classification (which is also not described in detail, see issues section) will lead to lower classification performance. Experiments of this nature were unfortunately not performed, so it is difficult to estimate what the actual impact would be.\n\nIt is also a clear weakness of this paper that the introduced method is equivalent to simply introducing an additional, non-exclusive cluster to the existing idea of contrastive clustering, which is mostly a very incremental improvement. However, as an evaluation of this approach to a novel problem (few-shot human in the loop learning), the paper is valuable as a case study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the proposed method is mostly an incremental improvement over an existing, known approach; it is also an interesting case study of how this approach can be applied to a new & interesting problem in the context of human-in-the-loop learning. As a result, I lean that the paper be accepted, but do have concerns about the limitations of the approach and how these were not exactly addressed (e.g. mostly avoided) in the experiments.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "After reading the paper, two clarity issues I noticed are below, and it would be great if these could be addressed.\n\n1. The idea of predefined, fixed-count \"clusters\" is not adequately introduced, I believe this is due to the authors' familiarity with the related works such that this idea comes naturally. However, as a robotics reviewer not extremely familiar with contrastive learning it was difficult to understand from this paper alone where the clusters came from, and their limitations.\n\n2. While the method for training the embeddings is described in detail, how the embeddings are used to classify an object for picking or not is not immediately obvious. While this similarity score idea is mentioned in the experiments it is still not fully clear what exactly is used for the pick/dont pick decision, and I think this is better suited for the method section.\n\nAdditionally, it is a valuable contribution (although maybe too costly at this point) to actually provide the evaluations of scenarios where the human prefers some objects in some clusters but not all."}}, {"review_id": "J-TSHQ5BMtu", "reviewer": "Reviewer_o67U", "summary": "The paper proposes a method for learning visual object representations with human supervision. For this purpose, the authors suggest three components: (i) instance prediction for multiple views of the scene, (ii) cluster prediction for varying instances of the same cluster, and (iii) human supervision for human-selected examples. The authors evaluated their method with offline experiments as well as in real-time with a robot in a recycling domain.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper focuses on a critical topic and is well motivated. Its technical quality is good, and there are detailed ablation studies showing the contribution of each suggested component. It has also been evaluated in a case study with a recycling robot, and I appreciate the authors for deploying the system to the real world and showing its applicability. \n\nWeaknesses:\n\nThe paper is motivated by stating that \"a robot may be asked to interact with a wide variety of objects, making it hard or even impossible to pre-program visual object classifiers suitable for the task of interest\". However, the suggested method uses a pre-defined number of object categories $C$ and clusters $K$. Consequently, it focuses on a limited number of objects, and handling a wide variety of objects can be challenging for the suggested system in more complex applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper focuses on a limited number of objects, and it could have been improved by including more baseline methods from the literature, it presents a strong contribution to a critical topic with good technical quality and detailed analysis.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- The paper nicely provides the background on self-supervised and contrastive learning. However, human-in-the-loop robot learning hasn't been mentioned in the related work section, which is one of the main contributions of this paper. It would be nice to include this literature for more comprehensive related work.\n- What is $N$ in Equation 1? Is it the number of all positive pairs in a mini-batch?\n- The authors mention 80% train and 20% validation set in the data collection. Was there a separate test set for including human supervision?\n- Was the model shown in Figure 1 trained end-to-end?\n- The offline evaluation results are not easy to follow in the text:\n  - As far as I understand, Figure 4 (b) and Experiment 5 in Figure 3 represent the same condition (Cluster Loss + Human Supervision). If it is the case, it would be great to use the same notation in both figures. Also, I am not sure whether Figure 4(c) and experiment 7 in Figure 3 represent the same condition. \n  - It is not clear which conditions have been taken from the literature. I think Experiment 2 in Figure 3 is identical to SimCLR [10]; are there any others?\n  - The colors in Figure 4 are not straightforward to interpret. It might be better to make this figure gray-scale."}}]

Author rebuttal:

Decision: Accept (poster)