Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations
Authors: Miyu Goko
Motonari Kambara
Daichi Saito
Seitaro Otsuki
Komei Sugiura
Venue: CoRL 2024 (2024)
Keywords: Task Success Prediction
Open-Vocabulary Manipulation
Multi-Level Aligned Visual Representation
TL;DR: We introduce a task success prediction model for open-vocabulary manipulation. The model focuses on the differences between multi-level aligned representations of images. It outperformed existing models, including representative multimodal LLMs.

Abstract: In this study, we consider the problem of predicting task success for open-vocabulary manipulation by a manipulator, based on instruction sentences and egocentric images before and after manipulation. Conventional approaches, including multimodal large language models (MLLMs), often fail to appropriately understand detailed characteristics of objects and/or subtle changes in the position of objects. We propose Contrastive $\lambda$-Repformer, which predicts task success for table-top manipulation tasks by aligning images with instruction sentences. Our method integrates the following three key types of features into a multi-level aligned representation: features that preserve local image information; features aligned with natural language; and features structured through natural language. This allows the model to focus on important changes by looking at the differences in the representation between two images. We evaluate Contrastive $\lambda$-Repformer on a dataset based on a large-scale standard dataset, the RT-1 dataset, and on a physical robot platform. The results show that our approach outperformed existing approaches including MLLMs. Our best model achieved an improvement of 8.66 points in accuracy compared to the representative MLLM-based model.

Task Success Prediction for Open- Voca-Vocyy Manipulation Based on Multi-Level Aligned Representations
Miyu Goko* Motonari Kambara* Daichi Saito Seitaro Otsuki Komei Sugiura
Keio University, Japan (miyu.goko goko, aetonari.k714. daichi-s, otsu8sei14, komei sugiurasugurra @eeaoe,keee...
Abstract: In this study, consider the problem of predicting task success for open-vocabulary manipulation by manipulator, based instruction sentences and egocentric images before and after manipulation. Conventional approaches, including multimodal large language models (MLLMs), often fail to appropriately understand detailed characteristics of objects and/or subtle changes in the position of objects. We propose Contrastive X-Repformer, which predicts task success for table-top manipulation tasks by aligning images with instruction sentences. Our method integrates the following three key types of features into multi-level aligned representation: features that preserve local image information; features aligned with natural language; and features structured through natural language. This allows the model to focus on important changes by looking at the differences in the representation between two images. We evaluate Contrastive X-Repformer on a dataset based on a large-scale standard dataset, the RT-1 dataset, and on a physical robot platform. The results show that our approach outperformed existing approaches including MILMMs. Our best model achieved improvement of 8.66 points in accuracy compared to the representative MLLM-based model.
Keywords: Task Success Prediction, Sgen-Vocabulary Manipulation, Multi-
Level Aligned Visual Representation
[b]
place a red can on The fron gid"
"erove the map to the left did
"pick the yellow banana"
"erove the l's cube dose to the mag cuc"
Encoder
Rolld...
CUP, SpLi
Text Enooder
Represenaatin
Aigaee Representatio
Raprmeraiion
R+Pepessentation
Succoss
Success
Fallure
Fallure
Figure 1: (left) An overview of the novel representation: A- Represennattinn, which is an integration threc types of representations. (right) A few examples of our task. The task is to predict success or failure based on open- voccbuocayllary instruction sentence, and egocentric images taken before and after the manipulation.
Introduction
Task success prediction in object manipulation ensures precise and efficient operations, enhancing reliability and consistency across robotic applications in healthcare, manufacturing, agriculture, and logistics. For example, in object manipulation tasks such as assembling parts in manufacturing [1, 2] and harvesting crops in agriculture [3, 4], task success prediction can improve the quality, efficiency, and productivity the tasks. The ability manipulator to accurately predict the success or failure
denotes equal contribution. Project page available https://Sei7440 ott.tht.....cccaaaaaaeeeeaaaaaar
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

of sub-tasks is particularly important for long- bong-horizon tasks because a failure in a sub-task may affect subsequent ones.
In this study, we focus on task which involves predicting the success or failure of an open- vocabulary manipulation, given an instruction sentence and egocentric images before and after the manipulation. Typical use cases involve scene where a manipulator is given the instruction sen- tence: "Place the mug into the sink." In the case where the manipulator drops the mug, the model is expected to predict failure based on the instruction and egocentric images. On the other hand, in the case where the mug is successfully placed into the sink, the model is expected to predict success.
Our target task is challenging because demands two key aspects. First, it is necessary have an adequate understanding of the changes in the images taken pre- and post-manipulation, information about the objects in the images, and open voca-uocy instructions. The task also requires the model to determine if the elements above align. Even multimodal large language models (MLLMs [5, 6, 7J) demonstrate limited performance on this task as we will show in the experimental results (See Section 4.2). This because MLLMs often fail to appropriately understand detailed characteristics of objects (e.g., colors and shapes) and subtle changes in the position of objects, both which are critical for success prediction.
We propose Contrastive X-Repformer, which performs task success prediction for table-top open- vocabulary manipulation by aligning images with instruction sentences. The method achieves this by utilizing visual representations that integrate three key types of features which are the following: (i) features that preserve local image information, (ii) features aligned with natural language, and (iii) features structured through natural language (Fig. 1). This addresses problem in conventional methods that rely solely on single visual representation extraction mechanism: they struggle to extract both detailed visual features, such textures and shapes of objects, and global structural representations, such spatial relationships between objects. The method also employs a represen- tation of the difference between the images, allowing it to effectively align the manipulations with the instruction sentences. This alignment enables the model to understand instruction sentences by considering the specific characteristics of objects and their spatial relationships.
We make the following contributions:
We introduce A-Representation Encoder, which computes the aforementioned three types visual representations and integrates them into X-Representation for the image. A- Representation integrates three types of features: (i) features retaining visual characteristics such as colors and shapes, (ii) features aligned with natural language, and (iii) features that are structured through natural language.
We propose Contrastive >-Representation Decoder, which identifies the difference between X- A-Representations of two images. This allows the model to take into consideration the alignment between the differences in the images and the instruction sentence when per- forming task success prediction.
2 Related Work
Recent research on foundation models (e.g. [8, 5, 9]) has made significant breakthroughs in the field of robotics [10, 11, 12, 13. 14, 15]. Several surveys [16, 17, 18] provide a comprehensive summary of various ML.LM-based models in the robotics field. In multimodal language understanding tasks for robotics, various datasets are utilized as representative benchmarks real-world settings [19, 20, 21, 22] and in simulation settings [23, 24, 25, 22]. These datasets primarily focus on object manipulation tasks within indoor environments.
LLM-Based Task Planning. For object manipulation tasks, large language models (LLMs) are often employed as task planners [26. 27, 28, 13, 29, 30. 31]. For example, in some studies, LLMs are utilized to generate sub-goals from high-level instruction sentences [26, 27, 28, 31]. This ap- proach involves replanning using the LLMs based feedback received from the environment when task failure is detected. On the other hand, some methods (e.g. [13, 29]) use LLMs to directly generate Python code for robot policies based on natural language instructions. Other works have

---

Scone Representatin Module
Contrastive A-Representation Decoder
Algned fleemsentaaion Module
Trar Transformrr
Faod Farwer
Cros Adt ton
A-Representation Encoder
MPP
-P(==1)
Food
M
Scone Repreon
h
Nater
Algned Rieprssettatinn
Narrative Regresentation Module
A-Representation Encoder
Mis
drawer and place on counter"
top
Language Encecer
Figure 2: Overview of Contrastive X-Repformer. Given instruction sentence and images before and after manipulation, our model outputs the predicted probability that the robot successfully performed the manipula- tion.
also explored LLM-based reward generation, including grounding the reward in the 3D observation space [10, 32, 33, 34, 35]. While REFLECT [30] is closely related to method, it determines task success by predefining the target state for each object class and verifying whether these states are achieved. Consequently, unlike our method, it is difficult for REFLECT to perform success prediction without using predetermined target states.
Our method is also closely related to MLLL-basee task planning models (e-g. [27, 28, 36)). Unlike them, we employ MLLMs for the purpose of structuring images through natural language. Fur- thermore, introduce a mechanism that extracts visual representations through two other types of modules and integrates them alongside the MLLMs. This allows the model to consider visual rep- resentation with multi-level alignment that simple MLLM-based approaches cannot fully capture.
Task Success Prediction. Most tward-based approaches require both expert knowledge and sig- nificant effort to manually design rewards that consider all of the states during manipulation. Mean- while, our method needs only the states before and after manipulation. Inverse reinforcement learn- ing methods [37, 38, 39, 40] aim to learn reward functions from optimal demonstrations, However, obtaining such demonstrations can be costly and sometimes unfeasible. Alternatively, some strate- gies train agents by acquiring rewards through interactive human feedback [41, 42, 43, 44). How- ever, this approach is limited by the necessity of having human supervision for real-time queries. In contrast, representative methods [26, 45] that do not require optimal demonstrations or human su- pervision have been proposed. Notably, PaLM-E [26] is one prominent object manipulation model that utilizes Visual Question Answering, achieving 91% % success rate on the failure detection task with the dataset proposed in [27]. However, PaLM-E has large model size, which is problem in robotics where computational resources are often limited. Also, the dataset used for failure detection in [26] included only 101 episodes and 15 objects. Thus, we constructed a new dataset based on the RT-1 dataset [ [19], which has approximately 1,000 episodes and 30 objects.
The collision prediction task during object manipulation also related to our task. For example, there are some post-collision decision strategies (e.g. [46]). Furthermore, several methods predict collisions from image and placement policy [47, 48, 49]. Our method differs from these in that it can take into account factors other than collisions that contribute to task failure.
Using Captions. Scene change captioning models aim to generate descriptions about the differences between two images [50, 51, 52, 53, 54, 55]. This task related to our task that it requires the identification of the differences between two images. However, they have difficulty handling instructional sentences input. Thus, it is not possible directly apply those models to task.
3 Proposed Method
Our target problem is to predict whether an open-vocabulary manipulation task was performed suc- cessfully, given instruction sentence and egocentric images taken before and after the manipula- tion. We define this task as Success Prediction for Open-vocabulary Manipulation (SPOM). In this

---

task, models are expected to appropriately predict the success or failure of an object manipulation.
The inputs consist of an instruction sentence, one egocentric image taken before the manipulation, and another taken after. The expected output is the predicted probability P = 1), indicating the probability that the manipulator successfully executed the open-vocabulary manipulation specified in the instruction sentence. Here, 9 represents the success or failure of the manipulation, with 'I' indicating success. In this study, we only egocentric images as input images. Note that in some images, the scene is partially occluded by the manipulator. While the task is feasible, this often makes it challenging, as the objects or areas may be partially occluded.
Fig. shows the structure of the proposed method, Contrastive À- X-Repformer. Its input defined as [Finst: #before Fafter/, where Tinst represents a tokenized instruction, while #before and Fafter represent RGB images taken before and after manipulation, respectively. The main modules of the proposed method are X- X-Representation Encoder and Contrastive X-Representation Decoder.
3.1 >-Representation
In existing Vision-and-Lang Language studies, there are primarily three approaches for extracting visual features. The first approach uses unimodal image encoders [56, 57, 58] to extract visual features like textures and edges; we refer to these features as "Seene Representation." " The second approach employs multimodal image encoders [8, 59, 60, 61] to extract visual features aligned with natural language, referred to here as "Aligned Representation." " The third approach utilizes MLLMs [5, 6. 7] to extract structural features that directly represent complex referring expressions and spatial relationships through natural language, termed "Narrative Representation" in this paper.
However, most existing methods do not comprehensively handle all the above representations, lim- iting the expressiveness of visual features. Specifically, Scene Representation, despite its ability to capture visual information like shapes and colors from images, cannot extract complex referring relations, including spatial relations. This limitation highlights the insufficiency of using this repre- sentation exclusively. In addition, while Narrative Representation is capable of extracting structural features through natural language, it is difficult to capture all the detailed visual features, such tex- tures, with only this representation. Unlike these representations, Aligned Representation is aligned with natural language, sharing characteristics with both Scene and Narrative Representations. How- ever, using only Aligned Representation often leads to a lack of ability to structurally understand complex referring expressions in instruction sentences, because it does not extract structural fea- tures through natural language. From the above, expected that we obtain sufficient visual representations by using all these features in parallel.
3.2 >-Representation Encoder
We introduce A- A-Representation Encoder, designed to generate X- X-Representation effectively. In this module, we obtain the three types of visual representations and integrate them into X-Representation. As shown in Fig. 2, this module consists of three sub-modules: Scene Representation Module, Aligned Representation Module, and Narrative Representation Module. X-Representation Encoder takes either Tbefore or Patter as input. The following explanation will focus solely on Bbefore- be- cause the same process is applied Tafter
First, we obtain Scene Representation h, = ferm (xbeforr), where ferm(.) represents Scene Rep- resentation Module. Scene Representation Module consists of several backbone networks. In this paper, we use ViT [56], DINOv2 [58], and the CLIP image encoder [8] as backbone networks. For VIT and DINOv2, the output features are used, while the intermediate features are utilized for the CLIP image encoder. Then, h, is acquired by concatenating them.
Next, we acquire Aligned Representation h2 using Aligned Representation Module, which is com- posed of multimodal foundation models. These features can be regarded as Aligned Representa- tions, because they are well-aligned with natural language. We employ the CLIP image encoder and extract its output features.
Subsequently, Narrative Representation h. is obtained using Narrative Representation Module, taining a MLLM and multiple text embedders. We utilize InstructBLIP [7] to generate description

---

from Dbefore- We designed text prompt to focus on the colors, sizes, and shapes of objects, as well as how they are placed, their positions within the image, and their relative positions to other objects. From the output of InstructBLIP, we acquire its features using BERT and text-embeddingg3-- --lagge [62]. Then, these features are concatenated to obtain hn. Finally, we obtain X-Representation for #before denoted Ny [h! h1 []] Similarly, we obtain h2 X-Representation for Wafter
3.3 Contrastive A- A-Representation Decoder
We introduce Contrastive X- R-Representation Decoder to create representation of the difference be- tween hĄ and h Since the effects of the manipulation are included in the change between the images, the representation allows the model to focus on the difference, which may be attributed to the manipulation. On the other hand, a difference between the images does not necessarily indicate the success of the task specified by the given instruction sentence. For example, in the case shown Fig. 2, if the Pepsi can were to fall over, there would be a difference hetween the two images; however, the manipulation should be considered a failure. Thus, is hard to consider the success of manipulation based solely the differences between images. Consequently, when predicting the success or failure of manipulation, it is important to consider the alignment between the difference representation and the instruction sentence.
The inputs this module are hx, 1t', and h;, and the output is P(ù = 1). First, the representation of the difference hair between the two images are obtained as follows:
haiff CrossAttn (h's, hx),
where CrossAttn (..-) represents the cross-attention operation. We define this operation using two arbitrary matrices XA and XB B as follows:
X (XW,(X XBW. vdx
CrossAttn (XA, XB)= = softmax
(2)
where Wg. Ws. and W are trainable weights, and dy denotes a dimension X Ws. Then, the alignment feature haliga between haif and h; computed as follows:
aalign CrossAttn (hair ha)-
(3)
Finally, we compute P(ğ = 1) from haiggn as the output of this module using a multi-layer percep- tron. We use the cross entropy loss as the loss function.
Experimental Results
4.1 Experimental Setup
We constructed the novel SP-RT-1 dataset from the RT-1 dataset for the SPOM task. The task re- quires all of the following components for each episode: an instruction sentence, images taken before and after the manipulation, and labels indicating the success or failure of the manipulation. The RT-1 dataset is standard, large-scale dataset for real-world open-vocabulary manipulation. It includes instruction sentences, images collected during manipulation, and binary rewards. Because the RT-1 dataset cannot be utilized directly, the SP-RT-1 dataset was assembled from the RT-1 dataset. We collected the first and last images of each episode and got the ground truth success/failure labels by using the binary rewards from the RT-1 dataset. The dataset was preprocessed by modifying the in- struction sentences. Data cleansing was conducted because the rewards were sometimes erroneous. The details of the SP-RT-1 dataset are explained Section 1.
We used UNITER UNITER-base/large bbsse/lrrgeeeeeeeeeeeeee [59], the method by Xiao et al. [45], InstructBLIP Vicuna-7B (In- structBLIP) [7], GPT-4 Turbo with Vision (GPT-4V) [5], and Gemini 1.0 Pro Vision (Gemini) [6] as baseline methods. The capability of InstructBLIP was evaluated in a zero-shot manner, while GPT-4V and Gemini were evaluated in both zero-shot and few-shot settings. Each method was used as baseline method for the following reasons. UNITER demonstrated competitive performance in many Vision-and-Language tasks, including Visual Question Answering tasks. The model by Xiao et al. is failure detection model based on the two images and an instruction sentence. This per- formance is reported to be competitive with PaLM-E, large-scale model for object manipulation
5

---

in robotics. Additionally, InstructBLIP, GPT-4V, and Gemini are representative MLLMs that have been pretrained on large-scale datasets and have demonstrated outstanding performance on various tasks. The details of baseline methods are explained Section A.3.4.
For a comprehensive evaluation, we also validated model in physical environment using mobile manipulator with zero-shot settings (SP-HSR bench- mark). Fig. 3 shows the experimental environment, which is based on the standardized environment of WRS2020 [63]. We used Toyota's Human Support Robot, which is standardized in RoboCup @ Home competitions [64]. This dataset was annotated by hu- mans during its construction. Specifically, each sam- ple was labeled as 'Success" if the images matched the instructions; otherwise. it was labeled as FFail- ure.' In the experiment, all methods were evaluated in zero-shot settings. This means no additional train- ing was conducted using the collected data. The de- tails of the dataset for this experiment are explained in Section A.3.2. The implementation details are also explained in Section A.3.3.
Experimental environment. The left and right images show the state beforc and after manipulation, respectively. Instruction sentences, such as "place mug in front of the banana," were created based on the situation before the manip- ulation. Examples of the egocentric images are shown at the top right of each exocentric image.
Figure 3:
4.2 Quantitative Results
Table presents the quantitative sults of a comparison between several baseline methods and Contrastive A- Repformer. As listed in Table 1, on the SP-RT-1 dataset, Contrastive X-Repformer achieved the highest accuracy of 80.80%, outperforming UNITER-base, UNITER-large, and the method by Xiao et al. with accuracies of 62.78%, 63.52% and 68.26%, respectively. Furthermore, Contrastive X-Repformer also out- performed MILMs: InstructBILIP, GPT-4V (Zero-shot), GPT-4V (Few- shot), Gemini (Zero-shot), and Gem- ini (Few-shot) with accuracies of 50.50%, 63.90%, 72.14%, 67.28%, and 68.44%, respectively. These results demonstrate that the proposed method outperformed both zero/few-shot MLLMs and other baseline methods. The dif- ferences in accuracy between Contrastive A-Repformer and each baseline method were statistically significant (p < 0.001).
Accuracy [%]
Method
SP-RT-1
SP-HSR
UNITER- base [59]
62.78 1.01
52 1.6
UNITER- [59]
63.52 + 1.84
48 = 1.8
Xiao et [45]
71.59 1.95
InstructBLII [7]
50.50 + 0.00
50 = 0.0
GPT-4V [5] (Zero-shot)
63.90 1.04
59 = 1.9
GPT-4V [5] (Few-shot)
72.14 0.92
56 = 1.9
Gemini [6] (Zero-shot)
67.28 0.80
53 0.40
Gemini [6] (Few-shot)
68.44 0.76
53 3.3
Contrastive X-Repformer
80.80 0.86
60 = 1.8 79
Human (Reference)
90
Table 1: Quantitative results of the baseline and proposed meth- ods on the SP-RT-1 dataset and the SP-HSR benchmark. Here, SP-HSR represents our benchmark using physical environment. Bold indicates the accuracy with the highest value.
Table also shows the quantitative results of the SP-HSR benchmark. The accuracies of UNITER- base, LargeR-large, and Contrastive X-Repformer were 52%, 48%, and 60%, respectively. More- over, InstruciBLIP, GPT-4V (Zero-shot). GPT-4V (Few-shot), Gemini (Zero-shot), and Gemini (Few-shot) were 50%, 59,, 56%, 53%, and 53%, respectively. The accuracies of most meth- ods were nearly at chance level. On the other hand, GPT-4V (Zero/Few-shor and Contrastive X-Repformer showed better results compared to the other methods. Furthermore, the accuracy of Contrastive X-Repformer slightly outperformed GPT-4V (Zero-shot) and (Few-shot).
We conducted a subject experiment with five subjects to evaluate the human performance for the task. For the SP-RT-1 dataset, 100 samples were randomly selected from the test set and the sub- jects performed the SPOM task on these samples, achieving an accuracy of 90%. For the SP-HSR benchmark, the entire dataset was used, with humans achieving an accuracy of 79%. From this result, it is found that the SPOM task can be difficult even for humans.

---

(ii) "pick rxbar chocolate" Figure 4: Successful cases of Contrastive X-Repformer the SP-RT-1 dataset. Examples (i) and (ii) are true positive cases, and (iii) negative In each example, the left and right images show the scene before and after the manipulation, respectively.
(i) "place water bottle upright"
(iii) "pick apple from white bowl"
(ii) "move the rubik's cube close the banana" Figure 5: Qualitative results of the proposed methed in zero-shot transfer experiment. Examples (i) and (ii) true positive and negative cases, respectively. each example, the left and right images show the scene before and after manipulation, respectively.
(i) "place purple cup the front right"
4.3 Qualitative Results
Fig. exhibits successful cases of Contrastive X-Repformer on the SP-RT-1 dataset. Fig. 4 (i) and (ii) are true positive cases, and Fig. (iii) is a true negative case. Fig. 4 (i) presents an example where the given instruction was "place water bottle upright." The manipulator successfully manipulated the water bottle, setting it down so that it was upright. Therefore, the example was labeled as success. Contrastive X-Repformer correctly predicted success for this example where all of the baseline methods excluding InstructBLIP failed to do Fig. 4 (ii) is instance where the manipulator executed the instruction of "pick rxbar chocolate," which can be observed from the fact that the chocolate is being held by the manipulator in the right image. While most of the baseline methods predicted that the example was failure, Contrastive X-Repformer was able to predict it as a success. Fig. (iii) is one example where the manipulator was not able to follow the instruction: "pick apple from white bowl." Neither the apple nor the white bowl is visible either of the images, which indicates failure in the manipulation. Contrastive X-Repformer successfully predicted that the manipulator failed in the task. Meanwhile, all of the baseline methods predicted success.
Fig. 5 shows successful examples in the SP-HSR benchmark. Fig. 5 (i) and (ii) are true positive and true negative cases, respectively. In Fig. (i), the instruction given was "place a purple cup on the front right." " The manipulator successfully put purple cup on the front right of the table. Therefore, this episode was labeled success. Contrastive X-Repformer successfully predicted while UNITER-base/lan incorrectly predicted could appropriately understand the spatial expression 'front right.' Fig. (ii) shows episode in which the instruction "move the rubik's 's cube close to the banana" was given. This episode was labeled as a failure because the manipulator moved a blue can instead of the Rubik's cube. In this episode, Contrastive X-Repformer made an appropriate prediction, while Gemini and InstructBLIP failed. This episode shows that the proposed method can also appropriately align natural language expressions with objects in the image.
failure. This result shows that the proposed method
4.4 Ablation Study
We conducted ablation studies to investigate the contribution of each representation in A- Representation. Table 2 presents the results. We set the following conditions:
Scene Representation Ablation. We removed Scene Representation from λ Repereseettaiion to as- sess its contributions. From Table 2, it can be observed that the accuracy of Model (i) was 73.72% which was 7.08 points lower than that of Model (vii). This signifies that Scene Representation en- hanced the visual representation by capturing detailed visual information such as shapes and colors.
Aligned Representation Ablation. Aligned Representation analyze its contributions. As shown in Table 2, the accuracy of Model (ii) was 79.94%, which was
omitted from X- A-pepeseenoation to
7

---

0.86 points lower than that of Model (vii). This shows that Aligned Representation improved the alignment between the instructions and the images, including better identification of object names.
Narrative Representation Ablation. We removed Narrative Representation from X-Representation investigate its contributions. Table 2 shows that Model (iii) achieved an accuracy of 79.70%, which was 1.10 points lower than that of Model (vii). This
Model
SR
AR
NR
Accuracy [%]
73.72 = 0.86
79.94 = 0.40
V
79.70 = 0.89
(iv)
80,36 + 0.62
indicates that Narrative Representation enhanced the visual representation by extracting features struc- tured through natural language.
V
74.90 0.55
(vi)
61 31.80 + 0.47
(vii)
80.80 = 0.86
The accuracy of the models with only Scene Rep- resentation, Aligned Representation, and Narrative Representation were 80.3, 74.0, and 61.8, respec- tively. From this result, it can be concluded that Scene Representation alone yields the highest accuracy when only a single representation is used, but has a lower accuracy than our proposed model with all three of the representations.
Table 2: Results of ablation study. Bold indi- cates the highest value. SR, AR and NR represent Scene, Aligned and Narrative Representation. spectively.
The results demonstrate that each representation in X- >-Representation significantly contributed to the overall performance of the model. Particularly, it was found that Scene Representation contributed the most to performance improvement. Therefore, it can be said that this task is too challenging to be solved solely by MLLMs without explicitly using features of detailed characteristics. Constructing a model that integrates features obtained from MLLMS and other features, such as those represented by A- A-Representation, is effective for the task.
Conclusions and Limitations
In this study, we focused on a task to predict the success or failure of open-vocabulary manipulation, given an instruction sentence and egocentric images before and after the manipulation. Our contribu- tions can be emphasized as follows: We introduced the J-Representation Encoder, which generates the multi-level aligned visual representation, A-Representation. This representation consists of: (i) features that maintain visual characteristics such colors and shapes, (ii) features aligned with nat- ural language, and (iii) features structured through natural language. We also introduced Contrastive X- Reeeresentation Decoder, which finds differences between two images, and enables the model to consider the alignment between the difference and an instruction sentence. Additionally, Contrastive X-Repformer outperformed baseline methods, including representative MLLMs.
Limitations. Although Contrastive X-Repformer generated compelling results, it has several lim- itations. Firstly, it assumes the availability of either local (e.g. InstructBLIP [7], LLaVA [65]) or cloud-based (e.g. Gemini [6]. GPT-4V [5]) MLLMs to extract Narrative Representation; however, there are limitations associated with them. The former has limitations in terms of memory and in- ference time due to the large parameter size during inference. The latter cannot be used within a stand-alone system. Second of all, as stated in Section 3, the input images of this study were ego- centric images. Thus, there were samples where objects directly related to the manipulation were occluded or were outside the photographed scene. these cases, it is difficult to execute the task appropriately. Finally, in the experiments conducted for this study, we focused on limited set of open-voue vocuuuuaab manipulation tasks, such pick and place. Therefore, Contrastive X-Repformer is not intended to be applied directly to tasks such as navigation and mobile manipulation, making it difficult to solve such tasks. In future research, we plan to apply the method to a wide range of manipulation and navigation tasks (e.g., [66, 27, 14J). A possible solution could be to compare the images taken before and after the mobile manipulation. For example, when given an instruction "move the cup on the dining table to the shelf." model can predict the success of the task based on the images of the dining table prior to the task and the shelf afterward.
8

---

Acknowledgments
This work was partially supported by JSPS KAKENHI Grant Number 23K03478, JST Moonshot, and NEDO.
References
[1] Zachares, M. Lee, W. Lian, and J. Bohg. Interpreting Contact Interactions Overcome Failure in Robot Assembly Tasks. In ICRA, pages 3410-3417, 2021.
[2] J. Behrens, R. Lange, and M. Mansouri. A Constraint Programming Approach to Simultancous Task Allocation and Motion Scheduling for Industrial Dual-Arm Manipulation Tasks. In ICRA, pages 8705-8711, 2019.
[3] C. Lehnert, A. English, C. McCool, A. Tow, and T. Perez. Autonomous Sweet Pepper Har- vesting for Protected Cropping Systems. IEEE RA-L, 2(2):872-879, 2017.
[4] J. Jun, J. Kim, J. Seol, J. Kim, and H. Son. Towards an Efficient Tomato Harvesting Robot:
31) Perception, Manipulation, and End-Effector. IEEE Access, 9:17631-17640, 2021.
[5] J. Achiam, S. Adler, S. Agarwal, et al. GPT-4 Technical Report. arXiv:2303.088 2023.
arXiv preprint
[6] G. GeminiTeam, R. Anil, Borgeaud, Y. Wu, B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, al. Gemini: A Family of Highly Capable Multimodal Models. arXiv preprint arXiv:23/2.//805, 2023.
[7] W. Dai, J. Li, D. Li, A. Tiong. Zhao, W. Wang. B. Li, P. Fung, and S. Hoi. InstructBLIP: To- wards General General-purpose purpose Vision-Language Models with Instruction Tuning. In NeurlPS, 2023.
[8] A. Radford, K. Wook, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, et al. Learning Transferable Visual Models From Natural Language Supervision. In JCML, pages 7748-8763. 2021.
[9] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting Twenty-thousan Classes using Image-level Supervision. In ECCV, pages 350-368, 2022.
[10] W. Huang, C. Wang, R. Zhang, Y. Li, Wu, and L. Fei-Fei. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. In CoRL, pages 540-562, 2023.
[11] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and Where Pathways for Robotic Manipulation. In CoRL, pages 894 906, 2022.
[12] Y. Ma. S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. VIP: Towards Univer- sal Visual Reward and Representation via Value-Implicit Pre- Pre-Training In ICLR, 2023.
[13] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, et al. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In ICRA, pages 11523- 11530, 2023.
[14] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3M: A Universal Visual Repre- sentation for Robot Manipulation. In CoRL, pages 892-909, 2023.
[15] R. Korekata, M. Kambara, Y. Yoshida, S. Ishikawa, Y. Kawasaki, M. Takahashi, and K. Sug- iura. Switching Head-Tail Funnel UNITER for Dual Referring Expression Comprehension with Fetch-and-Cary Tasks. In IROS, pages 3872-3872, 2023.
[16] R. Firoozi, J. Tucker, S. Tian, A. Majumdar, J. Sun, W. Liu, Y. Zhu, S. Song, al. Foundation Models in Robotics: Applications, Challenges, and the Future. arXiv preprint 2xX1v.231..7884, 2023.

---

[17] F. Zeng, W. Gan, Y. Wang, N. Liu, and S. Yu. Large Language Models for Robotics: A Survey. arXiv preprint arXiv:231 1.07226, 2023.
[18] K. Kawaharazuka, Matsushima, A. Gambardella, J. Guo, C. Paxton, and A. Zeng.
Real-World Robot Applications of Foundation Models: A Review. arXiv:2402.05741, 2024.
arXiv preprint
[19] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, et al. RT- I: Robotics Transformer for Real-World Control at Scale. arXiv preprint arXiv:2212.06817, 2022.
[20] Y. Qi, Q. Wu, P. Anderson, X. Wang, Y. Wang, C. Shen, and A. Hengel. REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments. In CVPR, pages 9982- 9991, 2020.
[21] Hatori, Y. Kikuchi, S. Kobayashi, K. Takahashi, Y. Tsuboi, et al. Interactively Picking Real- World Objects with Unconstrained Spoken Language Instructions. In ICRA, pages 3774-3781, 2018.
[22] Z. Liu, A. Bahety, and S. Song. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. In CoRL, pages 3468-3484, 2023.
[23] M. Shridhar, Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, et al. ALFRED: Benchmark for Interpreting Grounded Instructions Everyday Tasks. CVPR, pages 10740-10749, 2020.
[24] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, et al. BEHAVIOR-IK: A Benchmark for Embodied AI with 1,000 Everyday Activities and Realistic Simulation. In CoRL, pages 80-93, 2023.
[25] K. Zheng, X. Chen, C. Jenkins, and X. Wang. VLMbench: A Benchmark for Vision-and- Language Manipulation. NeurIPS, 35:665-674 678, 2022.
[26] D. Driess, F. Xia, M. Sajjadi, Lynch, Chowdhery, Ichter, et al. PaLM-E: An Embodied Multimodal Language Model. ICML, volume 202, pages 8469-8488, 2023.
[27] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, et al. Do As Can, Not As I Say: Grounding Language in Robotic Affordances. In CoRL, pages 287-318, 2023.
[28] W. Huang, F. Xia, T. Xiao, H. Chan, J. Liang, P. Florence, et al. Inner Monologue: Embodied Reasoning through Planning with Language Models. In CoRL, pages 1769-1788 2023.
[29] Liang, Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and A. Zeng. Code as Policies: Language Model Programs for Embodied Control. In ICRA, pages 9493-9500, 2023.
[30] Z. Liu, A. Bahety, and S. Song. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. In CoRL, volume 229, pages 3468-3484, 2023.
[31] L. Sun, D. Jha, C. Hori, S. Jain, R. Corcodel, X. Zhu, M. Tomizuka, and D. Romeres. In- teractive Planning Using Large Language Models for Partially Observable Robotics Tasks. In ICRA, 2024.
[32] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K. Lee, M. Arenas, al. Language to Rewards for Robotic Skill Synthesis. In CoRL, 2023.
[33] H. Ha, P. Florence, and S. Song. Scaling Up and Distilling Down: Language Skill Acquisition. In CoRL, pages 3777,
Language-Guided Guided Robot
10

---

[34] T. Xie, S. Zhao, C. Wu, Y. Liu, Q. Luo, V. Zhong, Y. Yang, and T. Yu. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. In ICIR, 2024.
[35] Sharma, Sundaralingam, V. Blukis, C. Paxton, Hermans, Torralba, Andreas, and
D. Fox. Correcting Robot Plans with Natural Language Feedback. In RSS, 2022.
[36] M. Shirasaka, T. Matsushima, S. Tsunashima, Y. Ikeda, A. Horo, S. Ikoma, C. Tsuji, H. Wada,
T. Omija, D. Komukai, Y. Matsuo, and Y. Iwasawa. Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery. In ICRA, 2024.
[37] A. Ng and Russell. Algorithms for Inverse Reinforcement Learning. In ICML, page 663-670, 2000.
[38] D. Ramachandran and E. Amir. Bayesian Inverse Reinforcement Learning. In IJCAI, page 2586-2591, 2007.
[39] S. Arora and Doshi. A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress. Artificial Intelligence, 297:103500, 2021.
[40] J. Ho and S. Ermon. Generative Adversarial Imitation Learning. NIPS, 29:4572 4580, 2016.
[41] B. Knox and P. Stone. Interactively Shaping Agents via Human Reinforcement: the TAMER Framework. In International Conference on Knowledge Capture, page 9-16, 2009.
[42] V. Veeriah, Pilarski, and R. Sutton. Face Valuing: Training User Interfaces with Facial Expressions and Reinforcement Learning. arXiv preprint 1XXi:166666028077 2016.
[43] P. Christiano, J. Leike, T. Brown, M. Martie, S. Legg, and D. Amodei. Deep Reinforcement Learning from Human Preferences. In NIPS, page
2017.
[44] Lin, Z. Ma, R. Gomez, K. Nakamura, B. He, and G. Li. A Review on Interactive Reinforce- ment Learning From Human Social Feedback. IEEE Access, 1:120777-120755, 2020.
[45] T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson. Skill Acquisition by Instruction Augmentation Offline Datasets. In LangRob @ CoRL222, 2022.
[46] S. Haddadin, A. Luca, and A. Albu-Schäffer. Robot Collisions: A Survey on Detection, Isola- tion, and Identification. T-RO, 1396)11222-1112, 2017.
[47] Mottaghi, Roozbeh and Rastegari, Mohammad and Gupta, Abhinav and Farhadi, Ali. "What Happens If..." Learning to Predict the Effect of Forces in Images. In ECCV, pages 269-285, 2016.
[48] A. Nakayama, A. Magassouba, K. Sugiura, and H. Kawai. PonNet: Object Placeability Clas- sifier for Domestic Service Robots. In SNL-2019, 2019.
[49] M. Kambara and Sugiura. Relational Future Captioning Model for Explaining Likely Col- lisions in Daily Tasks. In ICIP, pages 2601-2605, 2022.
[50] 7. Guo, T.J. Wang, and J. Laaksonen. CLIP4IDC: CLIP for Image Difference Captioning. In AACL (Volume 2: Short Papers), pages 33-42, 2022.
[51] H. Jhamtani and T. Berg-Kirkpatrich Learning to Describe Differences Between Pairs of Similar Images. In EMNLP, pages 4024 4024-4034, 2018.
[52] D. Park, T. Darrell, and A. Rohrbach. Robust Change Captioning. In ICCV, pages 4624 4633, 2019.

---

[53] Q. Huang, Y. Liang. J. Wei, Y. Cai, H. Liang, H. Leung, and Q. Li. Image Difference Caption- ing With Instance-Level Fine-Grained Feature Representation. IEFE Transactions on Multi- media, 24:2004-2017, 2022.
[54] Y. Sun, L. Li, T. Yao, T. Lu, B. Zheng, C. Yan, H. Zhang, Y. Bao, G. Ding, and G. Slabaugh. Bidirectional Difference Locating and Semantic Consistency Reasoning for Change Caption- ing. International Journal of Intelligent Systems, 37(5):2969-299) 2022.
[55] L. Yao, W. Wang, and Q. Jin. Image Difference Captioning with Pre-training and Contrastive Learning. AAAI, pages
2022.
[56] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, pages 12888-12900, 2020.
[57] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, In ICCV, pages 10012-10022, 2021.
[58] T. Darcet, M. Oquab, J. Mairal, and Bojanowski. Vision Transformers Need Registers. In ICLR, 2024.
[59] Y. Chen, L., Yu, Kholy, F. Ahmed, Gan, Y. Cheng, and Liu. UNITER: UNiversal Image-TExi Representation Learning. In ECCV, pages 104-120, 2020.
[60] J. Li, D. Li, C. Xiong, and S. Hoi. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In ICML, pages 12888-12900, 2022.
[61] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid Loss for Language Image Pre-
Training. In ICCV, pages 11975-11986, 2023.
[62] OpenAl. oodels/embedding Accessed: May. 2024.
text-embeddingg--largg.. 2024. URL https://plattfrm.opmnn com/docs/
[63] World Robot Summit 2020 Partner Robot Challenge Real Space Rules & Regulations, 2020.
[64] Yamamoto, K. Terada, A. Ochiai, F. Saito, et al. Development of Human Support Robot as the Research Platform of A Domestic Mobile Manipulator. ROBOMECH Journal, 6(1):1-15, 2019.
[65] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual Instruction Tuning. In NeurlPS, volume 36, pages 34892-34916 14 2023.
[66] H. Shi, H. Xu, S. Clarke, Y. Li, and Wu. RoboCook: Long-Horizon Horizon Elasto-Plastic Object Manipulation with Diverse Tools. In CoRI., pages 642-660, 2023.
[67] W. Wu, H. Luo, B. Fang, J. Wang, and W. Ouyang. VipeVideo: What Can Auxiliary Captions Do for Text- Video Retrieval? In CVPR, pages 10704-10713, 2023.
[68] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In NAACL-HLT, pages 4171-4186, 2019.
[69] OpenAl. wwweemmbbbddnng-ddd--on 2024. URL https://platfform.ppenn ctmeenaiccom/ooss// models/embeddinggs Accessed: Feb. 2024.
[70] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, al. DROID: A Large-Scale In-the-Wild Robot Manipu- lation Dataset. In RSS, 2024.
[71] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open X-Embodiment: Robotic Learning Datasets and RT-X Mod- els. In ICRA, 2024.
12

---

[72] C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martin,Martin, C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, et al. BEHAVIOR-IK: A Benchmark for Embodied Al with 1,000 Everyday Activities and Realistic Simulation. In CoRL, pages 80-93, 2023.
[73] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. Chang, L. Guibas, and H. Su. SAPIEN: A SimulAted Part-based Interactive ENvironment. In CVPR, pages 11097-11107, 2020.
[74] J. Gu, Xiang, X. Li, Z. Ling, X. Liu, T. Y. Tang, S. Tao, X. Wei, Y. Yao, et ManiSkil12:
A Unified Benchmark for Generalizable Manipulation Skills. In ICLR, 2022.
[75] A. Gupta, Kumar, Lynch, S. Levine, and K. Hausman. Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning. In CoRL, pages 1025-1037, 2020.
[76] B. Calli, A. Walsman, A. Singh, S. Srinivasa, et al. Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeleey Object and Model Set. IEEE RAM, 22(3):36-52, 2015.
[77] P. Manakul, A. Liusie, and M. Gales, SelfCheckGPT: Zero-Resouree Black-Box Hallucination Detection for Generative Large Language Models. In EMNLP, pages 9004-9017, 2023.
13

---

Appendix
1 Additional Related Work
Cap4Video [67] a representative video retrieval model based on natural language queries. This method is similar to our proposed method that it generates visual representations through natural language. However, Cap4Video uses only the features aligned with natural language, extracted by CLIP. It neither uses features that preserve local image information nor those structured through natural language. Thus, its capability to understand complex referring expressions is limited. In contrast, our method uses all three types of features. Additionally, Cap4Video requires human- annotated captions, while our method does not.
2 Details of Modules
Our method is primarily composed of three modules: A- Representation Decoder, and Language Encoder. Below is a detailed explanation of Language En- coder.
A-Representation Encoder, Contrastive A-
We extract language feature h, from Tinst using Language Encoder. In this module, process Tinst with BERT [68] to obtain the feature corresponding the CLS token IERTT We also use the CLIP text encoder [8] and lext-embedding-adee ada-0002 [69] in parallel to extract the language features lcup and lada. respectively, from Pinst- Finally, we concatenate them to obtain the language feature
h: BERT CLIP. [dda]
3 Details of Experimental Setup
3.1 SP-RT-1 Dataset
As described in Section 4.1, constructed the SP-RT-1 dataset from the RT-1 dataset [19] for our task. The details are described below. We collected the first and last images of each episode. The dataset was preprocessed by modifying the instruction sentences, In the RT-1 dataset, 43.6% of the negative samples were incorrectly labeled as negative, despite the manipulator having successfully executed the manipulation. We replaced the instruction sentences for the incorrectly annotated sam- ples with alternative sentences that were randomly selected to create negative samples. This strategy was chosen instead of converting them to positive samples, because the original dataset contained fewer negative samples than positive samples, and converting negative samples to positive samples would further reduce the proportion of negative samples.
The SP-RT-1 dataset consisted of a total of 13,915 samples, with a vocabulary size of 49, a total word count of 78,790, and an average sentence length of 5.66. The dataset contains 10,000 positive samples and 3,915 negative samples. The SP-RT-1 dataset contained 11,915, 1,000, and 1,000 samples in the training, validation, and test sets, respectively. We used the training, validation, and test sets estimate parameters, tune hyperparameters, and evaluate models, respectively. We computed the accuracy the validation set every epoch. The performance on the test set was evaluated using the model that achieved the highest accuracy on the validation set.
Other related datasets and benchmarks. For multimodal language understanding tasks in robotics, various datasets and benchmarks are used both real-world [20, 70, 71] and simulation [72, 73, 74, 75] settings. Among them, the RT-1 dataset is the most relevant our target task of suc- cess prediction for object manipulation. Additionally, VLMbench [25] is standard benchmark for object manipulation tasks on tabletop. It provides natural language instructions, labels indicating the success or failure of each manipulation, and images captured from five camera views.
3.2 SP-HSR Benchmark
For a comprehensive evaluation, we validated the proposed method in physical environment using a mobile manipulator with zero-shot transfer settings (SP-HSR benchmark). The data was collected in the environment described Section 4.1. In this experiment, used subset of the YCB objects [76], which are standard objects for manipulation research. These selections were based on their suitability for grasping by the HSR end-effector.
14

---

In the experiment, we randomly selected up to four objects and arranged them on the table. Then, executable open-vocabulary instruction sentences were created and assigned to the episodes. The manipulations were performed by remote controlling the robot. The images of the scene before and after the manipulations were taken using the head-mounted camera of the robot. total, 112 episodes were collected, with 56 episodes for both positive and negative samples.
3.3 Implementation Details
Table 3 shows the experimental settings for the proposed method. Our model had approx- imately 64M trainable parameters and 7.25G multiply-add operations. We trained our model on GeForce RTX 4090 with 24 GB of GPU memory and an Intel Core 19- 13900KF with 64 GB of RAM. It took approximately 1.5 hours to train our model on the SP-RT-1 dataset. The inference time was approximately 1.6 ms/sample.
Optimizer Learning rate
Adam (31 = 0.9, 82 = 0.999)
1.0 x 10-6
Weight decay
1.0 x 10-1
Batch size Epoch
32
150
Table 3: Experimental settings for Contrastive X- Repformer.
For Narrative Representation Module in X-Representation Encoder, we used the following prompt to generate descriptions: "Give a clear, comprehensive and detailed description of the state of the objects shown in this image. For each object, mention their colors, sizes, shapes, how they placed (upright, etc.), position within the image and relative position to other objects. Begin with the phrase 'In the image,'. Only use information that can be gained from the image. Mention the objects that appear in the sentence string below. If the objects in the sentence string are not present in the image, mention that they are not present. Sentence string: 'instruction' Here, we inserted the instruction sentence for each episode into 'instruction'.
3.4 Baselines
For comparative experiments, five baseline methods were used. We used the following experimental settings for each baseline. For each multimodal large language model (MLLM)-based method: InstructBLIP [7], Gemini [6], GPT-4V [5], we tested more than ten prompts and adopted the one with the best results.
UNITER- baasel-baae/ [59]. We performed fine-tuning according to the hyperparameter settings de- scribed in [59].
InstructBLIP. InstructBLIP assumes a single im- age as the image input. Therefore, we concate- nated Thefore and Pafter as shown in Fig. 6, han- dling them as a single input image. The prompt used is as follows: "These two images show the robot executing the instruction 'instruction'. Based on them, please predict whether the robot has suc- cessfully completed the task and answer with 'suc- cess' or 'failure'"" " Here, we inserted the instruction sentence for each episode into 'instruction'. This ap- proach was applied similarly across all MLLM- -baeed model prompts.
Figure 6: An example of the image input to In- structBLIP. The left and right parts show the im- ages before and after manipulation, respectively.
Gemini. Gemini capable of handling multiple images as input [6]. Therefore, during inference, provided Dbeforee Eafter, and the following prompt input: "These images show the robot exe- cuting the instruction 'instruction'. The first image shows the scene before the object manipulation by the robot and the second image shows the scene after. Based on the two images and the in- struction, determine whether the robot has successfully completed the task and answer with 'true' or "false"." " When we used a few-shot prompt. the model was also provided with three positive and three negative samples from the training split of the SP-RT-1 dataset, along with the sample to be evaluated. The instruction-based prompt given Gemini was "These images show the robot exe- cuting an instruction. The first image shows the scene before the object manipulation by the robot and the second image shows the scene after. Based on the two images and the instruction, deter-
15

---

"move brown chip bag orange"
(ii) "move orange redbull can"
(iii) "place 7up upright"
(iv) "move blue chip bag near sponge"
(v) "move rxbar blueberry near paper bowl"
(vi) "move rxbar blueberry near redbull can"
Samples used for the prompt in the few-shot prompted foundation model methods. The instructions given are shown below the image pairs. (i)-(ii) positive samples, and (iv)-(vi) are negative samples.
Figure 7:
SR
Model
Freeze
AR
NR
Accuracy [%]
CLIP [8]
VIT [56] DINOv2 [58]
v
80.8
V
79.2
V
73.7
V
67.7
V
77.5
V
79.9
(vii)
V
77.1
(viii)
75.2
Table Quantitative results the experiments where parameters the backbone networks were unfrozen on the SP-RT-1 dataset. Bold indicates the accuracy with the highest value. In this table, freeze, SR, AR, and NR represent the freezing of the parameters in the backbone networks. Scene Representation, Align Representation, and Narrative Representation, respectively.
mine whether the robot has successfully completed the task and answer with only 'true' or 'false'"" The samples provided to the MILLMs are shown in Fig. 7 with its instruction. The samples were randomly selected.
GPT-4V. Similarly, GPT-4V can also process multiple images [5]. Thus, the experiments, we inputted #before. Fafter, and the following prompt: "These images, taken from a single viewpoint camera, show the robot executing the instruction 'instruction'. Based these images and the in- struction, please determine whether the robot has successfully completed the task and answer with 'true" or "false"." When using few-shot prompt, as with the prompt Gemini, we provided the model with the text prompt, three positive samples, and three negative samples. Here, the samples provided were the same as those given to Gemini. The instruction- based prompt given to GPT-4V was "Two images, taken from a single viewpoint camera, show the robot executing an instruction. Based on the images and the instruction, please determine whether the robot has successfully com- pleted the task and answer with 'true' or "false'."
Additional Ablation Study
4.1 Unfreezing Backhone Networks' Parameters
We conducted additional ablation studies where the parameters of the backbone networks were unfrozen. Table 4 shows the quantitative results. As shown in the table, the scores for unfreezing models on the RT-1 dataset were lower compared to the score for Model (i) where every backbone networks was used and frozen. On the other hand, when the backbone network was unfrozen, Model (vi) performed 0.7 points better than Model (ii). This indicates that in comparisons between models with unfrozen backbone network parameters, simpler architectures sometimes be more effective.
16

---

"move rxbar blueberry blue chip bag" A sample of Ambiguous Instruction. In this case, the given instruction was "move rxbar blue- berry near chip bag." ground truth label was false. The success or failure of the manipulation de- pends the definition of 'near.'
"open middle drawer"
Figure 8:
An example of sample in the Multi- modal Language Comprehension Error category. The instruction for this sample was "open middle drawer."
Figure 9:
However, it was shown that the proposed model, which freezes the backbone network parameters and utilizes all backbone networks, achieved the best performance.
4.2 Attention Mechanism
We used cross-attention instead of contrastive loss for before-after image differentiation, because cross-attention would better capture the differences required for task success prediction than other approaches such contrastive loss. While the contrastive loss is beneficial in determining if there is a difference between features, we hypothesized that it is difficult to perform task success prediction using contrastive loss. This is because difference between the images does not necessarily indicate task success. An example of a case where such model could struggle when there are slight object movements or non-target object moved. As matter of fact, the cross-attention mechanism is successfully applied image difference captioning tasks [50].
We conducted additional ablation study to investigate the contribution of the cross-attention operation in Contrastive X- Representation Decoder. Table presents the results.
Model
Attention Mechanism Accuracy [%]
ii
Self-Attention
78.88 + 1.05
Cross-Attention
80.80 + 0.86
Table 5:
Results of additional ablation study. Bold
In this experiment, attention operation to self-attention operation to investigate its contributions. From the table, it can be observed that the accuracy of Model (i) was 78.88%, which was 1.92 points lower than that of Model (ii). This indicates that the cross-attention operation is suitable for identifying the differences between images.
changed the cross-
indicates the highest value.
5 Error Analysis
The confusion matrix for Contrastive X-Repformer on the test set of the SP-RT-1 dataset includes 431, 114, 386, and 69 samples that are true positive, false positive, true negative, and false negative cases, respectively.
Thus, there
total of 183 samples where the proposed method failed the set of the SP-RT-1 dataset. Table shows the results of the error analysis, where we randomly selected 100 samples of failed cases. We classified them into the following six categories:
Multimodal Language Compre- hension Error: This refers to cases where the model incorrectly interpreted visual information and instruction sentences, such as mis- understanding the target object and misinterpretation of referring expressions.
Error type
#Errors
Multimodal Language Comprehension Error
63
Partial Visibility
14
Narrative Deficiency
11
Ambiguous Instruction
8
Erroneous Data Sample
Total
Table 6: Error analysis on failure cases.
Partial Visibility: This category in-
cludes cases where the target object area is only partially visible, making it difficult to make
17

---

appropriate predictions. This can occur when the target object is more than half occluded by the manipulator or other objects, or when more than half of the target object is outside the photographed scene.
Narrative Deficiency:
This addresses cases in which the narrative from the MLLM is missing.
This involves cases where interpretations of success or failure may vary depending on the criteria for success. Fig. 8 shows a sample included in this category. In this exam- ple, the instruction given was "move rxbar blueberry near blue chip bag." " As shown in the figure, the 'rxbar blueberry' moved closer to the 'blue chip bag' before and after the object manipulation. However, the ground truth label for this example was false. this case, the success or failure of the task depends on the definition of 'near.'
Ambiguous Instruction:
Erroneous Data Sample: This category covers cases where the input images of the sample are inadequate for the SPOM task, making it difficult to perform the task. For instance, a case where the instruction given "pick a green can" and the manipulator is already grasping a green can the Dbefore applies this category.
As shown in Table 6, the main bottleneck was the Multimodal Language Comprehension Error. This issue is mainly due to the fact that the MLLM in the Narrative Representation Module generated incorrect sentences that could directly affect the success of the SPOM task. Fig. 9 shows a sample categorized as a Multimodal Language Comprehension Error. The left and right image in Fig. 9 show Thefore and Pafter, respectively. The captions created by the MLLM for Whefore was "In the image, there is an open middle drawer on a metal table. Inside the drawer, there are two objects: sandwich and can of soda. The sandwich is upright, while the can of soda is on its side." The captions for Pafter was "In the image, there is an open middle drawer with robotic arm reaching into it. The robotic arm appears to be picking up something from the drawer. Additionally, there is a can of soda sitting on top of the drawer:" The former caption states that the middle drawer was already open before the manipulation. This makes it difficult for the model to make appropriate predictions based on the information.
This issue may be due the difficulty of designing prompts for large language models (LLMs). Despite experimenting with many prompts and selecting the best one, erroneous generations still occurred. Indeed, object hallucination is a known challenge in image captioning by LLMs [77]. Therefore, a possible solution could be to investigate prompt designs that reduce the likelihood of such errors. For example, instead of describing everything at once, several elements could defined advance and short responses could obtained for each of them.
Additional Qualitative Results
Figs. 10 and 11 provide additional success examples of Contrastive X-Repformer on the SP-RT-1 dataset and in the zero-shot transfer experiment, respectively. For the sample shown in Fig. 10 (iii), all bascline methods except InstructBLIP [7] made incorrect predictions. Likewise, for the sample displayed in Fig. 10 (ix), all baseline methods except UNITER-bact [59] made incorrect predic- tions. It was found that for episodes with only a subtle difference between the images before and after the manipulation, the baseline methods had difficulty in making accurate predictions, whereas Contrastive X- R-peforrerr was able to predict appropriately.
Furthermore, all MLLM-based methods except Gemini [6] made incorrect predictions for Fig. 10 (ii), and all MI..M-based methods made incorrect predictions for Fig. 11 (ii). This indicates that even MLLM-bas -based methods can struggle with referring expression comprehension and aligning im- ages with natural language. From the examples in Figs. 10 and 11, can be said that Contrastive X-Repformer performed successfully in scenarios involving complex relational and spatial instruc- tions, as well as in non-tabletop rearrangement settings. can also accurately identify failures when the changes in the target object do not match the changes specified in the instructions. Especially, Fig. 10 (iv) and Fig. 11 (x), (xi), (xii) show that Contrastive X-Repformer performed successfully in scenarios involving complex relational and spatial instructions.
18

---

(i) "pick apple from white bowl"
(ii) "move rxbar chocolate near apple"
(iii) "place 7up upright"
(iv) "pick green can from bottom drawer and place on counter"
(v) "pick 7up can from top shelf of fridge"
(vi) "close right fridge door"
(vii) "knock 7up can over"
(viii) "pick coke can from middle shelf of fridge"
(ix) "move green rice chip bag near sponge"
(x) "place water bottle into top drawer"
(xi) "move orange near green can" (xii) "mave coke can near 7up can"
Figure 10: Additional qualitative results on the SP-RT-1 dataset. In this figure, (i)-(vi) represent true positive cases, and (vii)-(xii) are true negative cases. These visualized in the similar manner to Fig.
Fig. 12 shows failed cases of the proposed method. Fig. 12 (i) and (ii) show the failed examples on the SP-RT-1 dataset, and Fig. 12 (iii) and (iv) exhibit the failed examples in the zero-shot transfer experiment.
Fig. 12 (i) shows an example with the instruction of "open middle drawer." The ground truth label for this example was success, because the robot opened the middle drawer. Nonetheless, our method predicted that the robot failed in carrying out the instruction. This error can be explained by the fact that most of the middle drawer lies outside the photographed area, making it hard even for humans to deduce correctly.
The instruction for the instance displayed Fig. 12 (ii) is "pick orange from white bowl" and the ground truth label was failure. This result is most likely because the bottom of the orange still touching the other oranges. Meanwhile, the baseline and proposed methods predicted success. This error arises from the ambiguity of the situation, where predictions would likely be divided even among humans.
Fig. 12 (iii) presents failed example in the zero-shot transfer experiment. In this example, the in- struction sentence was "move the mug near the spam can." This sample was labeled success, whereas Contrastive X-Repformer predicted this sample as failure. To predict appropriately, the model needs to appropriately understand both the 'mug' and the 'spam can.' particular, to understand 'spam,' approaches such optical character recognition required, which makes it challenging.
Finally, Fig. 12 (iv) exhibits failed case with the instruction of "move the apple close to the red can." Contrastive X-Repformer predicted that the manipulator succeeded in following the instruc- tion. while the ground truth label was failure. In this sample, there are three red objects: an apple, a red can, and red mug. The manipulator brought the apple close the red mug. Therefore, possible that the model judged the success of the manipulation based solely the characteristic of being 'red.'
19

---

"place the red pringles next to the yellow bottle"
(ii) "move the ycllow bottle close to spam can"
(iii) "pick the red tomato can"
(iv) "place spam can on the front
(v) "pick the rubik's cube in front the banana" "
(vi) "move the red cup to the back
left"
area"
(vii) "place mug next the apple"
(viii) "move the red mug next to the rubik's cube"
(ix) "pick the yellow French's
bottle"
(x) "place purple cup the front (xi) "move the yellow bottle to the (xii) "move the chips can near the orange"
left"
back right"
Figure 1I:
Successful examples of Contrastive X-Repformer in the zero-shot transfer experiments. In this figure, examples (i)-(vi) show true positive cases, and (vii)-(xii) depict true negative cases. The examples visualized in the same manner Fig. 8.
ii) "open middle drawer"
(ii) "pick orange from white bowl"
(iv) "move the apple close to the red can" Figure 12: Failed cases of the proposed method. These visualized in the same manner as Fig. 8.
(iii) "move the mug near the spam can"
Human Errors in Subject Experiment
Fig. 13 depicts examples where the human predictions were incorrect. In Fig. 13 (i), the instruction sentence for this sample "pick 7up from bottom shelf of fridge." Although the ground truth for this sample was success, the human prediction was failure. In this example, it is difficult to identify the label of the can that the manipulator grasped, as well as to determine where the can was retrieved from.
In Fig. 13 (ii), "pick the red mug" was the instruction. this example, the mug was successfully grasped by the manipulator. However, the mug was mostly occluded, making it difficult to judge. As shown in the example, the SPOM task can be difficult even for humans.
20

---

(ii) "pick the red mug" Figure 13: Samples of human errors. These are visualized in the same way in Fig. 8.
ii "pick 7up can from bottom shelf of fridge"
(+)= 0 (ii)=1
(i
tiv)
(-)=4
(vi) 5
(vii) f = 6
(viii) t = 7
t-
(x)==9
( 10
( 11
(xiii) 12
(xiv) f 13
(xv) 14
(xvi) 15
Figure 14: Successful example of video classification problem using Contrastive X-Repformer. The in- struction "pick green rice chip bag." The images frames to 15, indicated by the numbers in the image. These frames are an episode in the SP-RT- dataset. The instruction given to the manipulator was "pick green rice chip bag," and the ground truth label was 'success.'
8 Application Video Classification Problem
We applied Contrastive A-Repformer to the video classification problem. While the method only uses two images to perform the SPOM task, is possible to perform video classification using it. The problem can be solved by predicting the success failure of object manipulation each time for the 0, N). Here, (t = 0, n) represents an image pair consisting of frames at times f = 0 and = n. In this approach, video classification can be done by making predictions based on whether the proposed method outputs 'Success' at any point or continues to output 'Failure' until the end.
input image pairs, as follows: (t 0, 1), (t 0,t 2), ..... (t 0, N N-1),
Fig. 14 shows a successful sample. In this sample, the instruction and ground truth label were "pick green rice chip bag" and success, respectively. The example contained 16 frames, with the success state changing at 14. The proposed method was able to detect this change appropriately. This
21

---

indicates that Contrastive X-Repformer can also solve video classification problems. An advantage of this method is its ability to perform success prediction in real-time, unlike methods which require video input.
22


Reviews:
[{"review_id": "yrVx3Nt8Iu", "reviewer": "Reviewer_yUdH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a model called Contrastive lambda-Repformer, which curates aggregate features from various VLMs and LLMs and fuses them in a novel multi-level architecture to predict task success given a pair of before-after pictures and task in natural language. The experiments find that this model, trained on subset of RT-1 dataset, does better than contemporary MLLMs on predicting task success for SP-RT-1 test dataset. Qualitatively, in a zero-shot setting, we see similar improvements for this model against MLLMs. The paper also contributes refined SP-RT-1 dataset as part of the SPOM problem it formulates.", "summary_of_recommendation": "I find this paper well-written, with well founded and articulated contributions. It is sufficiently rigorous in its experiment to motivate my vote of acceptance. The paper can benefit from going deeper into this topic and reviewer has outlined potential improvements that they would like to see. While I have concern about general applicability of this exact same setup, I find the scalable nature of methodology and results convincing to warrant sharing with the community and critical debate to push along this problem axis.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- Why did you choose ViT, DINOv2 and CLIP specifically?\n- Why use intermediate CLIP features for Scene Representation vector while last layer for Aligned Representation?\n- Why use cross-attention instead of contrastive loss for before-after image differentiation? Could you please outline your motivation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CgEXlZl1es", "reviewer": "Reviewer_cPTW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a method for performing success/failure detection in robotic manipulation tasks, by ensembling several representations from multimodal foundation models.", "summary_of_recommendation": "Overall, the proposed idea and methodology seem sound, but I don't have enough info to contextualize how this will work in a real-world setting and how it will compare to other more compelling baselines.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "- Can you show some more qualitative examples of what correctly detected failed vs. successful states look like in these benchmarks? How did you get ground truth success/failure labels?\n- Can you show some demonstration in the real world where you use this approach either for detecting & recovering from failures, or use the success/fail detector in some other way (i.e. sparse reward RL)?\n- Is it possible to compare against some baseline that is not just a few-shot VLM?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tri4kkMUZ8", "reviewer": "Reviewer_ELhE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method to do success detection for manipulation tasks by crafting an ensemble of vision and language models.", "summary_of_recommendation": "More experiments could be done to improve the success detection performance as well as to compare against a more competitive baseline (by using in-context learning to steer the foundation models).", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "- Why was the ViT/CLIP frozen? Could you do an experiment with only the Scene Representation, but allow gradients to flow back into the ViT? Effectively, your scene representation is frozen, but you could actually tune the scene representation to be more aware of robotics scenarios.\n- It's actually quite heavyweight to have an ensemble of 5 models (ViT, DINOv2, CLIP, InstructBLIP, BERT). It would be nice to take a step back and see if a simpler architecture could implicitly capture the scene representation, aligned representation, and narrative representation jointly in the weights of the backbone.\n- Could you add a baseline using some few-shot prompted foundation models (GPT, Gemini) to see if that can lift performance on the SPOM task? Would be nice to have an in-context learning baseline.\n- Would it be possible to add ablations with only SR, AR, and NR? It would be interesting to understand which representation alone yields the highest accuracy.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- Well written
- Reproducible methodology and useful ablations

Weaknesses:
- Narrow evaluation:
	- How do the failures discussed generalize to real world scenarios 
	- How can the approach be integrated into a closed-loop recovery 
	- Strong (general) baselines considered but not directly in domain
		- Comparison beyond zero or few-shot VLMs
- Writing:
	- Motivation of backbone
		- Choice/comparison of visual backbone
		- Cross-attention vs contrastive loss 
		- Expensive ensemble
		- Decisions on what to freeze
	- Additional qualitative results and recovery

------
Updated:
Thank you to the authors for answering all the raised questions and providing additional results/updates. I appreciate the incorporation of many of these points into the manuscript which clearly explain these decisions, results, and (in some cases) references to the appendix.  The resulting paper will be substantially more impactful than the submission.  There are a few points not included and the new document is longer so please take care when editing not to weaken the presentation.
Decision: Accept