Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation
Authors: Zhiwei Zhang
Yuliang Liu
Venue: Accepted by TMLR ()

Abstract: The recent success of ChatGPT and GPT-4 has drawn widespread attention to multimodal dialogue systems. However, there is a lack of datasets in the academic community that can effectively evaluate the multimodal generation capabilities of Visual Language Models (VLMs) in textual-visual chat tasks. In this paper, we address this gap by introducing two novel multimodal datasets: the synthetic CLEVR-ATVC dataset (620K) and the manually pictured Fruit-ATVC dataset (50K). These datasets incorporate both visual and text-based inputs and outputs. Furthermore, to facilitate the accountability of multimodal systems in rejecting human requests, similar to language-based ChatGPT conversations, we introduce specific rules as supervisory signals within the datasets. This allows the trained VLM to provide a yes or no answer after engaging in visual and textual reasoning, accompanied by a language explanation to clarify the reasons behind the inability to execute the given human instruction. Our proposed method involves a two-stage training procedure, which includes training the image auto-encoder and the auto-regressive transformer from scratch. The first stage employs a discrete variational autoencoder (dVAE) to compress each image into concise tokens, which are then combined with text tokens into a single data stream. This stream is subsequently fed into the decoder-based transformer to generate visual re-creations and textual feedback in the second stage. We conduct comprehensive analyses of experimental results, focusing on re-created image quality, answer accuracy, and the model's behavior when faced with uncertainty and imperfect user queries. Through our explorations and findings, we aim to contribute valuable insights into the accountability of textual-visual generative models.

Published in Transactions on Machine Learning Research (02/2024)
Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation
Zhiwei Zhang
bitzaw@gmail. com
The Chinese University of Hong Kong Centre for Perceptual and Inteructive Intelligence
Yuliang Liu
glliahhust edu.ccm
The Chinese University of Hong Kong Huazhong University of Science and Technology Centre for Perceptual and Interoctive Intelligncee
Reviewed on OpenReview: https: // openevvee......... net/ forum? id= kQm=1BMIY
Abstract
The recent success of ChatGPT and GPT-4 has drawn widespread attention to multimodal dialogue systems. However, there is a lack of datasets in the academic community that can effectively evaluate the multimodal generation capabilities of Visual Language Models (VLMs) in textual-visual chat tasks. In this paper, we address this gap by introducing two novel multimodal datasets: the synthetic CLEVR-ATVC dataset (620K) and the manually pictured Fruit-ATVC dataset (50K). These datasets incorporate both visual and text-based inputs and outputs. Furthermore, to facilitate the accountability of multimodal systems in rejecting human requests, similar to language-basee based ChatGPT conversations, we introduce specific rules as supervisory signals within the datasets. This allows the trained VLM to provide a yes or no answer after engaging in visual and textual reasoming, accompanied by a language explanation to clarify the reasons behind the inability to execute the given human instruction. Our proposed method involves a two-stage training procedure, which includes training the image auto-encoder and the auto-regressive transformer from scratch. The first stage employs discrete variational autoencoder (dVAE) to compress each image into concise tokens, which are then combined with text tokens into a single data stream. This stream is subsequently fed into the decoder-based transformer to generate visual re- creations and textual foedback in the second stage. We conduct comprchensive analysos of experimental results, locusing on re-created image quality, auswer accuracy, and the model's behavior when faced with uncertainty and imperfect user queries. Through our explorations and findings, we aim contribute valuable insights into the accountability of textual-visual generative models. Datasct and code are available at httpss/mmatiixxlllaagghhbb.b
1 Introduction
Recently, the most important breakthrough was made by ChatGPT (OpenAl 2023a) and GPT-4 (OpenAl, 2023b), which unveiled the emerging potential of the conversation between human and artificial intelligence system. Chat serves as chatbot. that operates with language both input and output, while GPT- 4 is multimodal model capable of accepting both image and text inputs and producing text outputs. A successful multimodal generative model should excel in both textual and visual reasoning, generating high-quality text and image feedback. Visual ChatGPT (Chenfei Wu & Duan 2023) is a pioncering work that combines ChatGPT with a series of pre-trained visual foundation models, enabling text-image chat.. Another relevant work, FROMAGe (Jing Yu Koh, 2023), also involves image-text inputs and outputs for
Coorespooning author: Vuliang Liu.

---

Published in Transactions on Machine Learning Research (02/2024)
multimodal dialogue, with the fine-tuning of linear layers and the freezing of pre-trained LLM. However, existing datasets lack definitive ground truths to effectively measure the quality of text and images generated in multimodal dialogue systems. Therefore, there is an urgent need for a dataset that can evaluate the performance of multimodal generative models. In this paper, we aim to address this need by constructing new multimodal datasels that require models to output high-quality images and provide textual feedback while accepting a text-image pair. We introduce the synthetic CLEVR-ATVC dataset (620K), created by rendering images using approxinately 200 GPU days, and the real Fruit-ATVC dataset (50K), manually captured and annotated.
Another significant contribution of this paper pertains to the issue of responsibility in text-to-image gen- eration models, specifically the need for models to learn how to reject human instructions. Previous works (Doshi- Velez et al. 2017 Loi & Spielkamp 2021 have highlighted the importance of accountabil- ity in AT systems, including the ability to hold decision-makers accountable for adhering to procedural and substantive standards. The responsible management and deployment of AI systems is a crucial and often overlooked topie. GhatGPT (OpenAT 2023a implements a similar requirement. at deployment time by employing rules defined by OpenAl to restrict certain requests. However, to the best of our knowledge, we are the first to consider the issuc of responsibility in textual-visual dialogue models. Previous text-to- image generation methods {Ramesh et al. 2021a Nichol et al. 2021 Wu et al. 2021 Ramesh et al. 2022: Saharia et al. 2022 Ding et al.] 2021) have produced random and uncontrolled outcomes lacking human oversight. While some works have explored the controllability of image generation by focusing on object categories or attributes (e.g.. position, pose, color) (Li et al.] 2019 Niemeyer & Geiger 2021 Jiang et al. 2022 Patashnik et al. 2021 ; such as StyleCTIIP (Patashnik et al. 2021), which manipulates visual image attributes using text, the rejection of human instructions and providing explanations for such decisions have not been adequately addressed in the textual-visual dialogue context. Therefore, in this paper, propose that the machine should be capable of rejecting human instructions and providing explanations, particularly for instructions that cannot be executed or are prohibited.
To serve the aforementioned research goals, we introduce a novel lask called accountable lext-based visual re-creation, which explores whether textual-visual chat models can learn to reject human instructions. This task requires the model to generate both visual re-creations and language-basod feedback while accepting text-image pair. Our datasets include rules supervision signals to teach the model to reject certain instructions. As illustrated in Figure D when a chatbot receives instructions from human, it responds with a "no" to forbidden or unfeasible instructions, accompanied by an explanation. An instruction may be deemed unfeasible if the corresponding objects mentioned in the text- query are absent in the visual input. Prohibited inst ructions are determined manually, taking into account the laws of physics. Prohibited actions encompass both actions that are feasible but not allowed, as well as actions that are simply not possible. In the case of prohibited instructions that cannot be executed, the model should provide additional explanations to clarify the reasons behind their infeasibility. These instructions are elaborated and listed in Table 1 Our proposed method, as depicted in Figure 1 employs a multimodal generative model comprising an image auto-cncoder and an auto-regrossive transformer. We propose a two-stage training procedure to train these components from scratch. The first stage involves a discrete variational auloencoder (dVAE) that compresses each image into concise tokens, which are then concatenated with text tokens and fed into the dasoder-based transformer responsible for generating the visual re-creations and textual foodback.
Additionally, we provide comprehensive analysis of our experimental results, including assessments of re- created image quality, answer accuracy, and model's behavior when faced with uncertainty and imperfect user queries. Wc also compare two different image auto-cncoders, VQVAE (Oord et al. 2017) and VQGAN (Esser et al. 2021), and analyze the reasons behind the subpar performance of VQGAN. All the datasets used in this study are publicly available, and we will release the source code for our annotation tool, evaluation tool, implementation of baseline models, metric calculations, and detailed instructions.
To summarize, the main contributions of this paper are as follows:
We construct two multimodal datasets, CLEVR-ATVC (620K) and Fruit-ATVC (50K), which in- corporate textual-visual inputs and outputs to evaluate the performance of multimodal generative models.
2

---

Published in Transactions on Machine Learning Research (02/2024)
ATV Charbot
Please put the pitaya in the plate.
Plouse put pinasa planc
Pre-trained Codebook
No problem.
Text Tokenizer
Ploose exchange position of pear
Recause post.
2,60)
Nearest Neighbours
3y00)
Encoder
Please put the lange bnows menal cylinder on top the longe blze rubbber cube.
Ne problem.
Please pur sphere under cylinder.
This oction forskiden. Recanse undes
No problem.
Autoregressive Transformer
Textual-Visura Feedback
Figure 1: The left figure shows that multimodal generative model can simultancously recreate visual input and give textual feedback when faced with human instructions, especially it reject commands. The right figure illustrates the overall framework of our method. The model is required to generate re-created image (M) and textual feedback (A) conditioned on the visual input (V) and text-based user query (T), and the language-bard based explanation is also given for those instructions that cannot be executed and the prohibited instructions.
We consider the issue of accountability in multimodal generative models by embedding pre-set rules as supervised signals in our datasets. This enables the VLMs to learn to reject human instructions in multimodal conversations.
We propose a two-stage training procedure for training the image aulo-encoder and aulo-regressive transformer, aiming enable the models learn how to reject human instructions. All of our models trained from scratch, and their training took 350~900 GPU days our constructed datasets.
We provide extensive qualitative and quantitative results, evaluating the quality of generated images, the accuracy of answers, and the model's ability to handle uncertainty and incomplete queries.
The remainder of this paper is organized as follows. Section presents detailed information about our datasets. In Section 3 we introduce our models and training process. We then present extensive experiments and detailed analysis in Section to validate our proposed method. Section provides an overview of related work, and Section 0 concludes the paper. Section pertains to acknowledgments. Additional details of our datasets are provided in the appendix.
2 Datasets
In this section, we present the distinctive aspects of our datasets compared to existing ones. We then describe the data collection process. Additionally, we provide an overview of the information contained the annotation files for our datasets, with more detailed information available in the appendix. Finally, we summarize the dataset statistios in the tables.
3

---

Published in Transactions on Machine Learning Research (02/2024)
2.1 Definition of ATVC
Our proposed dataset and task introduce two key innovations compared to previous methods. Firstly, our multimodal dataset comprises image-text inputs and outputs, which facilitates the development of more effective multimodal generative models. Secondly, our datasets incorporate predefined rules that teach the model to reject certain human instructions, even if they are technically feasible. This gives rise to the concept of accountable text-based visual re-creation (ATVC) task, which can be defined as follows:
Given a visual input (V) and a text-based query (T), the multimodal generative model is expected to produce a re-created image (M) and provide a language-based explanation (A) for its decisions. A successful ATVC model should possess the ability comprehend the user's query and employ language cues reason about the visual imput, while also generating appropriate textual feedback and re-created visual results.
2.2 Data Collection
In order to validate the proposal of this paper, we conduct experiments on two distinct datasets. The first dataset, CLEVR-ATVC,
large-scale synthetic dataset specifically designed to comprehensively evaluate the proposed task. On the other hand, the Fruit-ATVC dataset is utilized evaluate our method on real- world scenarios. In the following scctions, we provide detailed description of the visual image collection process for both datasets.
CLEVR-ATVC: We utilize the default environmental settings, including object settings and rendering options, from CLEVR (Liu et al., 2019) to generate synthetic images. The rendered images have a size of 256, with each image containing 3 to 6 objects. In addition, we apply a desiged algorithmic filter to exclude scenes with objects located on the image border before rendering. This ensures that all objects in both the original and re-created images are within the boundaries, facilitating the evaluation of experimental results. The synthetic CLEVR-ATVC dataset is generated over approximately 200 GPU days.
This dataset consists of real-world scenarios where all the images are captured mamually by the authors using mobile phones and tripods. The images are taken with various mobile phone models, including iPhone, Huawei, Xiaomi, and others. For the Fruit-ATVC dataset, five researchers contributed to capturing the visual inputs and re-created images. Additionally, we incorporated 12 different scenes to enhance the diversity of the datasct. The diversity of collection scenes illustratod in the figures provided in the appendix.
Fruit-ATVC:
2.3 Generation of Textual Query and Feedback
For the CLEVR-ATVC dataset, we employ an automated program that simultancously generates visual inputs, text-based queries, re-created images, and textual feedbacks. This program saves the visual inputs and re-created images in their respective folders, while the text-based queries, textual feedbacks, and other data information are automatically stored in an amnotation file. Each visual input is associated with ten different text-based queries and their corresponding textual feedbacks. Among these queries, six instructions can be executed by the AT system, resulting in appropriate image re-creations, while two instructions cannot be executed due to the presence of objects mentioned in the text-based queries that are not present in the visual input. Additionally, two instructions are deliberately prohibited by mamual specification. The number of each instruction is approximately the above ratio, because some visual inputs cannot generate prohibited instructions.
The process of generating text- queries and textual feedbacks for the Fruit-ATVC dataset differs from the aforementioned process. Since the visual inputs and re-created images are manually captured, and the text-based queries and textual feedbacks are generated semi-automatically using our designed labeling tool. Initially, we collect and organize the visual inputs and re-created images according to predefined rules, ensuring the inclusion executable and non-executable queries for each visual input. Then, we utilize an annotation interface (illustrated Figure to assist in generating text-based queries and textual foodbacks, which are autonatically saved an annotation file. The specific process is as follows: we select image pairs consisting of the visual input and the re-created image for executable instructions. These image pairs are
4

---

Published in Transactions on Machine Learning Research (02/2024)
2000 Inated
8119
6837
8512
8085
7954
6207
9263
8082
Barn:: Eupple 2 2 wesage hasau Flank Fansag Tannssia B lessss Pylkee
Bard: agline Sartalag craset I-ientaurtittg shorteearraaaag ssstattng glant
2500
5000
7500
10000
Apple
Coconut Avocado
Orange
Banana Pitaya
Mango
Mention frequency
Annotation interface
Figure 2: The mention frequency of different fruits and the annotation interface for Fruit-ATVC dataset.
Table The detailed information of textual feedback.
Textual Feedback
Answer Type
Reason for prohibition
Explanation
1) No problem.
2) Cannot be done.
no mentioned objects no mentioned objects
3) Forbidden.
cannot put certain object under an object cannot put object top of certain object
4) Forbidden.
mentioned objects
Annotation
Text-based Query (T)
Visual Input (V)
Re-creation (M)
Textual Feedback (A)
idx
question
filename
objects
idx
filename
idx
answer type
explanation
Figure 3: The main format visualization of data annotation file. The details can be found in the appendix.
then imported automatically through the interface depicted in Figure 2 where the object ID, container ID, and query type are manually added to the designated fields on the interface ("fruit_ ids", "Ct__ds" and "Query", respectively). The "fruit_jdd ids" field includes all the fruit types depicted in the image, while the "Query" field filled with the object and container calegories involved in the operation. For instance, "Ob" represents an apple and a bowl. Finally, by pressing the "ERR" key, text queries and feedback are automatically generated.
2.4 Data Annotation
In this section, we present the structure of our data annotation file, which follows the format of the COCO dataset (Lin al. 2014). The main format, depicted in Figure 3 consists of four parts: text-based query (T), visual input (V), re-created image (M), and textual feedback (A). Since each visual input corresponds to multiple text- queries, we utilize an "idx" value to establish onc-to-one mapping between T, M, and A. The "objects" subkey enumerates the objects present in the visual input. The "explanation" provides a rationale for prohibition and explains the absence of certain objects. Table provides further details on the textual foedback. There are three types of answers: a) no problem; b) the action cannot be performed; and c) the action is prohibited. Two specific instructions are prohibited: placing a certain object under another object and placing object top of certain object. The prohibited actions include those that can be
5

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: The summary information of CLEVR-ATVC dataset.
CLEVR-ATVC
www.
Size
Train Pairs
Test size
Textual-Vise Visual Feedback
Data type
68 GB
619,741
5,000
Yes
Syuthetic
Please <actionl>>
{size color material] [object] <actionl> {size color material) [object].
Please <action2> of (size color material) [object] and {size color material)
[object].
Actionl
Action2
Size
Color
Material
Object
put on top
exchange color
big
red, green, blue, cyan metal
cylinder
put under
exchange position
small
purple, brown, yellow
rubber
cube, sphere
Table 3:
The summary information of Fruit-ATVC dataset.
Fruit-ATVC
Size
Train Pairs
Test size
Textual- Visual Feedback
Data type Real
168 GB
27,503
1,872
Yes
Please <actionl> [object].
Please <action2> of [object] and [object].
Please <action3> [object] <action3> [container].
Actionl
Action2
Action3
Object
Container plate, bottle, etc.
remove
exchange position
put in apple, coconut, lemon etc.
performed and those that cannot be performed. For prohibited instructions that cannot be executed, the model needs lo provide further explanations. Please refer to the example given in Table A In addition, the results in Table show that we conducted statistics on accuracy of answer type and accuracy of explanation in textual feedback.
2.5 Data Statistics
Table and Table present the summarized information of our constructed datasets.
CLEVR-ATVC consists of training set with total of 620,000 pairs and testing set with 500 visual inputs, each accompanied by 10 queries, resulting in total of 5,000 pairs. This dataset incorporates four action types: "exchange color", "exchange position", "put under", and "put on top". The rendered images encompass three object shapes, seven object colors, two object materials, and other attributes, leading to hundreds of millions possible combinations. The text-based queries follow two different templates: the first template is usod for "put on top" and "put under" actions, while the second template employed for "exchange color" and "exchange position" actions.
Fruit-ATVC comprises 27,503 training pairs and 1,872 pairs in the testing set. This dataset involves ten different types of fruits, namely applc, pear, coconut, orange, banana, kiwi, mango, avocado, lemon, and pitaya. The frequency of mentions for each fruit is depicted in Figure 2 Additionally, the dataset includes 14 types of containers. Regarding the queries, there are three types of actions: "put a fruit a container", "exchange positions of the fruits", and "remooe the container".
3 Method
In our method, the generative model includes an image auto-encoder and an auto-regressive transformer Vassann et. al. 2017) to predict tokens of re-created image and textual feedback. The framework of our

---

Published in Transactions on Machine Learning Research (02/2024)
method is shown in Figure 1 Our method is two-stage training procedure, similar to Ramesh et al. (02Taa).
Image Auto-encoder This stage forces the model to learn the sequential discretized latent space of high dimensional images to utilize the expressive transformer architecture, é.e., learning a codebook to encode the visual input. Given an image of resolution 256 x 256, quantization model encodes it into 32 x 1024 discretized latent codes where the codebook size is 8192. We have tried two alternatives, including Vector Quantized Variational Autoencoder (VQVAE) (Oord et all 2017) and VQGAN (Esscr et al., 2021p. As illustrated in Figure ! the VQVAE can learn latent space matrix with discrete learnable variables, which is trained end-to-end via straight-through estimation. The codebook and the VQVAE model can be trained end-to-end using the following objective:
LvQ(G, F) = ssgg [F()) e112 + B [|ggle] - E(
Loodebook
Coommie
(1)
+ ||U G(e)ll
Lree
where u is the visual input, is the embedding for lcarning the codebook-indices C is the Icarned codebook, E is the image encoder, G(e) is the decoder for reconstruction. sg is the stop-gradient operation, and B is a hyper-parameter to balance the commitment loss (Oord al.] 2017]. The quantized codebook index is determined by looking up the nearest codebook vector of input features E(t) in terms of the Euclidean distance. The VQGAN (Esser et al.) 2021) is variant of VQVAE, which adds perceptual loss and an adversarial loss produced by patch- patch-based discriminator. It can achieve high resolution while significantly roducing the sequence length. The complete objective for the compression model Q" is:
Q' arg min max En-p(r) Evon)))) (Vv(G,,C,))) G.C.E
(2)
+ACGAN(IG, D),
where LGAN is to learn the differences between real and reconstructed images:
LGAN([G, F.), D) = [log ((v) + log(1 D(6))]-
(3)
Auto-rogrossive Transformer Based on the first stage, the images can be highly compressod by the codeeook-innices of their embeddings. In this state, the image encoder E is fixed. Therefore, the V, T, M, and A can be represented by the embedding S of the sequence of tokens. We adopt the axial positional embedding IP to process the codebook-indices generating by the image reconstruction module R, which is practically effective for multi-dimensional data. In our implementation, the sequence Tseg of the transformer is sequentially formed by T, V, M, and A, which is defined as follows:
T Tseg (OnC))()(O(T),
which focuses to generate the re-crcated result and textual feodback. O represents the tokenize operation. The transformer is designed as decoder-only model, which same Child et al. (2019). In our architecture, text tokens have the ability to attend to each image token in any of the self-attention layers. The model employs two types of self-attention masks. The attention masks used for text-to-text attention follow the conventional causal mask. On the other hand, for image-to-image attention, the model applics either a row, column, or convolutional attention mask in different self-attention layer, as explained in Child et al. (2019) Ramesh et al. (2021a). During training, each token is asked to predict the distribution of the next token, which formulates as an autoregressive next-index prodiction. Therefore, it equivalent to maximize the log-likelihood of the data representations.
Croanforrer Ex= log
(5)
where p is the full representation of the likelihood of the possible next indices. Note that for the "cannot" and "forbidden" pairs, as there is no re-created ground truth, the loss of the its prediction will not be back- propagated. We have tried predict black image or the original visual input for these two cases, but both of them perform much worse than simply ignoring their loss. In the testing procedure, we only need to input the T and V for wixe iterative generation until the required length is achieved.
7

---

Published in Transactions on Machine Learning Research (02/2024)
4 Experiment
In this section, we introduce the experimental details, evaluation metrics, and analyse on the quantitative and qualitative results.
4.1 Experimental Details
All experiments are all conducted with Pytorch 1.8.1. The model parameters are updated through Adam (Diederik P. Kingma, 2014) with 6 = 0.9, B2 = 0.999. We first trains the image reconstruction module for 200 epochs with an initial learning rate of 0.001 and a decay rate of 0.99 per epoch. This stage takes about half day to train for 200 epochs. The number of attention heads, the attention key and value dimensions, the number of layers, and the dimensions of the models are set to 8, 64, 4, and 512, respectively, The text tokenizer operates 49,408 vocab size (Sennrich R & A 2015). The text sequence is bracketed with [SOS] and [EOS] tokens. The maximum length of the text sequence is set to 64, and the output length of the image codebook is 1024, which results in a total 2112 sequence lengths for the transformer model. The second stage is distributively trained over 200 cpochs with a fixed lcarning rate of 0.0003. Unless specified, all the paramcters of the architecture for the following experiments will remain the same. The model is trained for approximate 900 GPU days on CLEVR-ATVC dataset and 350 GPU days on Frui dataset.
lower-cased byte pair encoding (BPE) representation of the text with a
4.2 Evaluation Metrics
We evaluate the proposed method using the following two types of metrics.
Image Quality Metric. The first two are the Peak signal-to-noise ratio (PSNR) and the structural similar- ity index measure (SSIM), which are commonly used to measure the similarity and quantify reconstruction quality of images. We also adopt the FSIM metric Zhang et al. (2011), which could be more suitable for the human visual system (HVS) using phase congruency (PC) and gradient magnitude (GM).
Human Evaluation. Following the previous methods (Lee et al., 2020; Ding et al. 2021 Dong et al. 2017 Zhang et al. 2017; 2018 2020 Kayser et al.] 2021), we also adopt Human Rank (HR) to precisely reveal the performance. HR can be used to evaluate whether the synthesized image conforms to subjective effects (authenticity, matching degree with text, etc.). We set three different classes for the HR: a) if the action is perfectly done, the result would be ranked "A", representing score 1; b) if the action is correct, but other parts are affected, e.g., mistakenly change the color or erasing other irrelevant objects, the result would be ranked "B", representing score 0.5; c) the action is incorrect, the result is "C", representing score 0. Therefore, the final HR score is between and 1, and the Full-Match score (FM-Score) represents that both the re-creation and the answer are correct. The higher scores of all metries, the better performance is represented. The "accountable" part is totally automatic, for which we use the full string matching to measure the accuracy between ground truth answers and the final predictions. We also take into account that the textual foedback in the explanation part will change, such as that both explanations "There is no objectl and object2" and "There is no object2 and objectl" are correct. We also designed tools to help with human evaluation, and we provide its interface and evaluation criteria in the appendix. In addition, we assign 100 sets of results 5 works for scoring, we empirically find that our task usually has an explicit answer which is not casily affected by human subjective opinions.
4.3 Results and Analyses
In this section, we ovaluate the proposed method on the CLEVR-ATVC and Fruit-ATVC datascts, whose results include the image re-creation, textual feedback, qualitative visualizations and the model behavior when facing uncertainty.
4.3.1 Results on CLVV
For the CTVV dataset, the re-created results are shown in Table 5
We can see that the machine achieves FM-Score 52.6% (re-creation and textual leedback are completely correct), which also shows the
8

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: Qualitative results of CLEVR-ATVO dataset.
Visual (V)
Text (T)
Re-created (M)
Answer (A)
FM-Score
Please exchange the positions of the large brown metal cylinder and the large blue rubber cube.
No problem.
1.0
Please put the large brown metal cylinder top of the large blue rubber cube.
No problem.
0.75
Please put the small gray rubber cylinder on top the large yellow metal cube.
This action cannot done. Because there no large yellow metal cube.
1.0
This action is forbidden. Because you camnol pul the sphere under an object, and there no small gray metal sphere and small purple rubber cylinder.
Please put the small gray metal sphere under the small pur- ple rubber cylinder.
1.0
Table Image re-creation evaluation on the CLEVR-ATVC dataset.
Human Rank (%)
PSNR
SSIM
FSIM
A
B
C
Score
FM-Score
55.0
0.9944
0.669
25.9
16.2
58.0
39.0
52.6
Table 6: Texual feedback evaluation on the CLEVR-ATVC dataset. Type Acc. represents that the answer can correctly recognize the query that cannot be done or forbidden. Exp Acc. represents the explanation is correct in both answer type and the reasons.
Can
Cannot Type Acc.
Forbidden
Score (%)
Num
Aoc.
Num
Exp Acc.
Num
Type Acc.
Exp Acc.
1662
71.6%
1997
49.4%
46.5%
1341
99.7%
58.2%
66.2
difficulty of the lask. The high SSIM 0.9944 represents the model can obtain high image reconstruction quality. Based our observation, find that the "exchange color" queries are mostly correct, while "exchange positions" queries are somewhat limited by the image reconstruction.
The quantitative results of lextual feedback are shown in Table 0 We can see that there are 1662 pairs for the results on "can" queries; among them, 71.6% pairs are answered correctly. For the queries about "cannot", 49.4% pairs can be accuratoly recognized that the query is not doable, and 46.5% pairs can not only answer the type but correctly answer the reasons. For the queries about "forbidden", 99.7% can be correctly recognized, because the model only needs to complete textual reasoning. However, task requires

---

Published in Transactions on Machine Learning Research (02/2024)
Table 7: Qualitative results the Fruit-ATVC dataset.
Visual (V)
Text (T)
Re-created (M)
Answer (A)
FM-Score
PRED-MM
Please plate.
remove
the
No problem.
1.0
Please put the lemon in the bowl.
No problem.
0.75
Please exchange the position of the ba- nana and the co- conut.
This action cannot be done. Becanse there is no coconut.
1.0
This action cannot be done. Because there is no kiwi and no bottle.
Please remove the bottle.
0.75
Table 8: Image re-creation evaluation the Fruit-ATVC dataset.
Human Rank (%)
PSNR
SSIM
FSIM
A
B
C
Score
FM-Score
44.1
0.9272
0.420
12.8
29.4
57.9
27.5
46.1
the model further explain why the instruction cannot be executed, which requires the model to complete both visual and textual reasoning. Therefore, only 58.2% correctly explain the reasons. The score the weighted average of the above results. As there numerous possible answers, random guessing result is less than 1%, and thus it should be safe to conclude that the proposed method has shown impressive results.
Example qualitative results are shown in Table We can conclude that the model can cay "No" to instruc- tions and give accurate languago-based explanations. In addition, it is worth mentioning that the model also implicitly learns the depth of the image and size of the objects, and thus knows whether the object is occluded and how much it is occluded when you manipulate some objects.
4.3.2 Results on Fruit-ATVC
For the Fruit-ATVC dataset, the re-created results are shown Table S The final score for re-creation is 46.1%, which results can produce both correct re-crcation as well as the textual foodback. These results also demonstrate the the difficulty of the task, especially in real scenarios.
The quantitative results of answers shown in Table 9 We can that 78.3% of "can" queries be answered correctly, and 25.0% queries about "cannot" can be correctly explained. In the "exchange position" queries, we find that although it can perform correct re-creation, the reconstruction for other objects is challenging. For the "put in" queries, the errors usually occur by creating a new target fruit without crasing it the image. The re-crcation results of the "remove" are the best among three actions. All the results suggest the challenges of this dataset of real scenes. Example qualitative results shown Table 7
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 9: Textual feedback evaluation on the Fruit-ATVC dataset.
Can
Cannot
Score
Num
Acc.
Num
Type Acc.
Exp Acc.
950
78.3%
922
77.2%
25.0%
64.7%
Table Uncertainty evaluation on CLEVR-ATVC dataset.
Visual (V)
Text (T)
Re-created (M)
Answer (A)
Ranking
Please exchange color of the large blue rubber sphere and the small purple rubber cylinder.
No problem.
A
Please exchange the color of the large purple metal cube and the large red metal cylinder.
No problem.
A
Please exchange the color of the large cyan metal sphere and the small red rubber cube.
No problem.
A
Please exchange the position of the small yellow rubber cube and the small purple rubber cube.
No problem.
A
Please exchange color of the large purple metal cube and the small blue metal cylinder.
No problem.
A
Please put the large purple metal cube under the large gray metal sphere.
No problem.
B
4.3.3 Uncertainty of Image Re-creation
In this section, we would like to explore whether the model can handle the uncertainty appcared in the query. The results show that our method deals with emerging uncertainties well. As shown in Table I0 all the image re-creation results get ranking A, except for the result of the last row. The score is due an object being crasod during image reconstruction, not due uncertainty. For example, as shown in the fourth row of Table 10 our model only exchanges the position of one "small yellow rubber cube", which is consistent with the settings in constructed dataset. The "exchange position" only needs to process
11

---

Published in Transactions on Machine Learning Research (02/2024)
Table 11: Evaluation results on the CLEVR-ATVC sub-dataset.
Human Rank (%)
Methods
Text
Image
PSNR
SSIM
FSIM
A
C
Score
Ours w/ VQVAE
Short
128 x 128
56.3
0.9957
0.689
65.9
14.4
19.7
71.9
Ours w/ VQVAE
Long
128 x 128
56.6
0.9963
0.693
66.4
14.8
18.8
73.8
M
M
(a) Please put the big red cube on top of the big blue cylinder.
(b) Please put the big blue cube on top of the big blue cylinder.
(c) Please put the big blue cube on top of the hig cyan cylinder.
M
M
M
(d) Please put the big red cabe top of the big blue cylinder.
(c) Please-put the big red cube on top of the big blue cylinder.
(f) Please put the amazing big red cube on top of the amazing big blue cylinder.
Figure 4: Image re-creation results on the CLEVR-ATVC sub-dataset using imperfect query. The red text represents the short or imperfect query.
of the objects, but "exchange color" needs to exchange the colors of all satisfied objects. Existing models are able to make different decisions for the above different situations, so our method shows good behavior.
4.4 Single Action without Answer
In this section, we firstly evaluate the image auto-cncoder mothod VQVAE (Oord et al. 2017) with short query and the imperfect query on the CLEVR-ATVC sub-dataset. We only select one "put on top" action without textual feedback for verification and evaluation. Subsequently, we evaluate the image auto-encoder methods, VQVAE (Oord et al. 2017) and VQGAN (Esser et al. (021), on the CLEVR-ATVC sub-dataset.
4.4.1 Short or Imperfect Query for Re-creation
Qian et al. (2021) concluded that the model can produce entity memorization on the form of the input, so we test the model's 's ability to cope with changes in the input form. The short query means that we remove redundant text in the query, such "please", "put", and "the" removed, while the latter randomly removes keywords (such as "color", "size" or "shape"). The quantitative results shown in Table 11 show that the intact queries (long text) perform slightly better than the changed version (short text). This demonstrates that the model can cope with small changes in the form of the input queries.
The visualization results are shown Figure 4 with the observations on the imperfect queries: a) If the "blue" is removed from query, the generated cylinder seems to be shorter; b) If the query contains nonexistent "blue" adjective for the cube, the model creates one; c) If both the nonexistent. "blue" cube
12

---

Published in Transactions on Machine Learning Research (02/2024)
Table 12: Evaluation results on the CLEVR-ATVC sub-dataset.
Human Rank (%)
Methods
Image
PSNR
SSIM
FSIM
B
Score
Ours w/ VQVAE
128 128
56.6
0.9963
0.693
66.4
14.8
18.8
73.8
Ours w/ VQGAN
256 256
53.9
0.9947
0.604
52.7
27.4
19.9
66.4
Input
Re-creation
Ground truth
Query: Please put the small cyan cube on top of the small brown cylinder.
Query: Please put the small yellow cube on top of the small brown cylinder.
Figure 5: Analyse for the results of VQGAN-based image encoder. VQGAN frequently changes the colors of the objects, which downgrades the performance.
and "cyan" cylinder coexist in the same query, the model seems to mistakenly recognize the cyan cube, and find the small gray cylinder is also erased, which is incorrect; d) If simply remove the "cube", the model just dyes litter red on top of the cylinder; e) If remove the "please put", the result just remains the original figure; f) If we add the unseen word, e.g., "amazing", in the query, the manipulation result can still be achieved. We further test a lot of images, which show that the unlearned "create" and "dye" abilities commonly appear but not always.
The aforementioned findings also indicate that without incorporating restriction rules into the dataset, the model may fulfill human instructions by generating new objects, which is not the intended outcome and lacks control. Hence, it becomes imperative to govern the model's behavior by incorporating pre-defined rules supervision signals.
4.4.2 VQVAE vs. VQGAN
As shown in Table 12 although VQGAN-based image auto-cncoder can output large resolution and at 4 times faster of Training than the VQVAE counterpart, the latter is 7.4% better. The results shown in Figure and Bnd show that VQGAN-based image encoder always changes the color objects, erases objects and add irrelevant objects on the re-created images. These results also demonstrate that VQGAN can produce high- quality images using fewer computational resources, but it appears have lost its ability to discriminate on object properties in latent space.
13

---

Published in Transactions on Machine Learning Research (02/2024)
5
Related works
In this section, we first review some representative datasets and methods vision-and--anguuaee tasks whose inputs and outputs are always unimodal. Next, we introduce the prior multimodal dialogue models. Finally, we briefly summarize the recent controllable text-to-image generation methods and the main differences between them and our task.
5.1 Vision-and-Language Tasks
Recent years have witnessed the rapid development of vision-and-languaag tasks. Their development trend can be observed through the construction of the datasets, which can be roughly categorized into two groups: Vision to Language, Language to Vision We briefly summarize these relevant datasets and methods.
Vision to Language. There are several sub-tasks for this category such as Visual Description Genera-
tion (Oroneez et al. 2011), Visual Storytelling (Lin et al., 2014b, Visual Dialog (Das al., 2017 Kottur
et al.
2019), Visual Entailment (Vu et al.
2018
Liu et al.
2020 Visual Reasoning (Johnson et al. 2017)
Visual Question Answering Antol et al.
2015
Marino et al. 2019 and Visual Referving Expression Liu
ct al.
2019
Visual Description Generation, a.k.a., captioning, is the most well-known task, where many datasets
in the past decade are constructed to address different scales, scenes, etc. For examples, image
captioning datasets, (SBUIM (Ordonez et al. ,
Flickr8k (Hodosh et al. 2013
Flickr30k YYung
et al. 2014), MSCOCO Lin 2014)) Multi30k- CLID Elliott et al. 2016).
Flickr30k- -ntttiieeee Entities
[Plummer et al. 2015), STAIR Captions Yoshikawa et al. 2017), Conceptual Captions (Sharma
let all. 2018), MSCOCO- Entities (Cornia et al. 2019 Personality Captions (Shuster et al. 2019]
and video captioning datasets, MSVD (Chen & Dolan (011)) [PIII Cooking (Rohrbach et al.
2012h, YouCook (Das et al., 2013). TACoS) (Regneri et al., 2013), [TACoS-MultiLevel (Racoveaa
(Rohrbach
ct al. 2014), M-D (Torabi et al. 2015). MSR-VTT] (Xu et al. 2016), VTW (Zeng et al. 2016).
ANeeCCpp (Krishna al. 2017)) YouCook IT (Zhou et al. 2018)). ANeiiatities (Zhou al.
2019
COIN [Tang al.) 2019), HowTo000M (Miech al. 2019). have played an important role advancing visual captioning.
Visual Storytelling usually requires the methods to generate a series of text to describe a set or a sequence of the visual inputs, as shown in several datasets: NYC-Storytelling (Park & Kim 2015)
Disneyland Storytelling (Kim et al. 2015), SIND and VIST Huang et al. 2016).
Visual Dialog can be regarded as natural conversation system based on the image video. Ex- amples can be found in the existing benchmarks such as VisDial Das et al. 2017). CLEVR-Dialog (Kottur et al.
2019 and AVSD Alamri et al. 2018).
Visual Entailment is introduced in V-SNLI Vu et al. 2018), SNLI-VE (Xie et al., 2019 and VIOLIN (Liu et al.) (2020), which requires the method learn to select the correct premise and hidden semantic information that can be inferred from the visual contents.
Vision Reasoning is expected to provide the scene graphs between objects and then to answer questions based on the visual input and relationships. Many datascts such as CLEVR JJhnson let al.
(Yang et al., 2018 NLVR2 (Suhr et al. GQA (Hudson & Manning 2019), VCR (Zellers et al. 2019). and Visual COMET (Pakk et al.
2017), NLVR (Suhr et al.) 2017),
2018),
2020 have been constructed between 2017 and 2020, which demonstrate visual reasoning
abilities can be effectively learned by the deep neural networks [eCCun et al. 2015).
Visual Question Answering is another representalive vision-to-language Lask, in which the model is expected to generate text- based answer according to the question and the visual cues. The most influential datasct VQA v1.0 (Antol et al. 2015) as well as MovicQA (Tapaswi et al. 2016), TVQA
Lei et al. 2018), VKA [Marnn et al. 2019) KVQA (Shah al. 2019), and VQA v2.0. (Antol
et. al. 2015 have greatly promoted the development.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Visual Referring Expression is another task which requires the method to comprehend the referring expressions by showing evidence in the visual images, i.e., using bounding boxes to locate the corresponding objects. There are several datasets targeting this task, such as RefCLEE Kazemzadehi
ct al. 2014 and C [Luu et al. 2019).
Language to Vision. This task aims at image generation based on the pure natural language. Currently, there are only a few datasets for this task such as Oxford-102 Reed et al., 2016), Caltech-UCSD Birds
(CUB) (Reed al., 2016), [ (Xia et al.
2021), Text2Human (Jiang et al. 2022)
Laion 400M SShhuhmann al. 2021]. Laion-50
5B
(Schuhmann et al.
2022
and Text2Video (Xu et al. 2016
Li et al. 2018 Hong et al. 2022 Ho et al. 2022
2023 ?). These datasets are similar to Khachatryan et al.
the vision-to-lanee languuuge task above, and the text-image pairs contain text descriptions of the image content. The most representative task is text-based image generation, which has recently seen large progress in
multimodal learning (Brooks et al.) 2023 Li et al. 2023b). Many previous works train GANs with text- conditioning on publicly available image captioning datasets (Tao et al.. 2020 Zhang et al.] 2021). DALL-E 2021b) uses 250 million lext-image pairs successfully achieve promising results, which even perform zero-shot visual reasoning by using some prompts. Another zero-shot model termed CLIP [Radford et al. 2021 is used to rank and measure the similarity between image and text, usually one text prompt can produce numerous plausible results. The recent DALL-E 2 (Ramesh et al., 2022) uses diffusion prior on CLIP text latents and cascaded diffusion models to gemerale high resolution 1024 x 1024 images.
RRamesh et al.,
GLIDE (Nichol et al.
2021) also applies guided diffusion to the problem of text-conditional image synthesis.
Imagen (Saharia et al. 2022) combines the power of transformer language models with high-fidelity diffusion models to deliver an unprecedented degree of photorealism in text-to-image synthesis.
In contrast to the aforementioned datasets that typically concentrate one aspect, our proposed datasets introduce a novel requirement for models to generate both visual re-creations and textual feedback simulta- neously. The visual manipulation aspect necessitates not only performing the required actions accurately but also preserving a visually plausible background. Furthermore, the feedback generated by the model needs to be aware of the possibilities of feasible actions, actions that cannot be performed, and actions that are explicitly prohibited. This requirement ensures that our constructed datasets possess both intrinsic value and present a significant challenge.
5.2 Multimodal Dialogue Models
The task of vision-to-language involves using visual samples as input and gencrating text as output, as seen in tasks like Visual Dialog. Conversely, language-to-viiion lasks operate in the opposite direction. In multimodal dialogue methods, the model must reason about multimodal or unimodal inputs and produce multimodal or unimodal outputs. These methods can be categorized into the following two types
Multimodal Input and Unimodal Output (MIUO): This conversational task is similar to visual dialog, where the input typically consists of visual data and text-based prompts (i.e., multimodal input). The visual language model performs visual reasoning on the image and provides an answer (i.e.. unimodal output) ( [Alayrac et al.
2022
Brooks et al. 2023
Gong et al.
2023 Bo Li 2023
Li et al.
2023a
OpenAl 2023b Su et al.
2023
Yang et al.] 2023b
Zhao et al. 2023 Zhu et al. 2023 Liu
ct al. 2023 Mu al. 2023 Wang et al., 2023 Zhang et al.] 2023). GPT-4 (OpenA 2023b and Multi GPTT (Gong et al. 2023) can handle prompts consisting of interleaved visual inputs and text- bast-based queries, generating text outputs. VideoChat (Li et al. 2023al) introduces video- centric multimodal dialogue dataset, enabling trained models to understand and generate detailed conversations about vidcos.
Multimodal Imput and Multimodal Output (MIMO): This task requires the model to perform multi- modal reasoning and generation simultaneoully (Koh et al.] 2023 Jing Yu Kohj 2023 Chenfei Wu & Duan 2023 Yang et al. 2023a). Visual ChatGPT (Chenfei Wu & Duan 2023) pioneering work that combines ChatGPT and scrics of pro-trained visual foundation models, allowing them
htttps//github.. http:/htww...c////////////////ee wwwoooof-oullaimn Multimodal maattii---Daalggue-eaa Diialogue-Moell
15

---

Published in Transactions on Machine Learning Research (02/2024)
to accept and produce text and images during textual-visual conversations. GILL (Koh et al 2023
proposes mapping network that efficiently maps the output embedding space of a frozen text-only language model to that of a frozen generation model (e.g,, Stable Diffusion (Rombach et al. 2022) ).
This mapping only requires fine-tuning a small number of parameters on imago-caption pairs for lasks such as image retrieval, novel image generation, and multimodal dialogue. FROMAGe (Jing Yu Koh 2023) also involves image-text inputs and outputs for multimodal dialogue, with few lin- ear layers tinettued while keeping the pre-trained language model frozen. GPT4Tools (Yang et al. 2023a introduces an instruction dataset and extends Visual ChatGPT (Chenfei Wu & Duan 2023 to the image understanding task.
5.3 Controllable Text-to-image to-imaee Generation
The above text-based image generation methods primarily focus on generating high-quality images based on given text descriptions, without providing the user with the ability to manipulate specific visual attributes using natural language instructions (Anderson et al.
2018
Nguyen al. 2019; Thomason al. 2020 Cheng
ct al. 2014 Scalise et al.] 2018 Stepputtis ct al. 2020
Nam ct al. 2018 Tüksel et al. 2021 Shen ct al.
2020
Li et al. 2020: Richardson et al. 2020 Esser el al.
2021). However, some recent works have explored
techniques to control the synthesis process by representing scenes as compositional generative neural feature fields, enabling the disentanglement of objects from the background, as well as individual objects' shapes and appearances (Li et al. 2019 Bodla et al. 2018 Nam et al., 2018 Chen et al., 2021 Shuster et al. 2019). For instance, ControlGAN (Li et al.) 2019) can synthesize high-quality images while allowing control over specific parts of the image generation based on natural language descriptions. FusedGAN (Bodla all 2018) achieves enhanced controllability in sampling by discntangling different factors in the latent space. Text2Hunan (Jiang et al. 2022 can synthesize human images by specifying clothing shapes and lextures solely through natural language descriptions.
However, the aforementioned approaches focus on controlling the image generation process through compo- sition or disentanglement techniques. In this paper, our objective is to control the outcomes of re-created images, where the system learns to respond with a "no" to commands that are cither prohibited or cannot be executed.
6 Conclusion
In this paper, we raise an important yet underexplored concern regarding the accountability of multimodal generative models. To tackle this issue, we introduce two novel datasets, namely CLEVR-ATVC and Fruit- ATVC, designed for unique task called Accountable Text-based Visual Re-creation. The objective of this task is to train Visual Language Models (VLMs) to reject human instructions. Our datasets consist of both visual and textual inputs and outputs, requiring the model to perform visual re-creation while ensuring image quality when the answer is "yes". In cases where the model cannot complete the action or the action is prohibited, it is required to provide an explanation. We provide baseline models, experimental settings, evaluation metrics, and comprehensive analysis, presenting some promising results. These high-quality datasets can also be utilized in similar tasks, and we hope that our work inspires further rescarch the accountability problem, encompassing model design, label annotation, and the creation of larger datasets. We firmly believe that addressing the issue of accountability is crucial for advancing the development and deployment of multimodal generative models.
7 Acknowledgments
The authors would like to thank the anonymous reviewers for their helpful feedback and the funding agencies listed below for supporting this work. This work is supported in part by the National Key Research and Development Program China (Project Number 2022YFC2305002.. We would like to thank Ying Cai, Xiaoai Sha and other members for helping label text-image pairs and participate human evaluation experiments. Finally, we appreciate all the people who are working on making code, models and data publicly available.
16

---

Published in Transactions on Machine Learning Research (02/2024)
References
Huda Alamri, Chiori Hori, Tim K Marks, Dhruv Batra, and Devi Parikh. Audio visual scene-aware dialog (avsd) track for natural language generation in dstc7. In DSTC7 at AAAI2019 Workshop, volume 2, 2018.
Jcan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Peter Anderson, Qi WI, Damien Teney Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. laioo-and-languaae navigation: Interpreting gisually-gruundd navi- gation instructions in real environments. In Proceedings of the IEEE Conference on Compuler Vision and Pattern Recognition, pp. 3674 3683, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu,, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pp. 2425-2433, 2015.
Liangyu Chen Jinghao Wang Fanyi Pu Jingkang Yang Chunyuan Ziwei Liu Bo Li, Yuanhan Zhang. Mimic-it: Multi-modal in-context. instruction tuning. arXiv preprint ar orXiv:2306.054 2023.
Navaneeth Bodla, Gang Hua, and Rama Chellappa. Semi-supervised fusedgan for conditional image gener- ation. In Proceedings of the European conference computer vision (ECCV), pp. 589-683, 2018
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392 -8402,, 2023.
David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 190-200, 2011.
Tianyi Chen, Yi Liu, Yunfei Zhang, Si Wu, Yong Xu, Feng Liangbing, and Hau San Wong. Semi- supervised single-stage controllable gans for conditional fine-grained image generation. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 2264-9273, 2021.
Weizhen Qi Xiaodong Wang Zecheng Tang Chenfei Wu, Shengming Yin and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXie preprint arXiv:2303. NNN:22333.0461,, 2023.
Yu Cheng, Yunyi Jia, Rui Fang, Lanbo She, Ning Xi, and Joyce Chai. Modelling and analysis of natural language controlled robotic systems. IFAC' Proceedings Volumes, 47(3):11767 11772, 2014.
Rewon Child, Scott Gray, Alec Radford, and lya Sutskever. Generating long sequences with sparse trans- formers. Xrx preprint 1XX::1904,11000 2019.
Marcella. Cornia, Lorenzo Baraldi, and Rita Cucchiara. Show, control and tell: A framework for generating controllable and grounded captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8307-8316, 2019.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In Proceedings of the IEEE Conference on Compuler Vision and Paltern Recognition, pp. 326 335, 2017.
Pradipto Das, Chenliang Xu, Richard F Doell, and Jason J Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of the IEEE conference on compuler vision and pallern recognition, pp. 2634 2641, 2013.
Jimmy Ba Diederik P. Kingma. Adam: A method for stochastic optimization. 111 2014.
arXiv preprint
17

---

Published in Transactions on Machine Learning Research (02/2024)
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. arXiv preprinl arXiv:2105.13290 2021.
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo. Semantic image synthesis via adversarial learning. Tn Proceedings of the IEEE International Conference on Computer Vision, pp. 5706 5714, 2017.
Finale Doshi-Velez, Mason Kortz, Ryan Budish, Chris Bavitz, Sam Gershman, David O'Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, et al. Accountability of ai under the law: The role of explanation. arXiv preprint arXiv:1111.011 2017.
Desmond Elliott, Stella Frank, Khalil Sima's an, and Lucia Specia. Multi3kk: Multilingual english-german image descriptions. arXiv preprint arXiv: 00459,,4, 2016.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. CVPR, 2021.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans. arxiv preprint 2rx::22005.04 04790, 2023.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint wrXiv:2204.0344 2022.
Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853- 899, 2013.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv: 15868, 2022.
Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistios: Human Language Technologies, pp. 1233 1239, 2016.
Drew A Hudson and Christopher D Manning. Gqa: a new dataset for compositional question answering over woaldworld images. arxio preprint 1rXiv:1902.0 095009950, 2(3):11, 2019.
Yuming Jiang, Shuai Yang, Haonan Qju, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics (TOG), 41(4):1- 2022.
Daniel Fried Jing Yu Koh, Ruslan Salakhutdinov. Grounding language models to images for multimodal inputs and outputs. arXiv preprint arXiu:2301.1382,, 2023.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Tawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. e-vil: A dataset and benchmark for natural language explanations in vasion-language tasks. arXiv preprint or orXiv:2105.03700 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg, Referitgame: Referring objects in photographs of natural scenes. In Proceedings of the 2014 conference on empérical methods in natural language processing (EMNLP), pp. 787-798, 2014.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zere Text-to-image diffusion models are zero-shot video generators. Xiv preprint or Xiv:2303. 13439, 2023.
18

---

Published in Transactions on Machine Learning Research (02/2024)
Gunhee Kim, Seungwhan Moon, and Leonid Sigal. Ranking and retrieval of image sequences from multiple paragraph queries. In Proceedings of the IEEE Conference on Compuler Vision and Poltern Recognition, pp. 1993 2001, 2015.
Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Generating images with multimodal language models. arXiv preprint ar Xin:2390.. 7216, 2023.
Satwik Kottur, José MF Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. Clevr-dialog: A diagnostic datasct for multi-round reasoning in visual dialog. arXiv preprint Xrx: 09,600 2019.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. 706 715, 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, (7653))333 4444-44,,44444 2015.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5549-5558, 2020.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara I, Berg. Tyqa: Localized, compositional video question answering. Xiv preprint XXi::1809001690 2018.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. Ad- vances in Neural Information Processing Systems, 32, 2019.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Manigan: Text-guided image manipu- lation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7880-7889, 2020.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint orXiv:2305, 06355, 2023a.
Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In Proceedings of the AAAT Conference on Arlificial Intelligence, volume 32, 2018.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfong Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. arXiv preprint 0rXw::3301.07093, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. Furopean conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint D.XXi:2304408400 2023.
Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu. Violin: A large-scale dataset for video-and-languugge inference. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 10900- 10910,, 2020.
Runtao Liu, Chenxi Liu, Yutong Bai, and Alan Yuille. Clevr-reft: Diagnosing visual reasoning with referring expressions. Tn Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4185 4194, 2019.
Michele Loi and Matthias Spielkamp. Towards accountability in the use of artificial intelligence for public administrations. In Proceedings of the 2021 AAAT/ACM Conference on AI, Ethics, and Society, pp. 757 766, 2021.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3195 3195-3204, 2019.
19

---

Published in Transactions on Machine Learning Research (02/2024)
Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivie. Howtol00m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 2630 2640, 2019.
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodicdgpt: Vision-language pre-training via embodied chain of thought. arXiv preprinl Xiv:2230.. 15021, 2023.
Seonghyoon Nam, Yunji Kim, and Scon Joo Kim. Text-adaptive generative adversarial networks: manipu- lating images with natural language. arXiv preprinl Xrxiv:1810. 11000.119919 1919, 2018.
Khanh Nguyen, Debadeepla Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language- based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 12527-12537, 2019.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 0rXiv:2112.107// 2021.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings of the IERE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11453 11464, 2021.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arXte:17/11000933 2017.
OpenAl. Chatgpt. https://openai..om wwt.//ooo///a//aitttoeeeooooo/tt 2023a.
OpenAL. Gpt-4 technical report. arXiv preprint or arXim:2303.08774, 2023b.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in neural information processing systems, 24:1143 1151, 2011.
Cesc C Park and Gunhee Kim. Expressing an image stream with a sequence of natural sentences. Advances in neural information processing systems, 28:73-81, 2015.
Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. Visualcomet: Rea- soning about the dynamic context of a still image. In European Conference on Computer Vision, pp. 508-524. Springer, 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. arXiv preprint arXiv:2103.112 17249, 2021.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan Caicedo, Julia Hockenmaier, and Svetlana Lazobník. Flickr30k entities: Collecting rogion-to-phrase correspondenees for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp. 2641 26499 2015.
Kun Qian, Ahmad Beirami, Zhouhan Lin, Ankita De, Alborz Geramifard, Zhou Yu, and Chinnadhurai Sankar. Annotation inconsistency and entity bias in multiwoz. arXiv preprint vrXiv:2105.141500 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:210S. 210.00003000000 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021a.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot. text-to-image generation. arXiv preprint ar Xix:21100.. 12092, 2021b.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint or arXiv:22040061 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schicle, and Honglak Lee. Generative adversarial text to image synthesis. In International Conference on Machine Learning, pp. 1060 1069. PMLR, 2016.
Michaela Regneri, Marcus Rohrbach, Dominikus Wetzel, Stefan Thater, Bernt Schiele, and Manfred Pinkal. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1:25-36, 2013.
Elad Richardson, Yuval Alaluf, Or Patashmik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: stylegan encoder for image-to-image translation. Xi preprint arXiv:2008.00 2005500511 2020.
Anna Rohrbach, Marcus Rohrbach, Wei Qiu, Annemaric Friedrich, Manfred Pinkal, and Bernt Schiele. Coherent multi-senience video description with variable level of detail. In German conference pallern recognition, pp. 184- 195. Springer, 2014.
Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, and Bernt Schiele. A database for fine grained activity detection of cooking activities. In 2012 IEEE Conference on Compuler Vision and Pallern Recog- nition, pp. 1194-1201, IEEE, 2012.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 10684 10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Sara Mahdavi, Rapha Gontijo Lopes, al. Photorcalistic text-to- image diffusion models with deep language understanding. arXiv preprinl ar Xix: 1 11487, 2022.
Rosario Scalise, Shen Li, Henny Admoni, Stephanie Rosenthal, and Siddhartha S Srinivasa. Natural language instructions for human human-robot collaborative manipulation. The International Journal of Robotics Research, 37(6):558 565, 2018.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint vexiv:1111.01111 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. orXie preprint arXim:2210.084000 2022.
Haddow B Sennrich R and Birch A. Neural machine translation of rare words with subword units. arXiv preprint ar weXii:1508.079900 2015.
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. Kvqa: Knowledge-aware visual question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8876 8884, 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2556-2565, 2018.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolci Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9243 9252, 2020.
21

---

Published in Transactions on Machine Learning Research (02/2024)
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. Engaging image captioning via personality. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pottern Recognation.
pp. 12516 12526, 2019.
Simon Stepputtis, Joseph Campbell, Mariano Phiclipp, Stefan Lee, Chitta Baral, and Heni Ben Amor. Language-conditoned imitation learning for robot manipulation tasks. Neurips, 2020.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model instruction- follow them all. Xiv preprint or arXiv:2905.1000 16355, 2023.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi A corpus of natural language for visual reasoning, In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 217 223, 2017.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXio:1811.00441 2018.
Yansong Tang, Dajum Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao Jing, Fei Wu, and Bingkun Bao. Di-gan: Deep fusion generative adversarial networks for text-to-image synthesis. arXiv preprint arxv::0008. 05865, 2020.
Makarand Tapaswi, Yukun Zhu, Rainer Sticfelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fi- dler. Movieqa: Understanding stories in movies through question-answeringg In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1831-4640, 2016.
Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In Conference Robot Learning, pp. 394 406. PMLR, 2020.
Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville. Using descriptive video services to create a large data source for video annotation research. arXie preprinl arXiv:1503.01077 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint Miiv:1706.0371 2017.
Hoa Trong Vu, Claudio Greco, Aliia Erofeeva, Somayeh Jafaritazehjan, Guido Linders, Mare Tanti, Al- berto Testoni, Raffaella Bernardi, and Albert Gatt. Grounded textual entailment. ar preprinl arXiv:1806.n 1.Xiv:1806.05645, 2018.
Teng Wang, Jinrui Zhang, Junjie Fei, Yixiao Ge, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, Shanshan Zhao, Ying Shan, et al. Caption anything: Interactive image description with diverse multinodal controls. 2023.
arXiv preprint wxXi::2305.0267
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. N uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint Xiv:21111.12411, 2021.
Weihao Xia, Yujiu Yang, Hin Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image gen- eration and manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognilion, pp. 2256-2200 2021.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: novel task for fine-grained image understanding. arXiv preprint arXiv: 1901 0067066 2019.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large vidco description datasct for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288 5296, 2016.
22

---

Published in Transactions on Machine Learning Research (02/2024)
Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang, Jonathon Shlens, and David Sussillo. A dataset and architecture for visual reasoning with working memory. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 714 731, 2018.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpl4tools: Teaching large language model to use tools via self-instruction. arXiv preprint arXiv:2905.188 18752, 2023a.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXwv preprint (ttt:2333.1188... 2023b.
Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. Stair captions: Constructing a large-scale japanese image caption dataset. arXiv preprint arXi:: 775.00882, 2017.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual de- notations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Oguz Kaan Yüksel, Enis Simsar, Eagi Gülperi Er, and Pinar Yanardag. Latentelr: A contrastive learning approach for unsupervised discovery of interpretable directions. arXiv preprint arXiv:2104. 00820, 2021.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi, From recognition to cognition: Visual com- monsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6720 6731, 2019.
Kuo-Hao Zeng, Tseng-Hung Chen, Juan Carlos Niebles, and Min Sun. Generation for user generated videos. In European conference on computer vision, pp. 609-625. Springer, 2016.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pottern analysis and machine intelligence, 41(8))1947-1962, 2018.
Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 833 842, 2021.
Lin Zhang, Lei Zhang, Xuanqin Mou, and David Zhang. Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing, 20(8):2378 2386, 2011.
Tianjun Zhang, Yi Zhang, Vibhav Vincet, Noel Joshi, and Xin Wang. Controllable text-to-image generation with gpt-4. arXiv preprint Xrx::2235. 18583, 2023.
Zhiqiang Zhang, Wenxin Yu, Jinjia Zhou, Xuewen Zhang, Ning Jiang, Gang Hc, and Zhuo Yang. Cus- tomizable gan: A method for image synthesis of human controllable. IEEE Access, 8:108004-008017 2020.
Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as language catalyst. arXiv preprint XxXi:2330516610. 2023.
Luowei Zhou, Chenliang Xu, and Jason Corso, Towards automatic learning of procedures from web instruc- tional videos. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 32, 2018.
Yimin Zhou, Yiwei Sun, and Vasant Honavar. Improving image captioning by leveraging knowledge graphs. In 2019 IEEE winter conference on applications of computer vision (WACV), pp. 283 20-2293.11 IEEE, 2019.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. XrXiu preprint arXiv:2304, 10592, 2023.
23

---

Published in Transactions on Machine Learning Research (02/2024)
Figure A1: The example images of Fruit-ATVC dataset show the diversity of collection scenes.
A Additional Data Details
This section provides further information about the data used in our research. We begin by presenting a wider range of example images from the Fruit-ATVC dataset highlight the diversity of collection scenes. Next, provide summary of statistics regarding the various actions found in the datasets. Subsequently, we elaborate the process of human evaluation experiments. Lastly, we introduce the format of the annotation files for our datasels.
A.1 Diversity of Fruit-ATVC
The following Figure Al shows the diversity of collection scenes on Fruit-ATVC dataset. The image resolution is average 2700x2700, and the images were taken with different categories of mobile phones, including iphone,
24

---

Published in Transactions on Machine Learning Research (02/2024)
Exchange position
Exchange position
19.98%
Put on
19.80%
Remove
19.98%
17.95%
30.02%
Put under
30.02%
62.24%
Exchange color
Put on
Figure A2: The distribution different actions on CLEVR-ATVC and Fruit-ATVC datasets.
Figure A3: The visualization of bounding boxes in the CLEVR-ATVC dataset.
huawei, xiaomi, etc. We would like to emphasize that the image quality of this dataset is very high, which results in the size of the original Fruit-ATVC is 168 GB. In addition, the background of data collection, fruit categories and container types are also very diverse. We believe this dataset can also be used for other image generation tasks. The Figure A2 illustrates the distribution of different actions on our constructed datasets.
A.2 Annotation File Format
We describe the format of the annotation files for both the CLEVR-ATVC and Fruit-ATVC datasets. Listings and demonstrate the consistent structure of the annotation files, with the primary distinction being the inclusion of detailed 2D and 3D coordinate locations of objects in the CLEVR-ATVC dataset. Figure A3 provides a visual representation of the bounding boxes in the CLEVR-ATVC dataset. It is worth noting that we anticipate the potential use of this dataset for other related tasks, such as compositional visual reasoning, While we have previously introduced the main structure of the annotation files in the main paper, we now aim provide additional details.
The annotation files begin by providing a summary of the dataset. contributors, timestamp, project link, version, and license information. This introductory section ensures comprehensive documentation and proper attribution. Following this, the annotation files include the categories of objects present in the dataset, along with associated object adjectives, actions, and other relevant information.
The "images" section contains all the necessary information about the visual inputs. This includes the image index, filename, link, the number of objects in the image, and detailed descriptions of each object present. Each object is listed under the "objects" child list, accompanied by specific properties such index, category index, and bounding box coordinates.
25

---

Published in Transactions on Machine Learning Research (02/2024)
The text-based queries (Q) and textual feedback (A) included in the "questions" section of the annotation files. The answer types are indicated by numbers ranging from O to 2. Specifically, value of 0, 1, or 2 represents an instruction that cannot be executed, an instruction that can be executed, or a prohibited instruction, respectively. section will be emply. Notably, the CLEVR-ATVC dataset provides additional information regarding any changes in the color or position an object after displacement or property modification.
the action type is O or 2, the subsequent "actions" list in the "recreations"
B Human Evaluation Experiments
This section presents examples of the interfaces utilized for facilitating human evaluation. During the evaluation phase, the visual input, text-based query, re-created image, answer, and ground truths for re- creation and textual feedback tool. Evaluators then only required to assign scores to the respective results and save them. Subsequently, we calculate the final score of the model based on the saved results.
automatically imported into the corresponding sections of the auxiliary
As detailed in the main paper, we assigned the same results to different evaluators, with each evaluator providing identical scores for the experimental outcomes. The aforementioned experiments demonstrate that the auxiliary tools we have designed not only significantly reduce the evaluation time required but also ensure the accuracy and reliability of the widely used human evaluation process.
26

---

Published in Transactions on Machine Learning Research (02/2024)
Listing 1: The format of annotation file CLEVR-ATVC dataset.
contributor's "Zhiwei Zhang, Yuliang Liu maat_crccatdd "2023",
"url:: 'https:// /ttps:/mmatrixalllppppa matrix-alpha github.io/" ,o/"
description : "CVE Dataset",
'vertion', *1.0",
"licenses': {
'id': '1,, 'url'; 'Nnone', 'name": "Creative Commons Attribution (CC-BY 4.0)*,
}. "categories". ("cube, cylinder, sphere
sizes': s'small, large']:
colors'; I' l'gray, red. blue, green, brown, purple, cyan, yellow material's ('rubber, metal'], "actionss ['put top, put under, exchange position, exchange color' '],
'images":
'image_idx': "0000001',
immaee filename': *521100 03 000011 png",
'id': '521100 03 1010000030000011,, 0000110,
"objectnnumber" number': 3,
ccirte_creted': *2023 04- 00:17:14',
objjets': {
index 0,
category_ide id id": 0,
size id 1,
color id' 3,
'bbox: [29, 62, 81, 116]. '3d coordo''''' 1-2.96856045, -1.99158716, 0.6999999]
mmaterial_ii id": 1}.
}.
"questions:: [
question idx": '0000001',
id" "521100_00 03 ",00000000000011, question number": 10,
ques":
'ques id 0,
'ques idx' 1,
'id': '521100 03 00000000110000000 "type": 1,
'Q': P'Pase put the cylinder on top of the cube."]
'A': ['No problem
}.
"recreations"
27

---

Published in Transactions on Machine Learning Research (02/2024)
rec idx : '0000001",
rec id': "521100 03 "22100_0030000111,, 000011',
rec_num'; : 10,
rec num"
acttioss:
{
actions id'; 0,
actions_ied idxx: 1
rec_ gee_filename": 5521100 00000000111111110000000000000001 0 01 .0.pg".
conrcctr_mmerrss 3,
date created": 22023004-2 00:17:59',
"objects":
0,
{
'ndexx: 2,
category _i id" 1,
size id 0,
color id': 1,
bboxx: [39, 45, 65, 72].
3d _dd__coor''; [-2.96856045, -1.99158716, 1.75], mmateria_ id": 0)
28

---

Published in Transactions on Machine Learning Research (02/2024)
Listing 2: The format of annotation file Fruit-ATVC dataset.
contributor's "Zhiwei Zhang, Yuliang Liu', aatta_rreated" "2023",
"url:: "https://matri: matamriaiixaappha github.io/" ,o description' : "Fruit-AIVe Dataset", 'vertion', "1.0",
"licenses': {
'id': '1,,
'url': 'Nnone', 'name": "Creative Commons Attribution (CC-BY 4.0)*,
}. "fruits:: ('apple, cocount, orange, banana, kiwi, mango, avocado, etc. '],
containnes". ": ["plate, bowl, bottle'], "actions': : ['put in, exchange position, remove'],
"images":
"image idx': "0000001',
siage_filenamee. '0000001.pag' 'id' '0000001',
'fruit number": 3,
'container Sattaainrrnummbrrs..
cara__created' *2023 04-27 00:10:11',
fruits'
"index': 0, cateeoryy id"
"index': 1, "category i
"index 2, "category id':
"containers":
index 0
calegory id": 2
1,
b.
qqeestion': [
question_ias idx': '0000001',
question_idl id": '0000001",
question number": 3,
quess:
'ques_i"" id 0,
'ques idx' 1,
29

---

Published in Transactions on Machine Learning Research (02/2024)
"id" : '0000001_00011, 'type": 1,
'Q": I' Please put the kiwi in the plate. "], :A': ['No problem.') ']
b
recreations
rec idx' '0000001",
rec id'; "0000001"
rec num's 3,
acttoos':
actions id': 0,
actions idx": 1,
rec filename': '0000001 0 01. pug',
fruit mimit_number'; 3.
container number': 1,
date citeee_ceeeed -2023-04-3 27 00:10:18',
30

---

Published in Transactions on Machine Learning Research (02/2024)
Input
Re-creation
Ground truth
Query: Please put the large green cube top of the large yellow cylinder.
Qucry: Please pur the small cube on top the large yellow cylinder.
Input
Re-creation
Ground truth
Query: Please put the large green cube top of the small red cylinder.
Query: Please put the large green cube on top of the small red cylinder.
Input
Re-creation
Ground truth
Query: Please put the small yellow eube on top of the lurge blue cylinder.
Query: Please put the large brown cube on top of the small green cylinder.
Figure B4: More results are show why VQGAN-basod image encoder performs worse on image re-crcation.
31

---

Published in Transactions on Machine Learning Research (02/2024)
Input
Re-creation
Ground truth
Please exchange the color of the small yellow rubber cuhe and the large red metal sphere.
Query:
Answer: No problem.
Ground truth: No problem.
Human Evaluation
1. Please rate the visual re-creation?
A:
O 0.5
C: 0
2. Please rate the type of answer?
0
3. Please the explanation answer?
0
4. What is the score for textual feedback? [1.0]
5. What is the full-match score? [1.0]
Input
Re-creation
Ground truth
Query: Please put the large gray metal cylinder under the small yellow rubber cylinder.
Answer: This action cannot done. Because there no large gray metal cylinder and no small yellow rubber cylinder.
Ground truth: This action cannot be done. Because there is no large gray metal cylinder and small yellow rubber cylinder.
Human Evaluation
1. Please rate the visual re-creation?
A:
о B: 0.5
O C: 0
2. Please rate the type of answer?
0
3. Please rate the explanation answer?
0
4. What is the score textual feedback? [1.0]
S. What is the full-match score? [1.0]
32

---

Published in Transactions on Machine Learning Research (02/2024)
Input
Re-creation
Ground truth
Query: Please put the large tray metal cube on top of the large blue metal sphere.
This action is forbidden, Recause you cannot put an object on top of the sphere, and there is no large gray metal cube and no large blue metal sphere.
Answer:
object top of the sphere, and there large gray metal cube and large blue metal sphere.
Ground truth: This action is forbidden. Because you cannot
Human Evaluation
1. Please rate the visual re-creation?
A:
OB: 0.5
C: 0
2. Please rate the type of answer?
0
3. Pleasc the explanation of answer?
0
What is the score for for textual foedback? [1.0]
5. What is the full-match score? [1.0]
Input
Re-creation
Ground truth
Query: Please put the large red metal sphere under the small yellow rubber cube.
This action is forbidden. Because you canmot put the sphere under an object, and there is large red metal sphere and no small yellow rubber cube.
Answer:
Ground tuthh This action is forbidden. Because you cannot put the sphere under object.
Human Evaluation
1. Please rate the visual re-creation?
A:
O B: 0.5
O C: 0
2. Please rate the type of answer?
0
3. Please rate the explanation answer?
4. What is score for for textual feedback? [0.5]
5. What is the full-match score? [0.75]
33

---

Published in Transactions on Machine Learning Research (02/2024)
Input
Re-creation
Ground
Query: Please put the small yellow metal sphere under the large red metal cube.
Answer: This action is forbidden. Because you cammot put sphere under an object, and there is no small yellow metal sphere and no small red metal cube.
Ground truth: This action is forbidden. Because you cannot put the sphere under object, and there is small yellow metal sphere and large red metal cube.
Human Evaluation
1. Please rate the visual re-creation?
A:
OB: 0.5
C: 0
2. Please rate the type of answer?
0
3. Pleasc the explanation of answer?
O
What is the score for for textual foedback? [0.5]
5. What is the full-match score? [0.75]
Input
Re-creation
Ground truth
Query: Please put the small green metal cube under the large purple metal sphere.
Answer: No problem.
Ground truth: No problem
Human Evaluation
1. Please rate the visual re-creation?
A
B: 0.5
O C: 0
2. Please rate the type of answer?
0
3. Please the explanation answer?
0
4. What is the score for textual feedback? [1.0]
S. What is the full-match score? [0.75]
34

---

Published in Transactions on Machine Learning Research (02/2024)
Input
Re-creation
Ground truth
Please put the large purple metal sphere top of the large purple metal cylinder.
Query:
Answer: This action is forbidden. Because cannot put the sphere under an object.
Ground truth: No problem.
Human Evaluation
1. Please rate the visual re-creation?
A
O B:: 0.5
C: 0
2. Please rate the type of answer?
3. Please the explanation of answer?
0
4. What is the score for for textual feedback? [0]
5. What is the full-match score? [0]
Input
Re-creation
Ground truth
Please exchange the position of the large yellow metal cubc and the large green metal cylinder.
Query:
Answer: No problem.
Ground truth: This action cannot done. Because there large yellow metal cube and no large metal cylinder.
Human Evaluation
Please rate the visual re-creation?
A 1
O B: 0.5
C: O
2. Please rate the type of answer?
3. Please rate the explanation of answer?
4. What is the score for for textual feedback? [0]
5. What the full-match score? [0]
35


Reviews:
[{"review_id": "gyRZvM0m2C", "reviewer": "Reviewer_EcwY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ As authors mentioned in the conclusion section, the collected datasets will foster future research on the accountability of VLMs.\n+ The proposed baseline seems to manipulate image input well based on the textual prompt, or reject the prompt if it cannot be executed or is prohibited.\n\nHowever, the writing should be improved.\n- Generation of textual queries and feedbacks on Fruit-ATVC is unclear. Please elaborate on the interface shown in Figure 2.\n- Some of the sections should be differently represented; e.g., Section 4.3.3 and Section 4.4.3 describe redundant issues (VQVAE vs VQGAN), the descriptions about imperfect queries are mentioned in Section 4.4.1, not Section 4.4.2, and many examples mentioned in Section 4.4.2 are not about imperfect queries, but the queries that cannot be executed or unseen queries.\n- The attention mask used in the decode-based Transformer should be described in more detail. The following sentences in Section 3 are quite unclear: The transformer is a pure decoder, where text tokens can be attended by each image token in any one of the self-attention layers. Causal mask and full attention mask are used for the text-to-text and image to-image, respectively.\n- The paper addresses the accountability of VLMs, but it is limited in terms of object attributes. The VLMs must also reject prompts requiring unethical behaviors or that cannot be executed in more diverse scenarios, such as the prompt that defies the law of physics. The authors should address the accountability of VLMs in more detail in the Related Work section.\n\nAnd most importantly, conducting experiments with only the proposed baseline does not fully demonstrate the model's ability and reduces the reliability of dataset analysis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes new two datasets to evaluate the multimodal generation capabilities of Visual Language Models (VLMs): the synthetic CLEVR-ATVC dataset and the manually pictured Fruit-ATVC dataset. The most notable feature of these datasets is that on these datasets, the model is required to reject textual prompts if they are prohibited according to pre-defined rules or cannot be executed. The authors develops a simple baseline model consisting of VQVAE and an autoregressive Transformer, and show quantitative and qualitative experiment results for the generated image quality and answer using the baseline. They also evaluate the model's behavior when facing uncertain or imperfect queries.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please modify the draft to reflect what I mentioned above. Also, provide additional results using at lest two or three other VLMs.", "broader_impact_concerns": "Please add the broader impact of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QtPeEuvEsj", "reviewer": "Reviewer_RgT9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n-\tThe newly proposed datasets seem to have the potential to make a meaningful impact within this research field. Their intended design, which consist of image-text inputs and outputs including the prohibited requests, aligns well with the current trends on recent generative dialogue systems. Therefore, evaluations with these datasets are expected to be utilized in future studies. \n\n[Weaknesses]\n-\tEvaluating the ability to reject prohibited requests using the proposed dataset seems to be inappropriate. The exceptionally high performance of ‘Type Acc.’ (99.7%) for 'forbidden' in Table 6 suggests that recognizing prohibited requests may be a straightforward task. The test samples in the dataset may be too simplistic, diminishing the effectiveness to evaluate the challenges in rejecting prohibited requests.\n\n-\tThe paper does not seem to demonstrate a clear explanation for why Fruit-ATVC, one of the proposed datasets, does not include forbidden rules. Considering the ethical implications of rejecting specific requests on real images, it appears necessary for the paper to articulate the rationale behind the absence of forbidden rules within the Fruit-ATVC dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper points out the lack of a dataset for validating model performance in the textual-visual chat tasks and proposes two datasets specifically designed to evaluate the quality of image and text outputs in multimodal dialogue systems. The proposed datasets include CLEVR-ATVC, a synthetic dataset, and Fruit-ATVC, containing real images, both including visual and text-based inputs and outputs. For the first time in the field of multimodal dialogue systems, these datasets incorporate specific rules to enable the model to reject specific human requests.\nTo explore the capability of rejecting human instructions in a textual-visual chat task, the paper proposes an accountable text-based visual re-creation task using the proposed datasets. This task involves receiving text-image pairs and providing re-created images with language-based feedback including explanations for the rejection of requests in specific cases. To achieve this task, the paper suggests a two-stage training procedure; training a discrete VAE (dVAE) for compressing each image into short tokens and training a decoder-based transformer that takes the previously generated image tokens concatenated with text tokens as input for generating final outputs.\nThe experiments analyze the quality of the re-created images, the accuracy of responses, and the model’s behavior when handling imperfect and ambiguous user queries. Additionally, it compares the performance of VQVAE and VQGAN as image auto-encoders, revealing that VQGAN shows lower performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "-\tIt seems necessary to have test set requests that use different expressions but hold similar meanings to the forbidden rules present in the training set. This would enable the evaluation of whether the model robustly recognizes forbidden expressions.", "broader_impact_concerns": "-     It appears that there are no specific ethical issues that need to be particularly considered regarding the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "p6u4M5CZ8v", "reviewer": "Reviewer_KJPy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength \n- The task of generating images (while possibly rejecting human instructions) is a new and impactful task that should be studied. \n- The dataset is constructed to capture this task well. \n\n\nWeakness\n- Clarity of contribution. What distinguishes and motivates this particular multimodal dataset? From my understanding, it is that the dataset measures the quality of generated text and image. The generated text is only required since the authors introduce a new task of ATVC where the model can reject the request. If my understanding is correct, I think the paper should motivate this new task first and then the accompanying dataset and make this more clear in the intro. \n- Lacking GPT-4 results, though GPT-4 is the motivation. The motivation for constructing this multimodal dataset is to evaluate the performance of multimodal generative models like ChatGPT and GPT-4. However, there are no results of GPT-4 on this dataset. I understand GPT-4 requires paid access, but given that this is the motivation for the dataset in the first place, these results should be included to justify the motivation. Also, including results on GPT-4 would make this work more impactful and useful to others.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose a new task ATVC - a dataset where models have to generate images and decline generations if it is not possible to do so. They create 2 datasets for this task. They train a model similar to DALL-E on their dataset and analyze their results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major \n- clarify of contribution as listed above \n\nMinor \n- Add GPT-4 results \n\nQuestions \n- Are the answer types part of the ATVC framework? If so, it would make the paper easier to follow if the answer types are described in section 2.1 \n- “For example, six human instructions can be executed by the AI system and re-created the images accordingly, two instructions cannot be executed and two instructions are forbidden.” - Is the number of queries for each action type is randomly chosen for each image or always used the breakdown described in the example? \n- What is the difference between an instruction that is forbidden and an instruction that cannot be executed? An instruction that is forbidden also cannot be executed. \n- The two types of prohibition are cannot put object under/on top of other object? Is this prohibition based on physics? And why only these prohibitions of all the prohibitions? These two prohibitions seems arbitrary. \n- In the uncertainty experiments, the model handles it well by choosing one object (if multiple objects are correct). Shouldn’t the correct behavior be to clarify with the user which object is correct? \n- In the imperfect query for re-creation results, does this mean the model is not robust and cannot handle noisy inputs?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces new multimodal datasets/tasks that require models to reject human instructions that may be physically impossible. All three reviewers and the AE appreciated the novelty and importance of the task introduced, the effort in curating the dataset and tasks, and the thorough experiments to benchmark models for these tasks. The paper is recommended for conditional acceptance. I describe some of the remaining issues that need to be addressed below. Additionally, the authors did a commendable job in surveying the related work, and as such the paper is recommended for Survey certification. Congratulations to the authors!

Major comment:
- Given the limited set of actions/names/colors/etc, powerful transformers are expected to memorize these names and patterns, and it is likely that they won't generalize to unseen ones. As an example, see (Qian et al., 2021) where transformer-based dialog models do not generalize to unseen named entities. I suggest that the authors leave out some of the colors/names/actions/etc or create unseen ones for the test set only, and devise a clear recipe for testing generalization to unseen objects/actions/etc, which is one of the hallmarks of using foundation models for solving new tasks.

Qian, Kun, et al. "Annotation Inconsistency and Entity Bias in MultiWOZ." Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2021. 

Minor comments:
- End of page 5: Figure 2.4 -> Figure 3
- Please use \𝗰𝗶𝘁𝗲𝘁{} and \𝗰𝗶𝘁𝗲𝗽{} correctly or the paper will be hard to read. \𝗰𝗶𝘁𝗲𝘁{} should be used when the author name is intended to be part of the sentence, e.g., "X et al. (2024) introduced a new benchmark." \𝗰𝗶𝘁𝗲𝗽{} should be used when the citation appears in parentheses and is not read in the sentence, e.g., "There has been great progress in vision language models (X et al., 2024)."
- Please move the tables and images around so that they appear in the same order as they appear in the text, and preferably closer to where they appear in the text.
- Please remove the page break between Conclusion and References.
- Several lines in the annotation formats in the appendix don't fit the page. Please fix them.
Decision: Accept (poster)