DisCo: Improving Compositional Generalization in Visual Reasoning through Distribution Coverage
Authors: Joy Hsu
Jiayuan Mao
Jiajun Wu
Venue: Accepted by TMLR ()

Abstract: We present DisCo, a learning paradigm for improving compositional generalization of visual reasoning models by leveraging unlabeled, out-of-distribution images from the test distribution. DisCo has two components. The first is an iterative pseudo-labeling framework with an entropy measure, which effectively labels images of novel attribute compositions paired with randomly sampled questions. The second is a distribution coverage metric, serving as a model selection strategy that approximates generalization capability to test examples drawn from a different attribute combination distribution to the train set, without the use of labeled data from the test distribution. Both components are built on strong empirical evidence of the correlation between the chosen metric and model generalization, and improve distribution coverage on unlabeled images. We apply DisCo to visual question answering, with three backbone networks (FiLM, TbD-net, and the Neuro-Symbolic Concept Learner), and demonstrate that it consistently enhances performance on a variety of compositional generalization tasks with varying levels of train data bias. 


Jiayuan Mao Published in Transactions on Machine Learning Research (01/2023)
DisCo: Improving Compositional Generalization in Visual Reasoning through Distribution Coverage
Joy Hsu Department of Computer Science, Stanford University
joycj@stanford.ce edu
jayuanmmmmii.cc edu
Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology
Jiajun Wu
jiejunum@Sss edu
Department of Computer Science, Stanford University
Reviewed on OpenReview: https://ppenreeieuu..eee forr:
forum /a Bgankddda
Abstract
We present DisCo, a learning paradigm for improving compositional generalization of visual reasoning models by leveraging unlabeled, oul-of-distrbuutinn images from the lest distribu- tion. DisCo has two components. The first is an iterative pseudo-labeling framework with an entropy measure, which effectively labels images of novel attribute compositions paired with randomly sampled questions. The second a distribution coverage metrie, serving model selection strategy that approximates generalization capability to test examples drawn from different attribute combination distribution to the train set, without the use of labeled data from the test distribution. Both components are built on strong empirical evidence of the correlation between the chosen metric and model generalization, and improve distribution coverage on unlabeled images. We apply DisCo to visual question answering, with three backbone networks (FiLM, TbD-net, and the Neuro-Symbolic Concept Learner), and demonstrate that it consistently enhances performance on a variety of compositional generalization tasks with varying levels of train data bias.
1 Introduction
A long-standing goal of visual reasoning to build ma- chines that can respond queries about images flexible and general way as humans do. To achieve this, machines must contend with the combinatorial complex- ity of natural images and queries: scene has multi- ple objects, each object has collection of attributes, and objects form various spatial and functional rela- tionships. The combinatorial explosion of image spaces, together with practical data limitation downstream tasks, makes many learning problems ill-posed (Bienen- stock et al. 1996 Lake et al. 2017). In this paper, focus the compositional generalization to novel combinations of object attributes, generalizing from the reasoning of blue cubes and red cylinders to that of red cubes. This an important desideratum for machine learning systems: is impossible for any datasct to in- chude all possible combinations of object attributes for model training.
Compositional generalization
Labcled train data
Base
Prediction from base model: blue
Unlabeled data
DisCo
Prediction from DisCo: red
Figure The base VQA model in its original training paradigm trained on labeled data, while DisCo leverages wnlabeled, out-of-distribution im- ages from the test distribution to improve compo- sitional generalization performance.

---

Published in Transactions on Machine Learning Research (01/2023)
We address this problem by introducing DisCo, learning paradigm that leverages unlabeled, out-of- distribulion images from the test distribution to help visual reasoning systems better generalize compo- sitionally (See Figure D). Concretely, we focus on the task of visual question answering (VQA), though our framework is model-agnostic and can be used for a variety of vision domains that have combinatorial structures, such as in the tasks of referring expression comprehension, grounded instructions, and robotic manipulation Achlioptas et al., 2020 Shridhar et al. 2020 Choivlier-Boisvett et al.) 2019; Shridhar et al. 2022). Given a labeled train set of image, question, and answer triplets, and unlabeled images from the test distribution that contains novel attribute combinations, DisCo bootstraps visual reasoning model by iter- atively mining data instances derived from the unlabeled dataset. Starting from a base VQA model trained on a labeled, possibly biased dataset, our framework couples unlabeled images that are out-of-distribution with randomly sampled questions, and discovers pairs that are answerable. These newly-created data points are trained with equal weighting as the original labeled data points, and bootstrap learning: the model grad- ually labels more and more new image-question pairs, increasing distribution coverage on out-of-distibuutio image sets that contain novel attribute compositions, with individual attributes seen with labeled data but under different combinations.
Pscudo-labeling is particularly difficult in the visual question answering setting, as given an unlabeled cvestion-image pair, 1) there is a high probability of presupposition adherence failure (e.g., the question asks about the color of the cube in the image, but there is no cube), and 2) the current model may not have the capability to reason about the out-of-distribution image correctly. These two failure cases make pseudo-labeling images from the test distribution especially noisy. The effectiveness of our iterative learning paradigm is hence based on the empirical insight that, for pretrained visual reasoning model, the entropy of its predicted answer distribution correlates strongly with its accuracy on images with novel attributc compositions. This entropy metric approximates both presupposition adherence as well as compositional generalization accuracy, thus is crucial to DisCo choosing unlabeled samples to be pseudo-labeled.
Morcover, in the compositional generalization sctting, model tuning and model selection are challenging, as there is limited access to labeled data in the test distribution. Validation sets are in the same distribution as the train set, and hence unable to approximate test set performance, leaving methods unable to select for model checkpoints that best generalize to unseen data. To address this issuc, we propose a distribution coverage metric, which computes the percentage of unlabeled, test distribution that can be answered confidently by the current. VQA model. This distribution coverage measure well approximates model accuracy on generalization test splits with unseen attribute compositions, allowing us to effectively tune and select models without labeled data points from the test distribution.
distriddittiittionnn images drawn from the
We validate the effectiveness of our approach on biased versions of the CLEVR dataset (Johnson et al., 2017 created for compositional generalization in visual reasoning. Specifically, in addition to the original CLEVR CoGenT dataset, we also construct datasets that contain questions with referred objects as well one-hop relational questions, and demonstrate generalization improvement with DisCo compared to base VQA models with their original training paradigm. We demonstrate that DisCo consistently helps three VQA models, FiLM (Perez et al. 2018), TbD-net (Mascharka et al. 2018), and the Neuro-Symbolic Concept Learner (NS- CL; Mao et al. 2019), perform better on a test set with a distribution shift of attribute combinations from the train set. We also show that our framework outperforms other methods that leverage unlabeled data, including generative modeling and contrastive lcarning, on different levels of biases in training data, Our model exhibits an advantage in generalization to novel attribute combinations, and is a step towards contending with the combinatorial complexity of the visual world.
2
Related work
Compositional generalization. Prior work on improving the compositional generalization of visual rea- soning systems generally falls into two groups The first line of research leverages explicit structures of compositional concepts, such as visual grammars (Zhu & Mumfordd 2007 Chen et al. 2007], compositional embeddings (Misra et al. 2017). neural operators (Nagarajan & Grauman] 2018), neural module networks (Purushwalkam et al.] 22119) and causal graphs (Niu et al., [2021] Yang et al. (021b). The second line of research introduces additional supervision, such as the taxonomy of concepts improve model generaliza-
2

---

Published in Transactions on Machine Learning Research (01/2023)
tion (Han et al 2019). We present novel perspective on compositional generalization, which is to leverage unlabeled, out-of- distriiiinttonnn data from the test distribution.
Self-supervised learning for visual reasoning. Our iterative pseudo-labeling framework is also related to prior work on self-supervised learning for visual reasoning. Specifically, Kim et al. (2021) and Lin &
Parikh (0117) use active learning to select image and question pairs to be labeled. Askarian et al. (2021)
and Li et al. (2020) use curriculum learning to prioritize training data for visual reasoning models. Meds
such as Kim et al. (2119); Zhu et al. 2020);; Liu et al. (2018) apply adversarial self-supervised learning to overcome language priors in vasion-languaee models. Although some carlier work has explored similar entropy-based measures as ours, our work differs from them in two key aspects. First, our method studies a
different setting, where no additional labels will be requested
the unlabeled dataset. Second, in contrast to data efficiency or task performance, work shows that an entropy-based measure is especially beneficial for the compositional generalization capability of models in a test set with novel attribute combinations.
Semi-supervised learning. DisCo generally falls into the category of semi-supervised learning, whose idea is to leverage unlabeled data to improve model performance. Specifically, early work on pseudo-labeling [Nigam & Ghani 2000 Grandvalet & Bengio 2004) has drawn important theoretical connections between entropy-based self-training and expectation maximization algorithms. Prior work has also used signals such as high values in density- based clustering (Choi et al.) 2019) and label propagation [Iscen et al., 2019) to choose and infer pseudo-labeled examples, and introduced regularization techniques to learn better between- class separabilityy (Shi al. 2018)). A related work, Riive et al. (2021), proposes choosing pseudo-labels based on confidence and uncertainty of network predictions for classification, while DisCo tackles the complex VQA task with -uttof-distribution images from the specified test distribution that contain unseen attribute combinations. We refer readers to Van Engelen & Hoos (2020) and Yang et al. (2021a) as two recent comprehensive surveys. Our paper uses similar broader framework and focuses on the empirical evidence in visual reasoning domains to use unlabeled images to generalize in combinatorially complex settings.
3 Methods
We present DisCo as a method to leverage unlabeled, out-of-distribution image data from the test distribution for compositional generalization. At high level, DisCo is pseudo-labeling framework applied to VQA models, that iteratively learns to label images farther from the train data distribution. Intuitively, DisCo chooses unlabeled images from a test distribution of novel attribute compositions that the current reasoning model can effectively answer. As training progresses, DisCo selects more difficult question-image pairs, which increases distribution coverage on the unlabeled image set. We show that in the CLEVR dataset, after training, models can reason about simulated objects in scenes with new attribute compositions, not seen with labels during training.
In this section, we first describe our problem formulation (Section 3.1p and broader learning paradigm with unlabeled images (Section 3.2). We then discuss critical components in the framework. We describe image proposal methods for efficiently generating answerable images in DisCo (Section 3.3). Then, we propose an entropy-based threshold as a measure for accurate pseudo-labeling of out-of-distribation images from the test distribution (Section 3.4). Finally, we present a distribution coverage measure as an effective model selection strategy for compositional generalization (Section 3.5
3.1 Problem formulation
In this paper, we focus improving the object-level compositional generalization of visual reasoning models. Intuitively, objects in images are associated with many concepts, such as color, shape, and material. During training, the model may only see a finite number of possible concept combinations for objects. The goal of object-level compositional generalization is to let the model trained with limited labeled data generalize to novel object concept compositions. Our proposal is to leverage unlabeled, out-of-distribution images from the test distribution, which are significantly more available than human-annotated data, to improve the performance of visual reasoning models in combinatorially complex domains.
3

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm 1 The DisCo framework described in Section 3.2
Input:
Derain: the labeled train datasct; f: image proposal function derived from the unlabeled test datasct Drest; M(U, q 0): visual question answering model; entropy threshold.
Output: 0: M(,, 0).
1: Pretrain M with Dtrain- Track distribution coverage in C.
for i do
(+,44+aa)) Derain
e- + f()
Image proposal function Drest sample() or GAN generate(), Section 3.3 b Retrieve predictions from pretrained M.
6:
P+ t MM(4 (1;++9+;0)
7:
p- + -Mi0_.q=i0)
8:
entropy(p-) n then
> Entropy measure to select for pseudo label targets, see Section 3.4
9:
Update e with distribution coverage.
10:
Update M with zent(pt.a+) a+).
11: Update M with senl(p-, arg max(p-)).
Update M with equal weighting of + and
12: else
13:
Reject sample.
14: end if
15:
end for
16:
Choose checkpoint of M through e.
B Coverage used for model selection, see Section 3.5
DisCo is trained on a labeled dataset of VQA triplets, with each data point containing a visual scene, question, and answer; we denote this as (V+, 9+, (+,0+)+)+)++)) E Derain- The training dataset only contains a subset of attribute combinations of colors and shapes (e.g., blue cubes but not red cubes), while the test dataset contains objects of different color-shape combinations. Our training objective is thus to bootstrap from VQA model to iteratively improve the distribution coverage of out-of-distribution test examples.
3.2 Iterative pseudo-labeling
Our learning paradigm has three steps. First, assume a base visual question answering model M(e, 0), and train M to convergence on Dusin without modifications to the original training procedure. Second, we bootstrap the model on unlabeled, ind-of-distribution data from the test distribution with a proposed pseudo-labeling framework. Lastly, with the distribution coverage produced by DisCo, we perform model selection for checkpoint of M that generalizes best to novel attribute compositions. We descríbe this paradigm in Alg and show the overview in Figure
DisCo utilizes unlabeled images in Dtest through image proposal function derived from Duest (Alg L5). The proposal function can be random sampler of unlabeled images in Drest, or learned generative model, such as a generative adversarial network (GAN; Goodfellow et al. 2014), trained on the unlabeled images Drest- Either f yields unlabeled images e_ from the test distribution to be used in our framework.
In the iterative psoudo-labeling process, DisCo couples images v- with randomly sampled questions 4+ from the labeled train set (Alg L7); let p- denote the answer distribution produced by M, i.e., p- M(c_.9+:0) We use entropy measure to select confident predictions such that question-image pairs satisfy presuppositions and are answerable (Alg I L8). The (v-,4+) pair will be pseudo-labeled with arg max(p-), its own sharpened predictions (Alg 1 L11). During training, we keep track of the percentage of pairs (v_, that can be confidently answered by the model (Alg L9). This distribution coverage metric will be used for model tuning and checkpoint selection (Alg [ L16). At each pseudo-labeling step, given psoudo-labeled triplet (v_.q+, arg max(p-)) that satisfics the entropy metric, a labeled training triplet (u4, 94.44.000 a4) will also be sampled to be trained with equal weighting (Alg [ L10). This weight balancing allows the model to learn from both image distributions, and acts model correct regularization.

---

Published in Transactions on Machine Learning Research (01/2023)
Labelled images v.
Labelled answer
Pretraining
M
P.
xent(p,, 94)
Labelled questions 4+
"What color is the cube?"
Pseudo-
Labelled images
labelling
Labelled answer
Q4
Correctness regularization
Labelled questions
xent(p4, ..)
"What color cube?"
M
P_ (V4....)
Equally Weighted
Self-supervised learning
Unabelled images v-
xent(p_.,
Image proposal
Entropy measure
P
Reject sumple
function
Not satisfied
MM (+_,
Model tuning and selection
Max distribution coverage
M
Figure 2: Overview of the DisCo framework. At each pseudo-labeling step, labeled VQA triplets from the train distribution are used in equal weighting to pseudo-labeled VQA triplets, which contain images from the test distribution of novel attribute compositions.
3.3
Image proposals
DisCo is compatible with various kinds of unlabcled image distributions. In this paper, focus two prevalent choices for f: 1) direct sampling from unlabeled images in Dust, and 2) generation from a generative adversarial network (GAN) trained on unlabeled images in Drest-
We can directly sample unlabelod images from Drest and propose cach image as a potential pscudo-label target. We show in experiments that this method achieves strong performance on Drest, as well as on an unseen dataset that has the same distribution as Drest both of which are dut-nf-ddstriiution compared the labeled set Darain. A potential approach better cover test image distribution is to use generative models. We first train GAN unlabeled test images and make inferences of the trained model for image proposals, which essentially acts data augmentation. In this work, train unconditional StyleGAN v2 (Krrrs et al. 2020) on images from Drest We also show that both sampling methods achieve improved performance on a setting with an unknown test distribution, where unlabeled images are drawn from a distribution that different from both the training and test set.
3.4 Pseudo-label selection with entropy threshold
Pscudo-labeling out-of-distribution test examples in visual reasoning espocially noisy and challenging. This is only because unlabeled images contain novel altribute compositions correlated with visual challenges such occlusion, but also because randomly-samples questions may contain presuppositions that the images must satisfy. That is, the referred objects in the question may not exist in the image. When the vocabulary of concepts is large, most randomly sampled images and question pairs will be unanswerable. Without an effective measure for filtering the pseudo-labeled training data, the model will be corrupted with a high percentage of inaccurate labels that gives a poor signal in generalization.
Recall that there are two types of errors we want filter out. The first is presupposition failure, where the referred object in a+ does not exist in u.. For example, the question asks "what size the cylinder", but
5

---

Published in Transactions on Machine Learning Research (01/2023)
there is cylinder in the image. The second is questions that are difficult to answer due to limited training data other visual challenges. For example, questions regarding novel color-shape combination of a gray cylinder may be difficult to answer given the partial obstruction of the object. Below, we introduce and validate an entropy- measure, which effectively filters both types of errors.
In this work, we leverage a strong correlation between the entropy of p + M(u-,9+;:)) and compositional generalization accuracy. We demonstrate that this metric an effective measure for pseudo-labeling out-of- distribution images from the test distribution with unseen attribute compositions. The entropy is calculated from the softmax of model logits. Let denote the number of elements in the output vocabulary, the entropy is computed as M(X)= = Ei= p(x;) ((r:).
In Figure 3 we empirically verify this relationship between entropy and question-answering accuracy on images with novel attribute combinations. The left graph 3a) shows a cumulative entropy to accuracy plot on a log scale, with presuppasition adherence accuracy, prodiction accuracy, and prediction accuracy given presupposition adherence. Presupposition adherence accuracy (blue), the percentage of question-image pairs whose unlabeled image satisfies the sampled question's presupposition (i.e., the referred-to object exists). Prediction accuracy (red) is calculated such that the presupposition is satisfied and the predicted answer is correct. Prediction accuracy given presupposition adherence (purple), is the prediction accuracy of only pains that adhere to presupposition. The right graph 3b) depicts a cumulative entropy histogram.
Interestingly, we find that while prediction accuracy for images that satisfy presuppositions (purple) does not decrease significantly with entropy increase, the percentage of question- image pairs that violate presupposi- tions (blue) does decrease significantly. This suggests that our entropy measure well captures presupposition failure, and is hence effective and necessary for this learning paradigm. The black line both graphs indicates an approximately 30-th percentile entropy threshold, based the histogram of entropies, which yields 0.8207 prediction accuracy on aut-of-distribution test images and a 0.8506 presupposition adherence accuracy. Given images that passed the question presupposition at this threshold, 0.9648 were accurate.
3m)
Cumulative entropy to accuracy plot (log scale)
3b)
Cumulative entropy histogram (log scale)
3000
Appuracy
0.8
Coant
2000
0.7
Predictuus
1000
0.6
Ressupostion
Prod Prooeap
10-6
10
10-4
10-3
10-2
10-1
10"
10
10-6
10-3
10-1
10-1
10-2
10-1
109
10*
Entropy
Entropy
Figure 3: Relationship between entropy and compositional generalization accuracy and count. The base VQA model is trained on a biased train dataset; entropy and prediction accuracy evaluated on unlabeled images sampled from the test dataset paired with questions from the train set.
3.5
Model selection
For compositional generalization tasks, because we do not have access the ground-truth labels for images from the test distribution, there is no natural criterion for model tuning and model selection. A common practice of previous methods for model selection is through the maximization of validation set accuracy; however, the validation set has the same data distribution the train set, and thus is biased and not a good measure for test distributions. In this work, we propose a more effective measure by leveraging the unlabeled image set. for model tuning and model selection.
metric, which does not require any labeled examples from the test distribution. The high-level idea is to maximize the distribution coverage on the unlabeled dataset. Formally, recall that during training, the sampler produces pair (+) for pseudo-labeling,
Specifically, DisCo employs disstibbutioon-coeeng---ea

---

Published in Transactions on Machine Learning Research (01/2023)
where v_ is from the unlabeled distribution and a+ is from the train set. Our algorithm keeps track of the percentage of pairs (+- that are rejected from the entropy thresholding for each model (checkpoint). After training, we select the model with the maximum coverage of the test set, ie., the model that rejects the least number of pseudo-labeling pairs (V-,9+).
Empiricallyy, we validate that distribution coverage from our framework well approximates compositional generalization accuracy in the test set, while the standard validation set accuracy does not. In Figure 4 the left plot 4a) depicts the correlation between test set accuracy and validation set accuracy from model check- points, while the right plot 4b) depicts the correlation with distribution coverage. The Pearson correlation cocfficient between validation and test set accuracy is 0.2565. In this experiment, we can see that validation
set accuracies are mostly close to 1.0, while test set accuracies range from 0.90 to 0.98. In comparison, the distribution coverage value has a strong correlation with the test performance: the Pearson correlation coefficient is 0.6066, and thus more effective metric for model selection.
4n)
Correlation b/w accuracy test accuracy
4b)
Correlation coverage percentage test accuracy
0.98
0.98
Tost accuracy
0.96
Test Test accuracy
0.96
0.94
0.94
0.92
0.92
0.965
0.970 0.975
0.985 Validation accuracy
0.995
1.000
0.65
0.70
0.75
0.80
0.85
0.90
0.95
Coverage percentage
Figure 4: Correlation between test set and validation set accuracy (left plot, with Pearson correlation coefficient of 0.2565) as well as between test set accuracy and our distribution coverage metric (right plot, with a coefficient of 0.6066).
4 Experiments
We evaluate DisCo on a set of CLEVR datasets and three visual question-answering models FiLM (Perez et al. 2018), representative end-to-end attention-based approach, TbD-net (Mascharka et al. 2018), state- of-the-art neural module network-
approach, and NS-CL, a neuro-symbolie and object-centric approach. DisCo considerably improves the compositional generalization performance of base models compared to their original training paradigm or other semi-supervised learning approaches. Specifically, we compare our framework against two baselines that leverage unlabeled data: variational autoencoders (VAEs; Kingma & Welling 2014) and SimCLR. (Chen et al. 2020). For both baselines, we first train a VAE or a SimCLR model, and use their encoding networks to initialize the feature extractor of the visual reasoning model. Both baselines use the same amount and exact set of unlabeled data as DisCo. We describe our datasets and implementation details in Section 4.1 compare work against prior work in Section 4.2 and provide more ablation studies in Section 4.3 and more analyses in Section 4.4
4.1 Datasets & implementation details
In addition to the original CLEVR compositional generalization (CoGenT) dataset Johnson al. 2017) (relcased under the CC BY 4.0 license), we also report results on multiple CoGen datasets based on CLEVR. Specifically, we generate images with two to three objects per image, with CoGen split following that of the original CLEVR dataset. In this setup, there are two sets of colors, with the first as [gray, blue, brown, yellow], and the second as [red, green, purple, cyan]. CoGen split A contains cubes in the first set of colors,
We the term CoGen'T specifically refer to the original compositional generalization dataset introduced in Johnson et. al. (0117), and the term CoGen generally refer dataset splits (e.g., CoGen split A, CoGen split B) generated for testing compositional generalization.
7

---

Published in Transactions on Machine Learning Research (01/2023)
Table 1: DisCo performance compared to the original training paradigm and baselines, where 0.5% Ref (unk) is performance on an unknown test distribution, with the unlabeled dataset not drawn from the specified test distribution, and 0.5% Ref (unseen) is performance on unseen test set not exposed during training
0.0%
0.1%
0.5%
0.5%
0.5%
0.5%
0.5%
REF
REF
REF
REF (UNK)
REF (UNSEEN)
ONEHOP
COGENT
FILM
0.7589
0.7931
0.9265
0.9265
0.9251
0.9270
0.7859
FILM + VAE
0.7500
0.7993
0.9387
0.9396
0.9362
0.9228
0.7868
FILM SIMCLR
0.7520
0.8036
0.9288
0.9255
0.9257
0.9261
0.7926
FiLM + DisCo-S
0.7582
0.8363
0.9621
0.9609
0.9616
0.9469
0.8004
FiLM + DisCo-G
0.7760
0.8191
0.9545
0.9524
0.9510
0.9311
0.7979
Table 2: Comparison of DisCo with baselines the
Table 3:
Comparison of DisCo with baselines the NS-CL model. Performance reported on the original, unknown, and unseen lest set of 0.5% Ref.
TbD-net model. Performance reported the orig- inal, unknown, and unseen lest set of 0.5% Ref.
0.5%
0.5%
0.5%
0.5% REF
0.5%
0.5%
REF
REF (UNK)
REF
REF
REF
(UNSEEN)
(UNK)
(UNSEEN)
TBD
0.8993
0.8993
0.9027
NS-CL
0.7622
0.7622
0.7633
TBD + VAR
0.9018
0.9027
0.9025
NS-CL + VAR
0.7572
0.7583
0.7589
TBD + SIMCLR 0.9073
0.9054
0.9077
NS-CL + SIMCLR
0.7739
0.7727
0.7758
TbD DisCo-S 0.9206
0.9205
0.9197
NS-CL + DisCo-S
0.8024
0.8016
0.8027
TbD + DisCo-G
0.9189
0.9156
0.9145
NS-CL + DisCo-G
0.7820
0.7798
0.7793
and cylinders in the second. CoGen split B is reversed in the attribute combinations. Both CoGen split A and B contain spheres of all eight colors. We study a biased setup, where Dirain consists of images in CoGen split A with either zero or only a small percentage p of objects from CoGen split B. We show performance with train datasets of p E {0.0, 0.001, 0.005), evaluated on a full CoGen split B test set.
Based on the aforementioned image setup, we generate two additional datasets with different types ques- tions. The first the referred object dataset (Ref), with questions of the form, "What [attribute] is the [referred object]?" (e.g., "What material is the red object?"). Although this question template is simple, it reflects one of the most important problems in ggeudo-labeling for VQA: the satisfaction of question presup- positions. The second dataset consists of "one-hop P questions from the CLEVR dataset (OneHop), which consists of more complex relational questions, such as "How many red objects are to the left of the sphere?" or "There is a large object to the right of the metal thing: what is its color?". We also evaluate the models on a biased CLEVR CoGenT dataset (CoGenT) with the full set complex objects and questions.
We the official implementations of FiLM, TbD-net, and NS-CL along with their original hyperparameters in our framework. The GAN image proposal function is the unconditional StyleGAN2 (Karras et. al.) 2020). trained with the Adam optimizer of learning rate 0.002. We set our entropy threshold to be at the 30th percentile. Empirically, we find this threshold value to be robust to differences of around 10 percentile increase or decrease. For the more complex CoGen'T dataset, we lowered the entropy threshold to be at the 10th percentile to account for the naturally lower percentage of presupposition adherence. All models are trained on a single Titan RTX GPU.
4.2 Results
We train DisCo with FiLM on five datasots- three Ref datasets with varying bias levels, from 0.0% (fully biased), to 0. 1% biased, to 0.5% biased, as well as the 0.5% biased Onellop dataset and 0.5% biased ToGenT
2In hnp questions, the target objects are referred by relating another unique object.
8

---

Published in Transactions on Machine Learning Research (01/2023)
dataset. We show the results of our framework with both image proposal functions (DisCo-S as direct sampling and DisCo-G as GAN generation). Note that for VAE, SimCL,, and our framework DisCo, the VQA models see unlabeled test set images during pretraining or pseudo-labeling. Thus, for a fair comparison, we additionally report accuracy on a larger, test set that contains unseen images following the same distribution as the original test set, for 0.5% biased Ref. In addition, while we mainly focus on experiments that leverage unlabeled data from a known test distribution to improve the performance of that specific distribution shift, we also present results in a setting where the test dataset is from an unknown test distribution. In this setting, the unlabeled data is drawn from expanded set of distributions including the test distribution, but not solely consisting of samples that follow the test distribution. We set up this experiment with the training dataset from CoGen split A, the test dataset from CoGen split B, and the unlabeled dataset from the full CLEVR split consisting of both split A and B with all color-shape attribules.
Table T shows our results; DisCo outperforms the original training paradigm and both baselines. In addition, our framework, with both image proposal functions, is robust to both images from an unknown test distribution as well as unseen images in known test distribution. By comparing the two image proposal approaches across experiments, DisCo-S achieves better performance than DisCo-G. We conjecture that this is because, in the sampler method, the model is trained with the exact set of real, unlabeled test images that we evaluate with. Morcover, in the fully biased Ref experiments, the GAN image pro- posal function outperforms the direct sampler. We attribute this to the GAN's generation of more diverse unlabeled images, which better cover the image space of possible camera angles and lighting conditions, allowing DisCo to improve model performance when there are few signals from the labeled VQA dataset. The fully biased Ref experiments illustrate a difficult and important setting; there are often cases where it is useful to perform well without any labeled data in the compositional generalization setting, e.g. when we only have access to objects with a specified set of attributes during training and do not know the test distribution, when we need to adapt to new test distributions without labels.
Table 4: Ablation of DisCo with di- rect sampler, trained on the 0.5% bias Ref. (PL = lseudo-labeling, EM entropy mcasure, CS = cov- erage selection).
FILM
BASE BASE PL. BASE PL+EMM
0.9265
0.6749
0.9605
PASE+PL+EM-
0.9621
Integration with TbD-net and NS-CL. DisCo can be integrated with other visual reasoning models, too. In this paper, we implement TbD-net and NS-CL with DisCo, to showcase fiexibility. The results on ThD-net are presented on the 0.5% Ref split in Table 2 and results on NS-CL in Table 3 The observations are consistent with the FiLM-based experiments. DisCo similarly outperforms the original training paradigm and basclines by noticcable margin.
4.3 Ablations
In Table WC present ablation studies of our framework with direct sampler as the image proposal function. First,, we see that directly adding the pseudo-labeling (PL.) module (i.e., pseudo-label all images in the test set with the pretrained FiLM model without any thresholding) significantly degrades model accuracy. Second, adding our entropy mcasure (EM) improves compositional generalization performance. This finding is consistent with our visualizations of the correlation between entropy and accuracy in Fig. Moreover, adding coverage-based selection (CS) further improves test accuracy.
Table 5 zooms in into the candidate model selection methods. Specifically, we compare test accuracy from our coverage- coverage-based selection strategy (Coverage) with test accuracy from the standard, validation accuracy-based selection strategy (Val acc). Our framework shows consistent advantage across FILM, TbD-net, and NS-CT, VQA models. Note that the distribution coverage metric is not directly applicable to pretraining- based basclines (VAE and SimCLR) because they do not compute pseudo-labels for test images.
4.4 Analyses
Qualitative examples We qualitatively analyze the performance gain brought by our framework. Figure 5 (top row) shows two example images in the CoGen split B test set. We apply the base FiLM model, the
9

---

Published in Transactions on Machine Learning Research (01/2023)
Table Comparison of test accuracy with baselines on model selection strategy, trained on 0.5% Ref.
FILM
TED-NET
NS-CL
VAL ACC COVERAGE
VAL. ACC COVERAGE
VAL ACC COVERAGE
BASE
0.9265
N/A
0.8993
N/A
0.7622
N/A
BASE
VAE
0.9387
N/A
0.9018
N/A
0.7572
N/A
BASE SIMCLR
0.9288
N/A
0.9073
N/A
0.7739
N/A
Base + DisCo-S
0.9605
0.9621
0.9000
0.9206
0.7887
0.8024
Base + DisCo-G
0.9508
0.9545
0.9020
0.9189
0.7780
0.7820
FiLM model trained with DisCo-S (direct sampling), and the FiLM model trained with DisCo-G (GAN) the fully biased Ref dataset and retrieve their predictions.
In the top left example, the test set question asks "What color is the cylindor?" the brown cylinder in the image. Brown cylinders are never seen in the fully biased train set, hence the FiLM model answers incorrectly with red, unable to identify the referred object. DisCo with direct sampling also produces a wrong answer, likely due to lack of signal from the labeled image that DisCo with GAN is able to answer correctly with brown and show better compositional generalization. We conjecture that this is due to the GAN covering a denser image distribution. In the bottom row of Figure
bootstrap visual reasoning. In this case, we see
5 we see two similar images the left image taken from the train set with broun cube and cyan cylinder, and the right image generated by our GAN with a brown cylinder in a closely aligned scene. We hypothesize that is diverse image proposals like this that enable visual reasoning models to better learn the concept. a brown cylinder.
In the top right example of Figure similarly, purple cubes never seen in the fully biased train set, thus the FiLM model answers completely incorrect color, while DisCo with both image proposal functions is able to generalize to this novel attribute combination.
Limitations DisCo provides a framework for boot- strapping visual reasoning; however, it relies on the in- ductive bias of convolutional networks compositionally generalize. Without priors visual attributes, given red cube (a novel color-shape combination not seen in the train set), the model could learn that the object is nei- ther red nor cube as red could be learned as a color only cxisting on cylinders and spheres, while cube could be learned as a shape that is only paired with colors that are gray, blue, brown, or yellow, as seen from the labeled train set. Assumptions about inductive biases of visual attributes in convolutional networks allow our framework to learn generalization and hence DisCo is limited to, and also especially powerful, in the vision domain.
What color is the cylinder? What color cube?
Ground truth: brown FiLM: red
Ground truth: purple
FiLM: blue
FiLM DisCo-S: red FiLM DisCo-G: brown
FiLM DisCo-S: -S purple
FiLM DisCo-GG G: purple
We do not report test results on more real-world datasets due to the lack of datasets that evaluate compositional generalization. This leaves several open questions for fu- ture work, such as how DisCo may perform with more variability in the dataset, and with base model that cxhibits poor generalization ability and is challenging to bootstrap from. Additionally, the effects of utilizing a thresholding metric based on variants of the softmax score (Hendrycks & Gimpel 2017 Liang et al. 20188) and the energy score (Liu 2020 may also be fruitful to plore future work.
Image from train set
GAN image proposal
Figure 5: Top row: two prediction examples from FiLM, DisCo-S, and DisCo-G on fully biased Ref. Bottom row: two closely aligned images from the train set and from the GAN proposal function.
10

---

Published in Transactions on Machine Learning Research (01/2023)
5 Conclusion
We have presented DisCo, framework for improving compositional generalization by leveraging unlabeled, cud-of-distrbuution images from the test distribution through iterative pseudo-labeling. We studied and proposed the entropy measure as an effective signal for presupposition adherence and pseudo-label accuracy in out-of-distribution test examples, and also introducod the distribution coverage model selection strategy, which well approximates test performance on novel attribute combinations while only requiring unlabeled data. We demoustrated our framework's ability to improve compositional generalization performance, and showed potential for future work to leverage unlabeled images to achieve generalization in evaluation regimes with combinatorial complexity.
Broader Impact Statement
Our work shows the importance of learning unbiased concepts from datasets with better distribution coverage. We expect minimal negative societal impact, however, when using our framework, it's important to ensure that the unlabeled dataset itself has enough distribution coverage to minimize dataset bias. Our goal is for DisCo to help models perform well in data-limited environments for good.
Acknowledgments
We thank Eric Chan for providing valuable feedback on the paper. This work is in part supported by the Stanford Institute for Human-Centered AI (HAI), Toyota Research Institute (TRI), NSF RI #42211258, ONR MURI N00014-22-1-27400 and Analog, JPMC, Meta, Salesforce, and Samsung. JH is supported by the Knight Hennessy fellowship and the NSF Graduate Research Fellowship.
References
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3D: Neural Listeners for Fine-grained 3D Object Identification in Real-world Scenes. In ECCV, 2020.
Narjes Askarian, Ehsan Abbasnejad, Ingrid Zukerman, Wray Buntine, and Gholamreza Haffari. Curricu- lum Learning Effectively Improves Low Data VQA. In Annual Workshop of the Australasian Language Technology Association, 2021.
Elie Bienenstock, Stuart Geman, and Daniel Potter. Compositionality, MDL Priors, and Object Recognition. In NewrIPS, 1996.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framowork for Con- trastive Learning Visual Representations. In ICML, 2020.
Yuanhao Chen, Long Zhu, Chenxi Lin, Hongjiang Zhang, and Alan L Yuille. Rapid Inference Novel and/or Graph for Object Detection, Segmentation and Parsing. In NewrIPS, 2007.
Maxime -hevveier-Boisvert,, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Baby A Platform to Study the Sample Efficiency of Grounded Language Learning. In ICLR, 2019.
Jaehoon Choi, Minki Jeong, Tackyung Kim, and Changick Kim. Pseudo-labeling Curriculum for Unsuper- vised Domain Adaptation. In BMVC, 2019.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In NewrIPS, 2014.
Yves Grandvalet and Yoshua Bengio. Semi-supervised Learning by Entropy Minimization. In NewrIPS, 2004.
Chi Han, Jiayuan Mao, Chuang Gan, Joshua B. Tenenbaum, and Jiajun Wu. Visual Concept Metaconcept Learning. In NewrIPS, 2019.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Dan Hendryeks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-distribatioo Examples in Neural Networks. In ICLR, 2017.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label Propagation for Deep Semi- Supervised Learning. In CVPR, 2019.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Flementary Visual Reasoning. In CVPR, 2017.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and Improving the Image Quality of Stylegan. In CVPR, 2020.
Dong-Jin Kim, Jinsoo Choi, Tac-Hyun Oh, and In So Kweon. Image Captioning with Very Scarce Supervised Data: Adversarial Semi-Supervised Learning Approach. In EMNLP, 2019.
Dong-Jin Kim, Jae Won Cho, Jinsoo Choi, Yunjae Jung, and In So Kweon. Single-Modal Eutropy based Active Learning for Visual Question Answering. In BMVC, 2021.
Diederik P Kingma and Max Welling. Auto-encoding Variational Bayes. In ICLR, 2014.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building Machines that Learn and Think like People. Behav. Brain Sci., 40, 2017.
Qing Li, Siyuan Huang, Yining Hong, and Song-Chun Zhu. A Competence-Aware Curriculum for Visual Concepts Learning via Question Answering. In ECCV, 2020.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the Reliability of Out-of-distribution Image Detection in Neural Networks. In ICLR, 2018
Xiao Lin and Devi Parikh. Active Learning for Visual Question Answering: An Empirical Study. (SS:::1111001988 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based Out-of-distribution Detection. In NewrIPS, 2020.
Xihui Liu, Hongshong Li, Jing Shao, Dapeng Chen, and Xiaogang Wang. Show, Tell and Discriminate: Image Captioning by Self-Retrieval with Partially Labeled Data. In ECCV, 2018.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences from Natural Supervision. In ICLR, 2019.
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning, In CVPR, 2018
Ishan Misra, Abhinav Gupta, and Martial Hebert. From Red Wine to Red Tomato: Composition with Context. In CVPR, 2017.
Tushar Nagarajan and Kristen Grauman. Attributes as Operators: Factorizing Unseen Attribute-Object Compositions. In ECCV, 2018.
Kamal Nigam and Rayid Ghani. Analyzing the Effectiveness and Applicability of Co-training, In CIKM, 2000.
Yulei Niu, Kaihua Tang, Hanwang Zhang, Zhiwu Lu, Xian Xian-Sheng Hua, and Ji-Rong Wen. Counterfactual VQA: A Cause-Effect Look at Language Bias. In CVPR, 2021.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Rea- soning with General Conditioning Layer. In AAAI, 2018.
Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Mare' Aurelio Ranzato, Task-Driver Mod- ular Networks for Zero-Shot Compositional Learning. In ICCV, 2019.
12

---

Published in Transactions on Machine Learning Research (01/2023)
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In Defense of Pseudo- Psendo-labeling: An Uneertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning. In ICLR, 2021.
Weiwei Shi, Yihong Gong, Chris Ding, Zhiheng MaXiaoyu Tao, and Nanning Zheng. Transductive Semi- Supervised Deep Learning using Min-Max Features. In ECCV, 2018.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. lation. In CoRL, 2022.
Cliport: What and Where Pathways for Robotic Manipu-
Jesper E Van Engelen and Holger H Hoos. A Survey on Semi-Supervised Learning. MLJ, 109 2020.
Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A Survey on Deep Semi-Supervised Learning. arXiv:2103.008: 2021a.
Xu Yang, Hanwang Zhang, and Jianfei Cai. Deconfounded Image Captioning: A Causal Retrospect. IEEE TPAMI, 2021b.
Song-Chun Zhu and David Mumford. A Stochastic Grammar of Images. Now Publishers Inc, 2007.
Xi Zhu, Zhendong Mao, Chunxiao Liu, Peng Zhang, Bin Wang, and Yongdong Zhang, Overcoming Language Priors with Self-Supervised Learning for Visual Question Answering. In IJCAI, 2020.
A Appendix
The supplementary material is organized as the following. First, in Section A.1 we provide released code, and in Section A.2 we describe our dataset construction. Section A.3 shows results from our image proposal functions, while Section of the distribution coverage metric. We show additional experiment results on settings with an unknown test distribution in Scction A.6 In Scction A.7 we provide details on bascline implementations. Last, in Section A.8and and Section examples.
A.4 and Section A.5 demonstrate ablations on entropy thresholds and robustness
A.9
report quantilative analyses of our model performance well qualitative
A.1 Code release
Code for DisCo with the FILM model can be found: FiLM codebase htttss//github.com httsss//gibubbcoooo/aaaaieee oethanjperezfilnn We want to highlight that when using DisCo, is important to ensure that the unlabeled dataset has enough distribution coverage to minimize dataset bias.
fttps://github.oom/ion how.:oyyssu000/////sccc based on the
A.2 Dataset
We generate additional CLEVR datasets of two to three objects based on the CoGen (compositional general- ization) split introduced in Johnson et al. (2017). As recap, in this setup, there are two sets of colors, with the first [gray, blue, brown, yellow], and the second as [red, green, purple, cyan]. CoGen split A contains cubes in the first set of colors, and cylinders in the second. CoGen split B is reversed in the attribute combinations. Both CoGen split A and B contain spheres of all eight colors. In our construction, the train set CoGen split A consists of 8,000 images, and the validation set CoGen split and the test set of CoGen split consist of 2,000 images each. The larger, unsoen test set consists of 8 8,000 images. The Ref datasets include questions of the form "What [attribute] is the [referred object]?", while the OneHop dataset inchudes one-hop relation questions defined in Johnson et al. 2017
13

---

Published in Transactions on Machine Learning Research (01/2023)
A.3 Image proposal examples
In Figure provide examples of image proposals from our two functions direct sampling and GAN generation.
Both capture range of novel attribute combinations not in the labeled train set images.
Direct sampling image proposals
GAN generated image proposals
Figure Image proposals from direct sampling and GAN generation, used as unlabeled image input DisCo.
A.4 Entropy thresholds
In this ablation study, we validate the robustness DisCo with respect to different. entropy thresholds. We show that DisCo yields strong results different values of the hyperparameter. See Table [ for results of FiLM with DisCo-S on entropy thresholds at varying percentiles. In addition, we present test accuracy curves for each run, compared to the base FiLM model in Figure The black line indicates where DisCo begins pseudo-labeling from the pretrained FiL. model. We observe that DisCo considerably improves upon the base VQA model. These results also show the robustness of our method against different random seeds. For each run shown, the network weights, as well as the data samples, are generated based on different random seeds, but the improvements are consistent.
Table 6: Comparison of different percentiles of entropy thresholds on the 0.5% biased Ref dataset.
30TH
35TH
40TH
45TH
FILM + DisCo-S 0.9612
0.9616
0.9667
0.9643
A.5 Distribution coverage
We demonstrate that our distribution coverage measure well approximates test accuracy, even when the VQA model is decreasing in performance. In Figure g we report the test accuracy and distribution coverage curves of FiLM + DisCo-S experiment at too high an entropy threshold, where performance quickly degrades. In Figure 0 we present the curves FiLM + DisCo-G experiment that slightly degrades in performance before recovering. The Pcarson correlation cocfficient between test accuracy and distribution coverage for these experiments are 0.6585 and 0.7987, respectively, both showing highly correlated values allowing for effective model selection.
We see that DisCo docs not tend to oversample or propagate errors; instead, it lowers the number of samples chosen when compositional generalization performance decreases. We conjecture that this is due to our model correctness regularization of training on labeled triplets, which contain different combinations of attributes. When the base VQA model is trained on corrupted pscudo-labels, the model no longer confident. in its predictions given conflicting signals from labeled and pscudo-labeled triplets, and hence fewer samples are chosen, intended in our framework.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Test accuracy plot
30th 35th
0.96
40th
Bose
Test accuracy
0.94
0.92
0.90
5
10
15
20
25
30
35
Ten thousand steps
Test accuracy curves of the base FiLM model (red), and DisCo-S at different percentiles of entropy thresholds.
Figure 7:
Tost accuracy plot
Distribution coverage plot
0.96
0.98
0.96
Distribution coverage
0.94
Test aocuracy
0.92
0.90
0.90
0.88
0.88
0.86
0.84
0.84
0.82
0
20
40
60
80
100
120
20
40
60
80
100
120
Thousand Steps
Thousand Steps
Test accuracy and distribution coverage curves of an experiment with large decrease in model performance, with Pearson correlation cocfficient of 0.6585.
Figure
A.6 Experiments with unknown test distributions
Although mainly focus settings where the unlabeled dataset is drawn from known test distribution, we also show experiments settings where the test distribution is unknown. We report results settings where the unlabeled dataset is superset of the unknown test distribution (UNK in the main text), as well as on settings where the unlaboled dataset is a subset of the unknown test distribution (UNK-SUB below). the latter setup, we train DisCo on labeled CoGen split A, leverage unlabeled CoGen split B, and test model performance on the full CLEVR split. We see Table 7 Table X and Table that DisCo shows consistent performance gain compared to base and baseline models in this setting for all three VQA backbones.
A.7 Baseline implementation
We implemented two pretraining approaches that leverage unlabeled images. Specifically, we first train a variational autoencoder (VAE; Kingma & Welling 2014 and SimCLR (Chen et al.) 2020 model and use their encoding networks to initialize the feature extractor of the visual reasoning model. The VAE implementation based on hattps://github.ccm APyrro//PTTTooch VAE, and the SimCLR implementation on https://gtthub.com http:///b..ooopppppeeee////lliia For both methods, we retrieved image features from 1esNet110
AntixK PPyTorch-
15

---

Published in Transactions on Machine Learning Research (01/2023)
Test accuracy plot
Distribution coverage plot
0.94
MY n
Distribution coverage
0.5
0.92
Test accuracy
0.90
0.4
0.88
0.3
0.86
0.2
0.84
0.82
0
50
100
150
200
50
100
150
200
Thousand Steps
Thousand Steps
Figure 9: Test accuracy and distribution coverage curves of an experiment with a slight decrease in model performance, with Pcarson correlation cocfficient of 0.7987.
Table 7: DisCo on the FILM
Table 8: DisCo on the TbD-nct
Table 9: DisCo on the NS-CL model. Performance reported on test set of 0.5% Ref from unknown test distribution.
model. Performance reported test set of 0.5% Ref from unknown test distribution.
model. Performance reported on test set of 0.5% Ref from unknown test distribution.
0.5% REF
0.5%
0.5%
REF
REF (UNK-SUB)
(UNK-SUB)
(UNK-SUB)
FILM
0.9634
TBD TBD VAE
0.9506
NS-CL NS-CL VAE
0.8810
FILM +
0.9683
0.9495
0.8789
FILM + SIMCLR
0.9635
TBD + SIMCLR 0.9510
NS-CL + SIMCER
0.8872
FILM + DisCo-S
0.9798
TbD + DisCo-S 0.9582
NS-CL + DisCo-S
0.9011
FILM + DisCo-G 0.9745
TbD + DisCo-G 0.9536
NS-CL + DisCo-G
0.8902
as input to the VAE and SimCLR model. We add in encoding layers of [Conv2d, BatchNorm2d, and LeakyReLU] to both networks, and feature-level reconstruction and rastive loss to supervise learning. After pretraining, use the newly added layers the additional encoding for our VQA models.
A.8 Quantitative analyses
In Table 10
we examine the test set accuracy per color-shape combination of referred objects in the fully biased Ref dataset. We compare FiLM with DisCo-S and DisCo-G, and report metrics attribute combi- nations not seen in the labeled train set.
Interestingly, the best-performing model with GAN-generated image proposals performs significantly better on cubes, with a decrease in accuracy on some cylinders in comparison to FiLM and direct sampling. We might instead expect uniformly increased performance on color-shape combinations, but empirical results reveal that models that compositionally generalize may learn do so better on some of novel attribute combinations.
A.9 Qualitative examples
In Figure 10 we show examples of VQA pairs at different values of entropy output by the model. We see that low entropy examples both satisfy presuppositions and correctness, while at higher entropy value,
16

---

Published in Transactions on Machine Learning Research (01/2023)
Table 10: Comparison of test accuracy per color-shape combination of the referred object, trained on the fully biased Ref dataset.
FILM
FILM DisCO-S FILM + DIsCO-G
RED CUBE
0.6794
0.6589
0.7658
GREEN CUBE
0.6399
0.638
0.7173
PURPLE CUBE
0.6478
0.6478
0.7676
CUBE
0.6313
0.6313
0.6774
GRAY CYLINDER
0.6481
0.6490
0.6292
0.6431
0.6450
0.5752
BROWN CYLINDER
0.6654
0.6774
0.6719
YELLOW CYLINDER
0.6385
0.6459
0.6106
RED SPHERE
1.0
0.991
0.9859
GREEN SPHERE
0.9937
0.9958
PURPLE SPHEEE
0.9958
0.9958
0.9917
CYAN SPHERE
1.0
1.0
0.9981
GRAY SPHERE
0.9948
0.9923
0.9794
BLUE SPHERE
1.0
1.0
0.9961
BROWN SPHERE
1.0
1.0
0.9981
YELLOW SPHERE
1.0
0.9980
0.9940
only presuppositions are satisfied but the answer predicted by the model for out-of-distribution objects is incorrect, and the highest values of entropy there exists presupposition failure. In this way, DisCo is able to choose suitable pairs to be added to training.
17

---

Published in Transactions on Machine Learning Research (01/2023)
We also present additional qualitative examples of predictions from FILM, DisCo-S, and DisCo-G. In Fig- ure
examples where DisCo-S and DisCo-G outperform FiLM (first row) and examples where DisCo-G outperforms DisCo-S and FiLM (second row), on the fully biased Ref dataset.
What eyan thing?
What shape rubber thing? What shape the small thing? What material the cylinder? What color sphere?
Entropy: 0,0000000009
Entropy: 0,0000042114
Entropy: 0.000101 9438
Entropy: 0.0019916531
Entropy: 0.6848732829
Presupposition: /
Presuppesition: /
Presupposition: /
Presupposition: /
Presuppasition:
Correct: /
Currect:
Currect: X
Currect: x
Correct: X
Ground truth: small
Ground truth: cube
Ground truth: cylinder DisCo-S: cube
Ground truth: metal DisCo-S: rubber
Ground truth: DNE
DisCo-S: small
DisCo-S: cube
DisCo-S: green
Figure 10: Qualitative examples VQA pairs at different levels of entropy.
What material is the cube?
What color is the large thing? What size the cylinder?
What material is brown thing?
What color the cylinder?
Ground trut: rubber
Ground truth: Fil M: brown
Ground truth: large
Ground truth: metal
Ground gray
Fil M: metal
FiLM: small
FiLM: rubber
FiLM: eyan
HiLM DisCo-S: rubber
FiL M DisCo-S: red
FiLM DisCo-S: large
HiLM + DisCo-S: metal
FiLM DisCo-S; gray
DisCo-G: rubber
FiLM DisCo-G: red
FiLM DisCo-Ga G: large
FILM DisCo-G: metal
fiLM + DisCo-G: gray
What shape the purple thing?
What color the cube?
What color is the cylinder?
What size cube?
What material small thing?
Ground truth: cube FILM: cylinder
Ground FiLM: gray FiLM DisCo-S: S: gray FILM DisCo-G: red
Ground trath: yellow FiLM: purple
Ground trut: large
Ground truth:
FiLM: small
FILM: rubber
FILM DisCo-S: cylinder
FILM DisCo-S: purple
FiLM DisCo-S: small
FiLM DisCo-S: rubber
FILM DisCo-G: cube
FILM DisCo-G: yellow
FILM DisCo-G: large
FILM DisCo-G: metal
Figure 11: Qualitative examples of FiLM, DisCo-S, and DisCo-G the fully biased Ref dataset.
18


Reviews:
[{"review_id": "HsoqaAaBWH", "reviewer": "Reviewer_5bCf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This work studies a novel setting where the VQA model can have access to unlabeled images and extract learning signals from them. Different from prior work, this new setting explores how to achieve better compositional generalization using unlabeled data resources. Considering the efforts in collecting human annotations for the VQA task, this unsupervised learning paradigm is practical and helpful for future research.\n\n- This work has conducted extensive experiments to validate the proposed method DisCo. Three base VQA models (FiLM, TbD-net, and NS-CL), two baselines (VAE and SimCLR), and various dataset settings have been evaluated and compared. DisCo can consistently improve the base VQA models.\n\n- The writing is clear and easy to follow.\n\nWeaknesses:\n\n- This work claims to improve the compositional generalization of VQA models, which means that the model can better generalize to images that contain objects with new compositions of attributes (e.g., shape+color in this work). However, this claim does not seem to align well with the current training and evaluation protocol: At training time, unlabeled **test images** are exposed to DisCo (and other unsupervised learning baselines as well), and thus the model can learn from the different composition distribution in the test images, even though these images are not labeled. During evaluation, the model is tested with exactly the same test images or new images drawn from the same test distribution (“Ref unseen” in the experiments). In other words, the model has already learned from the test-time data distribution during training before testing. This experiment setup seems to be weak for supporting the claim of improved generalizability. In practice, it is not likely to know the test-time compositions while training the VQA model. A stronger setting might be to evaluate the learned VQA model with another distribution that is different from both the labeled and unlabeled images.\n\n- The entropy-based metric for thresholding pseudo-labels is highly related to prior work in out-of-distribution (OOD) detection, such as [1][2][3]. A comprehensive comparison between the proposed entropy-based metric and the related methods is necessary. Moreover, on this specific task of VQA, is the proposed metric better than well-established OOD detection methods like [3]? Further experiments that compare the proposed metric and previous OOD detection methods might be helpful.\n\n- This work is mainly established on empirical observations. Some theoretical insights could further strengthen the proposed strategies.\n\n- The performance gain seems limited. For example, as shown in Table 1, the performance improvement brought by DisCo (from 0.7589 to 0.7760) is less than simply introducing only 0.1% labeled data from the target distribution (from 0.7589 to 0.7931). More clarification might be helpful for readers to understand how significant this improvement is.\n\nReferences:\n\n[1] Dan Hendrycks, and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.\n\n[2] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks. In ICLR, 2018.\n\n[3] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS, 2020.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work aims to improve the compositional generalization of visual question answering (VQA) models by leveraging unlabeled, out-of-distribution image samples. This work, DisCo, proposes two techniques for better exploitation of unlabeled images: 1) an entropy-based metric for filtering pseudo-labels on unlabeled images that are not appropriate for self-training, and 2) a distribution coverage metric for selecting a model checkpoint that approximately minimizes generalization errors. Both of the techniques are based on empirical observations. The proposed learning paradigm DisCo successfully improves various VQA models (including FiLM, TbD-net, and NS-CL) on the CLEVR positional generalization dataset, and outperforms unsupervised representation baselines such as VAE and SimCLR.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Additional explanation and experimental results can help to address the weaknesses as detailed above. In addition to those, the following minor changes can be made:\n\n- The “out-of-distribution images” introduced in this work contain objects with different **compositions** of attributes, but these individual attributes (e.g., colors or shapes) have already been seen in the labeled data. It might be better to clearly explain when this work first introduces out-of-distribution images.\n\n- In the introduction, it is mentioned that “our framework is model-agnostic and can be used for a variety of vision domains that have combinatorial structures.” Could the authors provide some specific examples other than VQA?\n\n- Figure 2: To be consistent with the notations in Section 3.2, please use subscripts for $+$ and $-$ signs.\n\n- Figure 3: To be consistent with the index of this whole figure, please rename the sub-figures as 3a) and 3b). Similar issue in Figure 4.\n\n- Figure 4-left: Why do some checkpoints (points on the left side) have validation accuracy close to 0.0, while having test accuracy close to 1.0? In the text, it is mentioned that “validation set accuracies are mostly close to 1.0.” Please clarify.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ma11ekqlh0", "reviewer": "Reviewer_h5vg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Technically sound and relevant to TMLR: I can find no obvious red flags with regard to technical correctness. The proposed entropy-based method for pseudo-label generation, or generation of samples using GAN, and the model selected using \"distribution coverage\" are all technically sound and can be expected to work in the manner proposed by the paper. The topic is also highly relevant to TMLR and should be of interest to a good subsection of its readers.\n\n2. Thorough experiments: The experiments are clearly presented and are thorough enough to allow readers to know the effect of different choices used in the model. I do not see any holes or additional experiments that I would request. I do have a concern about whether the current experiments are enough to validate the main claims in the paper but they are a slightly orthogonal issue which I discuss below. This should not change the range of experiments presented, and I would still be totally okay with the same set of experiments repeated with a corrected training setup.\n\nWeaknesses:\n\nThere is just one main weakness to me in the correct draft. \n\n1. There are two main ways that the paper introduces unlabeled \"OOD\" samples to the proposed pipeline: A) direct sampling from unlabeled images in Dtest, and B) generation from a generative adversarial network (GAN) trained on unlabeled images in Dtest. Both of these are quite problematic and are antithetical to OOD testing. To this reviewer, the whole point of OOD testing should be to estimate performance on *unknown* but different distribution to training. In practice, this is approximated by a differently distributed test set, but there is more than a single way to have OOD samples. Here, the paper studies OOD in terms of color distribution shift but it could have other ways to do OOD. It is totally okay to use unlabelled data from *different* distributions from training, but it is not okay to (only) use the one that is literally drawn from the same distribution as the test. That is not just OOD, but rather one specific distribution, that we use to measure OOD under a controlled setting. To properly validate the claims against the proposed technique, the work should focus on creating/sampling images from all possible distribution shifts. To me, the current setup invalidates the main claims of the paper that the use of unlabeled/pseudo-labeled data can improve OOD generalization. See more details below on \"requested changes\".\n\nThe same parallels can be said for model selection \"The high-level idea is to maximize the distribution coverage on the unlabeled dataset.\" is currently more like \"...coverage on unlabeled dataset sampled from one specific distribution, which also happens to be the same as our test set\", which is problematic in the same way as above! \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes to improve out of distribution performance by two main mechanisms. First, the use of pseudo-labeling and second, and more importantly, the use of unlabeled out of distribution data to improve distribution coverage. The experiments are conducted on the popolar CLEVR dataset with familiar backbone networks for CLEVR, and the results show improved compostional generalization to OOD samples in varying levels of distribution shifts introduced. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Both direct sampling and generative model should be sampled from an expanded set of distributions **INCLUDING** the test distribution, not solely consisting of samples that follow the test-set distribution. There should at least be a variant where the model should not have knowledge of the test-set distribution. The current experiments can be kept as-is but this should also be included. The use of samples generated from test distribution alone cast serious doubts on the claims made in the paper in the form that they appear currently. If the authors are unwilling to do this, the claims and writing should be changed to reflect the new setup (i.e., The model assumes direct and explicit knowledge of what exactly the \"test\" distribution looks like, OR, For known distribution shifts having unlabeled data can help improve performance on *that* specific distribution shift.). This is critical to securing my recommendation for accept.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eZJMyi9VUI", "reviewer": "Reviewer_SCqd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n[S1] Overall, the paper is clearly written; the proposed method is sensible and the results support the core claim of compositional generalization (if the scope of the claim is narrowed down (see weaknesses section)).\n\n[S2] Both sampling and using a generative model are sound approaches to gathering unlabeled, test-like data.\n\n[S3] Use of entropy threshold to filter out samples e.g., unanswerable ones is well motivated and Fig 3 does a really great job of justifying the use of entropy. This is also a simple measure to compute for practical purposes.\n\n[S4] Model selection procedure is also sound as long as one has access to unlabeled test/test-like data. Sec 3.5 clearly shows the need and utility of this approach over a labeled val set that does not reflect test-like shifts.\n\n**Weaknesses:**\n\nThe main weakness is that the experimental setup does not fully support the current broad claim of compositional generalization. To support the scope of the current claims, I think the paper needs to study different types and levels of shifts between unlabeled and test distributions, which is perhaps best studied with a more realistic dataset e.g., GQA instead of CLEVR. The other option would be to revise certain claims/sections.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a learning method called: DisCo (Distribution Coverage) to improve compositional generalization for the CLEVR dataset. It proposes two components: the pseudo-labeling framework pairs up compositionally novel, unlabeled images with randomly selected questions to improve compositional generalization and the model selection strategy uses a distribution coverage metric to select models that generalize better. The results show improved compositional generalization in the CLEVR dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It would be ideal to test the method on more realistic scenarios i.e., with varying types/degrees of shifts between the unlabeled and the actual test distributions preferably on datasets more realistic than CLEVR. I think such experiments could provide evidence for some of the claims with broad scope e.g., those made in Sec 3:\n\n> As training progresses, DisCo selects more “difficult” question-image pairs, which increases distribution coverage on the unlabeled image set. After training, models can reason about objects with new attribute compositions, not seen with labels during training.\n\nThis is however not critical for acceptance as long as the claims are edited to reflect the scope/assumptions.", "broader_impact_concerns": "The existing section looks good. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a learning method called DisCo for better distribution coverage to improve compositional generalization for visual reasoning on the CLEVR dataset. After author response, it received 2 Leaning Accept, and 1 Accept recommendations. 

On one hand, all the reviewers agree that the paper is clearly written, the proposed method is reasonable and the results support the core claim of compositional generalization after revision. The experiments are thorough and the ablation is clear. 

On the other hand, the main concern raised by all the reviewers is that the experimental setup does not fully support the broad claim of compositional generalization. Specifically, the model has already learned from the test-time data distribution during training before testing. This experiment setup is a little bit weak, since the specific test distribution should be unknown at training time. During rebuttal, the authors have adjusted the claims and added some new results, which the reviewers appreciated. 

Another weakness is that all the experiments are performed on the toy CLEVR dataset, so it remains unclear how the technique developed in this paper is really useful for real-world datasets. The authors have added this limitation discussion in the revision. 

Overall, the rebuttal solved most of the concerns raised by the reviewers, and all the reviewers tend to accept the paper. Though some weaknesses still exist, the paper itself is still self-contained, and will be interesting to researchers specifically working in this sub-field. Therefore, the editor decided to recommend accept by the end. 
Decision: Accept (poster)