Title: Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting
Venue: NeurIPS 2025
Accepted: yes
Keywords: Object Dependencies

Scene Understanding

Counterfactual

Inpainting

Support Relations

Generative Models
Abstract: This paper proposes a novel scene understanding task called Visual Jenga. Drawing inspiration from the game Jenga, the proposed task involves progressively removing objects from a single image until only the background remains. Just as Jenga players must understand structural dependencies to maintain tower stability, our task reveals the intrinsic relationships between scene elements by systematically exploring which objects can be removed while preserving scene coherence in both physical and geometric sense. As a starting point for tackling the Visual Jenga task, we propose a simple, data-driven, training-free approach that is surprisingly effective on a range of real-world images. The principle behind our approach is to utilize the asymmetry in the pairwise relationships between objects within a scene and employ a large inpainting model to generate a set of counterfactuals to quantify the asymmetry.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a counterfactual causal world model (CCWM) for offline reinforcement learning. The model explicitly disentangles factual and counterfactual transitions through latent variable modeling and performs structured credit assignment using a causal graph. A value-aware training objective is proposed to guide model learning toward better downstream policy optimization. The authors demonstrate improved performance over baselines such as MOPO and COMBO on standard D4RL benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n1. The idea of modeling counterfactual transitions separately from factual ones is novel and meaningful for improving policy generalization in offline RL.\n\n2. The framework incorporates causal structure and value-awareness, which makes the learned dynamics more aligned with policy optimization goals.\n\n3. Experiments on D4RL show consistent improvements across several tasks, especially in medium and medium-replay settings.\n\n4. The causal graph structure is predefined and fixed, which limits flexibility and may not generalize well to tasks where causal relationships are unknown or changeable.\n\n5. Implementation details around how counterfactual states are generated and trained are somewhat underexplained, especially the approximation and sampling process.\n\n6. Theoretical analysis is limited. There is no formal guarantee that counterfactual modeling leads to better policy learning, and no ablation on the causal structure is provided.\n\nQUESTIONS:\n1. How sensitive is the model to the predefined causal graph? Would learning this structure from data be feasible?\n\n2. How are counterfactual latent variables sampled during training? Is there a mechanism to avoid degenerate reconstructions?\n\n3. Does the value-aware objective introduce bias in model learning, especially if the critic is not accurate?\n\n4. Would the method generalize to high-dimensional environments like visual RL tasks?\n\n5. Could the same benefit be achieved with simpler model ensembles or uncertainty modeling techniques?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an interesting new visual task for \"visual jenga\"; where given a scene with multiple objects, the task is to remove them one by one in a structurally feasible manner until only background remains. For instance consider, an image with a book lying on top of a table in an otherwise empty room. The goal of the visual jenga is to remove the objects one by one in a physically plausible manner. In this case, remove book first and table second. The final image would be of an empty room. The key idea testing on this task helps evaluates the models understanding for structural interdependencies between objects. The paper also proposes a counterfactual inpainting based baseline which removes the object which if inpainting leads to on average the highest CLIP/DINO similarity with the original image. The authors also motivate the need for this task from the perspective of 3D understanding and both qualitative and quantitive evaluations for the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed task is novel and interesting\n* The visual jenga task provides a new way of thinking about how to evaluate current VLM models on their ability to understand 3D scenes and their structural interdependencies.\n* The use of counterfactual inpainting is also interesting\n* The paper provides both qualitative and quantitative results for showing the efficacy of the proposed approach.\n\nWeaknesses:\n* The paper motivates the need for new task for evaluating understanding capabilities of current VLM models. Given the same, it would be good to motivate the need for the same by evaluating current VLM models on this task\n* For instance, similar to the proposed pipeline (Fig. 4), the authors could use Molmo for placing a point on each object (with numbering). The generated image could then be fed to a VLM which can respond with which object to remove next.\n* Also Sec. 5, the authors mention that text alone as outputs for VLM can lead to ambiguity e.g., if there are multiple objects of same type. However, by placing specific counts on the point for each object (as suggested above), the VLMs should be able to output the desired object to remove without ambiguity.\n* Finally, it would be nice to see some comparisons with VLM + SFT (with curated data) to see if the problem can be simply address using targeted data for this task.\n\nQUESTIONS:\n* The current evaluations in Sec. 4, use other simple object removal ordering as baselines. Can the authors provide evaluations using state-of-art VLMs for these tasks?\n* In particular, it would be good to follow the numeric labelling strategy for getting VLM responses to avoid the problem of ambiguity when indicating target object by the VLM\n* Also for the curated data, it would be nice to estimate the performance of a simple VLM + posttraining e..g simple SFT. This will provide an estimate of the actual difficulty of the given task, and whether the same can be addressed using more targeted fine-tuning data alone.\n* Finally, I was curious about some qualitative examples since their could be multiple orderings for the same scene. For instance, for Fig. 6, I had imagine after removing the cars in the background, removing the hut in the background would lead to the highest DINO similarity between original image and inpainted one. Given the same, it was surprising that the hut was not removed until the very end. Can the authors please provide some explanation for the same?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes **Visual Jenga**, a scene-understanding challenge where the model must successively “pull” objects from a single image while ensuring the scene still looks physically plausible. The authors:\n\n* Define the task and release benchmark splits based on COCO, NYU-v2, and a new ClutteredParse set.\n* Offer a training-free baseline: segment objects → inpaint each one with a diffusion model → rank objects by the diversity of plausible inpaintings (measured with CLIP/DINO). High diversity signals the object is not structurally critical.\n* Show this diversity ranking outperforms size, depth, and similar heuristics in pairwise dependency tests and full scene deconstructions.\n\nOverall, the work introduces a fresh probe of object-level dependencies and demonstrates that generative inpainting can act as a zero-shot cue for physical importance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n*Originality & Significance.*  Recasting scene understanding as “Visual Jenga”—remove objects one-by-one while the image remains physically plausible—is genuinely novel.  It pushes beyond familiar segmentation or affordance tests toward causal reasoning: which item is truly load-bearing, which is cosmetic?  That framing could seed a new line of work at the intersection of vision, generative models, and intuitive physics.\n\n*Methodological quality.*  The authors deliver a surprisingly effective, training-free baseline.  By measuring how *diverse* a diffusion model’s inpaintings become when each object is masked out, they extract a proxy for structural importance that beats size, depth, convexity, and other simple cues.  Experiments span COCO, NYU-v2, and a purpose-built “ClutteredParse” set, and the evaluation protocol (pairwise dependency accuracy plus full deconstruction score) is transparent and replicable.\n\n*Clarity.*  The paper is easy to follow.  Diagrams show the step-by-step pipeline (mask → inpaint → CLIP/DINO diversity → rank → remove) and qualitative sequences make successes and failures intuitive.\n\n\n**Weaknesses**\n\n*Physical realism is only skin-deep.*  “Plausibility” is judged by a diffusion model’s ability to hallucinate sensible pixels, not by any physics engine or human study.  The system can therefore declare victory on scenes that *look* stable yet would collapse in reality (e.g., a table leg removed but magically inpainted).\n\n*Dataset scope.*  The benchmark is built from repurposed still images.  Real-world support relations—complex stacks, articulated furniture, liquids—are rare.  Without video or 3-D data, the task avoids many of the hardest physical ambiguities, so it is unclear how far results generalize.\n\n*Computational load and sensitivity.*  Calling a large diffusion model and dual feature encoders for *every* object is expensive.  There is no ablation across diffusion backbones or clip variants; performance may hinge on specific pretrained models or prompt engineering.\n\n*Missing stronger baselines.*  No learning-based alternatives are tried—e.g., graph neural networks trained on synthetic physics, or models fine-tuned to predict support graphs.  Without them, we cannot tell whether the zero-shot method is truly hard to beat or merely faces weak competition.\n\n*Temporal blindness.*  The task assumes a single static frame.  Some dependencies—stack order, friction, recent motion—cannot be inferred from a snapshot, capping the ceiling for any image-only approach.  A discussion of this limit and possible video extensions is needed.\n\nTaken together, the paper is creative and thought-provoking but would benefit from tighter physical validation, a broader dataset, and stronger comparative baselines to establish lasting impact.\n\nQUESTIONS:\n1. **Physical validity** – Have you verified that high-score removals are truly stable (e.g., human ratings or physics-engine tests)? Evidence here would solidify the task’s credibility.\n\n2. **Dataset realism** – Any plan to add images or videos with denser, load-bearing stacks? Even a small pilot set would show the benchmark’s growth path.\n\n3. **Backbone sensitivity** – How does performance change with a different diffusion model or feature encoder? One ablation would clarify robustness.\n\n4. **Stronger baselines & runtime** – Have you tried a learned support-graph model, and what is the per-image compute time? These numbers matter for judging practical impact.\n\nLIMITATIONS:\nThe paper briefly notes computational cost but skips two key aspects:\n\nPhysical-validity caveat – it should acknowledge that diffusion-based “plausibility” may mask structurally impossible scenes and propose human or physics-based verification.\n\nSocietal impact – manipulating images to hide structural flaws could enable misleading visuals (e.g., safety or real-estate fraud). The authors should discuss safeguards and ethical use guidelines.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}