Title: Holistic Order Prediction in Natural Scenes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Scene Understanding

Scene Geometry

Occlusion

Depth

Orderings
Abstract: Even in controlled settings, understanding instance-wise geometries is a challenging task for a wide range of visual models. Although specialized systems exist, modern arts rely on expensive input formats (category labels, binary segmentation masks) and inference costs (a quadratic amount of forward passes). We mitigate these limitations by proposing InstaFormer, a network capable of holistic order prediction. That is, solely given an input RGB image, InstaFormer returns the full occlusion and depth orderings for all the instances in the scene in a single forward pass. At its core, InstaFormer relies on interactions between object queries and latent mask descriptors that semantically represent the same objects while carrying complementary information. We comprehensively benchmark and ablate our approach to highlight its effectiveness. Our code and models are open-source and available at this URL: https://github.com/SNU-VGILab/InstaOrder.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel approach for occlusion and depth ordering of objects in a scene. In particular, while most approaches do this by evaluating pairs of objects in each forward pass, the authors propose a method for \"holistic\" ordering, meaning that a single forward pass is able to construct the whole ordering matrix. The approach is validated on a common dataset, and the authors test it against several baselines, from specialised to naive ones, and perform a number of ablation studies to prove the effectiveness of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper tackles an interesting topic, relevant and relatively under-explored so far. The solutions the authors come up with are interesting and mostly well-motivated, and the experimental evaluation is convincing.\n\nThe main limitation of the paper is that it is unclear. Many parts are hard to understand, and overall the approach is not clear once a reader goes over Sec. 3. Additionally, more details on the task itself could be shared, as it is a relatively uncommon task (for now), and not as easily-recognizable as other CV tasks. Also, the authors repeat several times that their holistic approach is more efficient as it predicts everything in a single forward pass, but its efficiency compared to the other baselines is not tested quantitatively.\n\nQUESTIONS:\nMajor:\n* More details on the task in Sec. 3.1. For example, on how the matrix G is usually constructed (differences between g_ij and g_ji? How does g_ij look like if object i and object j are completely unrelated? etc.)\n* More training details (loss functions unspecified, mentioned some auxiliary ones in line 252 without specifying which one and for what purpose, etc.), as the paper should allow people to reproduce the approach in its entirety, and it's currently not the case.\n* Quantitative analysis on efficiency of \"holisticity\": report inference time and/or GFLOPs of proposed approach and baselines.\n\nMinor:\n* Some figures/tables are not hooked in the text (Fig.1, Fig. 2b, etc.)\n* Contributions in the introduction unclear (Contrib #2 is unclear and should be reformulated, Contrib #3 is arguably not a contribution at all)\n* Include appendix at the end of the paper?\n* Why are the baselines in Sec. 3.3 discussed in the main section dedicated to the proposed approach?\n\nLIMITATIONS:\nLimitations are not really addressed, as something is mentioned in the introduction but it's mainly a limitation of the existing dataset to evaluate the approach. A proper evaluation of the limitations of the proposed approach is necessary. A (rather short) sentence on possible future works in, instead, present.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a solution for simultaneously predicting occlusion and depth orders of all instances in a scene from a single RGB image in a single forward pass. The key novelty lies in predicting the ordering matrix ​**G**​ in a single forward pass, thereby eliminating the need for pairwise inference.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nThe paper show state-of-the-art (SOTA) performance relative to pairwise methods.\nIt presents seemingly comprehensive experimental results and ablation studies.\n​\n\n**Weaknesses:**\nThe computational efficiency claims lack empirical validation, such as frame per second (FPS) metrics or memory usage, which are essential for assessing practical applicability. The scalability of the method beyond the dataset's 10-instance limit remains unverified, raising questions about its real-world applicability in scenarios with a larger number of instances.\n\nQUESTIONS:\nI must acknowledge that I am not an expert in this field; however, after reading this paper, I have the following questions:\n\n- **Scalability:** Can your method handle more than 10 instances? Handling fewer than 10 instances appears impractical. The maximum number of instances might be constrained by the value of N. Perhaps incorporating more tokens could address this.\n\n- **​Inference Efficiency:** How does the speed of your solution compare to pairwise methods? For instance, reporting Frames Per Second (FPS) on 1024×1024 images would be beneficial. Pairwise methods might achieve faster inference through batch processing, potentially outperforming the original approach. Therefore, it's essential to ensure that comparisons are fair and that both implementations are optimized appropriately.\n\n- **​Readability of Figure 3:** The rationale behind designing the method as depicted in Figure 3 is not entirely clear. For example, could you elaborate on the rationale behind using a transformer with only one layer? There may be underlying motivations that make your design unique and theoretically superior to alternatives. It would be helpful to elaborate on these motivations before delving into the specifics of the design.\n\n- **Theoretical Advantages of Predicting ​G​ in One Pass:** Does predicting the ordering matrix ​G​ in a single forward pass offer any theoretical guarantees of superiority over previous methods in terms of accuracy, independent of empirical results? It would be beneficial to understand if there are inherent theoretical benefits that position this approach as fundamentally more accurate or robust compared to traditional pairwise inference techniques.\n\nLIMITATIONS:\nThe paper does mention certain limitations of their approach. However, it would be beneficial if the method could handle an unlimited number of instances, thereby increasing its practicality and effectiveness in more complex scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a nicer design for order/occlusion prediction by internally combining a segmentation architecture (Mask2Former) and its resulting object queries to predict orderings/occlusion in a full attentional approach. This is easy to train end to end, and can rely more on object-centric context in order to make its predictions. Previous work was less integrated or relies on specific pairwise networks to produce these predictions. The authors show a variety of good baselines (including VLMs, and a more basic Mask2Former approach), while their method generally outperforms existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Everything here makes _sense_ and seems superior to previous designs. It's a solid design.\n2. This appears to, in turn, show solid results on the ordering/occlusion benchmarks.\n\nWeaknesses:\n\n1. While the authors appear to slight existing methods for requiring masks, the resulting approach of InstaFormer is still lacking in that it requires a full decode of an instance segmentation mask. The main innovation comes the idea that full Transformer attention of mask/image-derived features can be used to perform the pairwise prediction tasks in one shot (e.g. LightGlue, etc). Their ablation shows that it appears to heavily depend on this _explicit_ spatial information which is somewhat disappointing.\n2. The authors could spend less prose on comments like \"making their real-world inference impractical as the masks might be unavailable to the user.\" This detracts from the paper since it makes it seem like the idea of using predicted masks instead of the ground-truth masks is a heroic leap of faith.\n3. The idea that one can use the dot product here rather than separate classification steps is good, but it seems a little far fetched to call this \"holistic\". Is this implied from the idea of self attention? Is it from using the dot product for prediction (but this doesn't seem particularly holistic?).\n\nQUESTIONS:\n1. More information on how Mask2Order was trained. Is it trained (always) on results from Mask2Former or is this only at inference time?\n2. The final prediction is a dot product which is just binary classification. How does this determine the {1, 0, -1} results needed for occlusion ordering? Is it separate MLP for each \"class\"?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
The authors present a method for predicting object order (occlusion relations and depth ordering). The nice new aspect of the method is that it generates order in a single forward pass by modeling the interaction between object queries and latent mask descriptors. The experimental results are comprehensive, by comparing the method to past works, including three parallel works. Results are promising.

STRENGTHS AND WEAKNESSES:
Strengths:
- Modeling order in a single forward pass with good results.
- Motivating why order is important by linking it with the VLM which improves performance in depth based reasoning.
- Clear write up
- Comprehensive experiments

Weaknesses:
- The related work section could benefit from the authors reviewing some older works on the subject. For example, "Monocular Object Instance Segmentation and Depth Ordering with CNNs" by Zhang et al plus references within could be a good reference for older works. 
- Motivating why depth ordering is important could be done more convincingly. I'm still not sure where someone would actually use this method in practice.

QUESTIONS:
- Motivation is a bit weak. Highlighting an important use case could make a reader more excited about the area of work.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-08