Holistic Order Prediction in Natural Scenes
Authors: Pierre Musacchio
Hyunmin Lee
Jaesik Park
Venue: NeurIPS 2025 poster ()
Keywords: Scene Understanding
Scene Geometry
Occlusion
Depth
Orderings
Primary area: deep_learning

Abstract: Even in controlled settings, understanding instance-wise geometries is a challenging task for a wide range of visual models. Although specialized systems exist, modern arts rely on expensive input formats (category labels, binary segmentation masks) and inference costs (a quadratic amount of forward passes). We mitigate these limitations by proposing InstaFormer, a network capable of holistic order prediction. That is, solely given an input RGB image, InstaFormer returns the full occlusion and depth orderings for all the instances in the scene in a single forward pass. At its core, InstaFormer relies on interactions between object queries and latent mask descriptors that semantically represent the same objects while carrying complementary information. We comprehensively benchmark and ablate our approach to highlight its effectiveness. Our code and models are open-source and available at this URL: https://github.com/SNU-VGILab/InstaOrder.

Holistic Order Prediction in Natural Scenes
Pierre Musacchio Seoul National University pmusacchi
Hyunmin Lee 1.G Al Research Syyumin@@greeearcee gresearch.ai
Jaesik Park Seoul National University jaesik. parkêsnu. ac kr
mrusacchio@smu.cc o@snu. ac kr
Occlusion
Rackel
Pers
Pexsl
Depth
Pors
Ball
Vhich of the rumnr fintated
Which of the Tonrer fini ebed fizet ?
fires
Por.
Depth
Rucket
P
Ball
LLaVA-1.5-7b
LLaVA-1.5-77
Racket2
Dall
The blue
The runner the luft Fintahed firet:
unt.orm Finiehed. F1780
Figure I: In-the-wild inferences using InstaFormer display segmentation results along the occlusion and depth order predictions of images gathered from the web. Instances are represented as nodes in the graphs. Edges are arrows characterizing their order relations. InstaFormer provides accurate instance- wise geometries. Morcover, converting its outputs to text format enables VLMs to understand geometries in zero-shot manner better.
Abstract
Even controlled settings, understanding instance-wise geometries is a challeng- ing task for wide range of visual models. Although specialized systems exist, modern arts rely on expensive input formats (category labels, binary segmentation masks) and inference costs (a quadratic amount of forward passes). We mitigate these limitations by proposing InstaFormer, network capable of holistic order prediction. That is, solely given an input RGB image, InstaFormer returns the full occlusion and depth orderings for all the instances in the scene in a single forward pass. At its core, InstaFormer relies on interactions between object queries and latent mask descriptors that semantically represent the same objects while carrying complementary information. We comprehensively benchmark and ablate our approach to highlight its effectiveness. Our code and models are open-source and available at this URL: https:/ / github....... com ASN/SNU-NN HIIUUV/ InstaDrder.
1 Introduction
While scene understanding has always been an active domain in computer vision [3, 19, 21, 24, 50), its importance has recently become evident with the rising popularity of vision-language models (VLMs) [2, 5, 27, 28). In a VLM framework, the user textually interacts with an agent based on visual grounding from an input image [23, 27, 35, 42, 46]. However, current VLMs still struggle to understand the intricacies of the geometric relationships between the elements composing the scene. Instance-wise order prediction often overlooked due to its apparent simplicity and
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

tendency to opt for dense prediction networks [18, 34, 44, 45]. Yet, we emphasize that this simplicity deceiving. even for dense foundation models [20, 45]. From determining pedestrian car occlusions for the safe piloting of autonomous vehicles to estimating depth layouts and relative positioning in robot's perception module, understanding instance-wist orderings is a non-trivial task that is often overlooked yet crucial for grounding our interactions with machines.
Few existing approaches explicitly tackle this geometrical scene graph problem [24, 47, 50J. Yet, they suffer from expensive inference costs and impractical input formats. Specifically, these methods formulate the occlusion and depth ordering prediction tasks series of instance- wise edge prediction, meaning they require quadratic amount of passes to obtain the full graph (Fig. 2a). Moreover, all the aforemen- tioned arts require the binary masks of all the instances as input (Fig. 2b).
(a) Ordering prediction frameworks.
text
VLM
Foundation Modals
To address the input format issue, we first design straightforward approach called Mask2Order. Mask2Order is concatenation of pre-trained segmentation network [4. 8, 9] to a pairwise or- der network [24]. Given an input image, the seg- mentation network first generates masks which are then fed by pairs, along with the input image to the pairwise network to obtain their geometri- cal relations. Although this approach alleviates the input limitation since self-supplies the or- der head with generated masks, its performance degrades while continuing to require multiple inferences. Thus, introduce the main contri- bution of our work, InstaFormer, a joint segmen- tation and ordering prediction network capable of holistic order prediction.
Seg. Net
Painwise Net
A
FosaForeer
Hrviztic
InstaFormer
0
(b) Input-output and inference cost of ordering predic-
tion frameworks.
Input
Output ordering
Methods
Masks
Image
Occlusion
Depth
Pairwise Nets VLM Foundation models Mask20rderr
InstaFormer
Figure 2: Overview of our holistic approach. We compare the holistic approach to other inference strategies (Fig. 2a) and highlight the input-output format discrepancies (Tab. 2b).
In the holistic prediction paradigm, we reformu- late the edge-level occlusion and depth ordering prediction tasks to an adjacency matrix-level problem. This allows the network to predict the whole mnstance-wise geometries in constant inference cost, i.e., single forward pass, for any arbitrary number of instances the scene. This is made possible thanks to attention interactions between object queries and latent mask descriptors representing the same objects while carrying complementary information.
In summary, our contributions can be stated as follows:
We cast the traditional edge-level occlusion and depth order problems to adjacency matrix-level prediction task. We call this task holistic order prediction.
We introduce InstaFormer, network family capable of holistic order prediction, surpassing or matching the best available baselines on the tasks of occ occlusion and depth order prediction solely from an RGB input image.
2
Related Work
2.1 Order prediction
Occlusion order prediction has been introduced in the seminal work of [50] in which authors tackle the task of amodal instance segmentation. Amodal segmentation datasets such as COCOA [50] or KINS [32] subsequently annotate occlusion order annotations. Other datasets focus explicitly on labeling instance orderings, such as INSTAORDER [24] or WALT [37]. While PCNet-M trained in
an unsupervised manner [47], OrderNet [50] and InstaOrdenNet [24] show superior performances by

---

being trained in supervised setting. Inspired by instance-wise occlusion order, [24, 48] propose to also predict instance-wise depth orders. Ultimately, all these approaches leverage a pair of binary masks and an RGB image as inputs, making their real-world inference impractical. Additionally, the pairwise nature of these networks enforces a quadratic number of forward passes to obtain the full relations between all the instances in the scene.
On the other hand, we reformulate the pairwise edge-level ordering prediction problem a single adjacency matrix-level prediction. This holistic property of our InstaFormer model yields the full geometrical ordering relations in unique forward pass, solely from RGB image given input (Fig. 2a), effectively alleviating the input-output constraint induced by prior approaches while enabling single-pass inference.
2.2 Foundation models
Vision foundation models [20, 31, 44, 45] are trained on an extensive amount of annotations to excel in the task they were designed for. SAM can be prompted using points, bounding boxes, or masks to provide the most accurate segmentation corresponding to this prompt [20]. Similarly, Depth Anything models perform fine- grained monocular depth estimation [44, 45]. While initially designed to solve a specific task, their robustness and versatility allow them to stand as modules of a composite pipeline [38, 43J. In this art, construct simple baseline for order prediction by combining SAM and Depth Anything V2. Likewise, combining vision [33] and text foundation models [10, 40] together gives birth to VLM [2, 27, 28]. Grounding the image [23, 35] in the textual latent space of the large language model enables performing diverse visual zero-shot prompting tasks directly using
natural language queries.
To assess diverse types of models, we convert INSTAORDER to a visual-questie answering (VQA)
format named INSTAORDERVOA Not only do perform zero-shot prompting LLaVA [27], but we also finetune it on INSTAORDERVOR QA to observe if VLMs are capable of understanding geometrical orderings. We also propose simple foundation model pipeline consisting of SAM [20]
and Depth Antyhing V2 [45].
3 Method
3.1 Problem formulation
Occlusion and depth orders are geometric relations linking instance nodes scene graph. Orthodox literature [24, 47] formulate these tasks as a series of edge inferences between pairs of instances (A, B) living an input RGB image I E I "x x3 and their respective instance-level binary masks B) E {0, 1 Given E N such that 2 2 instances in I, the occlusion (resp. depth) ordering task aims at predicting the k-categorical ordering matrix G E {0.... k] where each element (i,j) E (O...., 1)2 represents the ordering relation, i.e. the edge, for instances (i,j), namely G. On the other hand, in our holistic scenario, we aim at bypassing edge-level inference of relations G.j and instead predict the adjacency matrix G all at once, directly from I. Formally, want to find a mapping f such that f[I) G.
From here, we use the following "->" notation to indicate both occlusion and depth relations. We read "A >> B" "A occludes B". for the occlusion order task and "A is in front of B" for the depth order task. We refer the reader to Appendix A for more details about these tasks.
3.2 Preliminaries: pairwise networks
Existing arts [24, 47, 50] formulate the occlusion prediction outputs as binary prediction problem, ie., [A-BB B-+A). This leads to no occlusion in the simplest case and a bidirectional occlusion in the most complex case. The depth ordering prediction task is formulated as a 3-classes classification problem (A-B, B AA A B), with an extra prediction explicitly indicating A and share overlapping depth ranges [24] (Fig. 2a, pairwise networks).
All these approaches leverage input binary masks, making them cumbersome, especially when ground truth masks are unavailable at test time (Fig. 2a).
3

---

Transformer
d gase
Tx
concat
Franformet Fneoder
chunk
DD~Pooiig
Backbone
Per Pixel Decodar
Encoder
MLP.
Sogmentation Module
Geometrical Order Module
Figure 3: Overview of InstaFormer. The architecture comprises two modules. The first module generates the mask embeddings Q. the per-pixel embedding P. and the masks M. In practice, we use Mask2Former [9]. Then, transformer-based geometrical order module predicts the orders from these three inputs. InstaFormer is capable of end-to-end holistic geometrical order predictions.
3.3 InstaFormer for holistic order prediction
Motivations. Here, introduce our main contribution: the InstaFormer family of networks. InstaFormer is capable of holistic order prediction. Given an RGB image as input, it predicts the full order matrices in single forward pass thanks the holistic objective formulation and its internal attention mechanism use. The main idea of our method is to create interaction between object queries and latent mask descriptors representing the same objects while carrying complementary information. A dot product, akin to a similarity operator, computes the final geometrical output from these two representations. We invite the reader follow equations along using Fig. 3.
Architecture. Let us assume a segmentation backbone f designed using the meta-aree veea-architecture as described by [9] and an input image I E "xx33 Then, fII returns the output mask embeddings Q E NxC. from the transformer decoder and the per-pixel embedding output from the per-piiee per-pixel decoder P E Cx 4 4 Here, N denotes the number of object queries and C the embedding dimension.
The binary masks M E {0, x are obtained from = ((a(QP)), where the matrix multiplication be seen dynamic convolution between set of segment kernels and feature map [17]. Resulting logits are sent to the sigmoid function a and binarized via the rounding function L-] to obtain the final mask predictions.
Following DETR [4], we obtain the optimal assignment between ground truth and predicted segments using the Hungarian Matcher [22]. This way, we identify all the no object segments (0) and discard them, resulting in selected tokens M {0, 1 4 4, Q R We now compute per-mask- per-pixel embedding by masking P using M, resulting in a E RaxCx 46 representation. This retains the values of the per-pixel embedding that falls in the region of the given segment while zeroing others.
We want to obtain a global descriptor from each E P, with i nz.. We found that directly pooling from P does not provide comprehensive enough description (Tab. 2b). Thus, first forward P to a single transformer layer [1 I] and max max-pool this updated representation instead. To obtain accurate segment-leve mask descriptors, we restrict the attention of each p; itself using Masked Aeff-Attention (MSA):
pQ (P*)T +M
MSA MSA(P,M) = softmax
pv
vd
{
-0o if p.(h, w| 0, 0 otherwise.
where M
(2)

---

Here, p(Q.KV) RCxd are the respective queries, keys and values of P [41]. We only now apply max-pooling on the masked region of to obtain our global descriptors:
maxpool(P) E R=C
While D contains -grounnded information, Q contains geometric and shape-relevant knowl- edge [17]. We concatenate the global descriptors D and the mask embeddings Q together, obtaining Eu E 22xx and forward this representation to {1)=1 LEN transformer layers [11, 41] to generate interactions between those two token types [25]. We denote the final output of this module as Eg E IR 2
We recover the updated mask embeddings Qz and the descriptor tokens Dr. by chunking E,, and then forward them to two different multi-layer perceptrons: Q* MLP, (Qt) and D = MLPo(Dr)
We obtain the final geometric ordering prediction G E " through matrix multiplication between the two representations: G QDD effectively computing the compatibility between each kernel and each global descriptor (Fig. 3). We project this representation one last time using task-specific MLPs with parameters for occlusion and 5 for depth orderings, yielding the final occlusion order matrix G E Ra and depth order matrix G RaXa where the (i. element of the matrices represents the predicted geometric order between the i-th and j-th segments. When training InstaFormer or InstaFormer", only one these MLPs is initialized, whereas we use the two different MLPs simultaneously when training InstaFormer
Adapters. To allow the network to fully benefit from the segmentation latent space the mask extractor, we attach adapters to every feed-forward network (FFN) in the transformer decoder of the backbone [6, 16, 19]. We refer the reader to Appendix D for more information.
Inference. We cannot leverage the Hungarian Matcher at inference time since it requires ground truth annotations, Instead, we simply follow [9] and consider segments as detected their confidence score lies above 0.8.
4 Experiments
4.1
Baselines
VLM. We introduce VLM baseline by converting INSTAORDER INSTAORDER-VOL and prompt the model with occlusion and depth order questions. We use LLaVA [27] as it is open-source, popular, and widely used. We run shot evaluation but also fine-tune on our data (denoted with $ in Tab. 1). We refer the reader to Appendix B and C for more information.
Non-pa carameraettii methods. We employ the Y-axis and Area two non- con-parametric methods to estimate occlusion and depth orders. These heuristics rely on the fact that an object at the bottom of the image might be closer to the camera (Y-axis), and objects that are in the foreground are usually bigger than objects in the background (Area).
Foundation models. By composing Grounded SAM [38] and Depth Anything V2 [45], we de- sign foundation pipeline capable of instance-wise depth order prediction. We refer the reader to Appendix for more details.
Pairwise paradigm. Pairwise networks represent the most competitive baselines and provide the closest comparison our setting. To the best of our knowledge, only three concurrent works attempt to predict geometrical orderings. Thus, we include them all. That is, we compare our work to PCNet-M [47], OrderNet [50], and InstaOrderNets [24].
Mask2Order. Since pairwise networks cannot perform inference unless pair of segmentation masks provided a priori [24, 47, 50] (Fig. 2a, pairwise networks), we introduce Mask2Order to solve this issue in the simplest possible way. It results from the concatenation of segmentation backbone and a pairwise ordering prediction network (Fig. 2a, Mask2Order). Motivated by recent advances in this domain [4, 8, 49]. we select Fosk2Former [9] mask generator. While the
5

---

Mask2Order framework can obtain geometrical orderings from an RGB image without additional binary masks from the user, this network family still requires multiple forward passes to get the relations between all the instances. We also train InstaOrderNets on top of Mask2Former (denoted with & in Tab. 1). this setting, we use the same recipe as [24] but modernize optimization by switching the optimizer for AwamW and using cosine schedule.
4.2 Datasets
We run experiments on INSTAORDER [24]. We convert INSTAORDER to a VQA version, i.e. INSTAORDER-VOM to evaluate LLaVA [28] occlusion and depth order prediction in zero-shot and finetuned manners (denoted with in Tab. 1). We release the conversion script and the dataset with this work. We refer the reader to Appendix B for more information.
4.3 Evaluation protocol
Following previous works [24, 47, 50], we evaluate the performance of our networks on the task of occlusion ordering prediction by reporting precision, recall, and Fl-score. We compare each element of the predicted ordering matrix to each element of the ground truth matrix. Given two instances A and B, the aforementioned metrics are expressed follows:
Metrics.
EAB 1 = and OAB =
Recall
(4)
ΣAB [(OAB =)
Precision ΣAB 1 = 1 and OAn = 1)
(5)
ΣAn L 1)
Fl-score x Precision Recall
(6)
Precision + Recall
where o and â denote respectively ground truth and predicted occlusion order, and is the indicator function.
We report the performances of our approaches for depth order prediction using Weighted Human Disagreement Rate (WHDR) [1]. WHDR indicates the percentage of weighted disagreement between ground truth d and predicted depth order d [24]. Each annotation of INSTAORDER accompanied by weighting factor w indicating the difficulty of the annotation based on how many annotation attempts it took for annotators agree on the ground truth. WHDR is computed independently for every (distinct, overlap, all] category and is defined as follows:
ΣAD WAB I(dAR + dAB)
WHDD
EAB WAB
where WAB
count
Decoupling order evaluation from segmentation predictions. In Mask2Order and InstaFormer, the geometrical prediction is performed after generating the segmentation masks. Thus, the perfor- mance of the ordering predictions is dependent the segmentation predictions of the backbone model. To curb this dependency, we leverage the Hungarian Matcher [4] at evaluation time to provide perfect matching between the predictions and the ground truths before performing the geometrical ordering evaluation. This way, we ensure that we obtain the full instance-wiss evaluation for all the samples in the dataset, providing a fair assessment and allowing us compare Mask2Order and InstaFormer with other baselines.
4.4 Evaluating naïve baselines
-on-paramettic methods. We use the center of the masks as the Y-score and rank them accordingly to obtain the depth orders (Tab. Y-axis). For the area method, we rank the masks by their size, the bigger the closer (Tab. 1, Area). These simple heuristics already provide fair approximation of the geometric order of the elements in the scene.

---

Table 1: Occlusion and depth order evaluation INSTAORDER and INSTAORDER- VQA. We benchmark LLaVA, non-parametric methods, pairwise approaches, foundation models, our Mask2Order baseline (zero-shot and trained, denoted with $), and our proposed holistic InstaFormer. All models are trained on INSTAORDER except for our foundation pipeline (off-the-shelf) and LLaVA (off-the-shelf and finetuned, denoted with $). Best results for model family are in yellow best overall results are in bold.
Input
Ouiput
Occlusion
WHDR
Datoset
Method
GT Masks
Imagee
Category
Text
Occ. order
Dopth ordes
Seg.
Rocall
Prec.
F1
Distinct
Overlap
All
VaA
LLaVA [27]
49.98
48.28
34.10
37.37
44.56
39.52
LLaVA [27]
85.41
55.10
60.41
15.95
27.87
25.98
Area
56.33
71.55
59.67
30.90
66
32.19
Y-axis
44.84
57.34
47.30
22.19
39.04
29.20
PCNet-M [47]
59.19
76.42
63.02
(ederNet""lexth [32]
/
84.93
78.21
77.51
InstuOrderNe'(MM
87.35
79.07
78.98
InstaOrderNe"
88.70
78.21
79.18
InstaOrderNer
89,38
79.00
79.98
79.83
INSTAORDDES DEER [24]
InstaOrdeiNer
22.96
30.46
25.23
(nstaOrdeeNeeMMM)
23.19
28.56
36.45
InstaOrderNer
13.33
26.60
17.89
InstaOrderNee
12.95
25.96
17.51
InstaOrderNet [24]
Miaan(MMan) [34]
10.42
37.67
21.70
MiDaS(Median) [34]
Foundation (Min-Max)
21.71
42.89
29.85
Foundation (Mean)
39.22
22.46
Foundation (Median)
10.80
39.11
22.36
Mask2Order
84.37
78.48
77.49
Mase2O
77.92
88.58
79.1
Mask2Order
2fsk2Orderft
12.96
28.6
18:40
Mask20rddr
27.51
85.50
77.17
12.29
27.03
17.09
79.81
86.86
79.09
12.44
28.44
18.19
Instal lomner"
89.82
78.10
81.89
InstaFormer
8.47
24.91
13.73
InstaFormer
89.57
78.07
81.37
7.90
24.68
13.30
VLM. Zero-shot prompting LLaVA [27] scores the least out of all the benchmarked methods (Tab. 1, LLa)A). We hypothesize that this is because the pre-training of its visual encoder does enforce precise spatial understanding [33]. We observe notable improvements under the finetuned setting (denoted with }), yet, performances remain behind visual experts.
Foundation models. While being incapable of performing occlusion order prediction, find foundation models to obtain modest results: their performance still lag behind those of specialized networks even the ones that use the same format (Tab. 1).
Mask2Order. Table confirms that Mask2Order performs worse than InstaOrderNet since it does not benefit from fine-grained segmentation input masks, even when InstaOrderNets are trained on top of Mask2Former (denoted with $). This implies that predicting geometrical orderings from self-generated binary masks is non-trivial task. We further ablate the backbone size Mask2Order in Tab. 4a and 4b (Appendix E). We observe positive correlation between the size of the backbone and the accuracy results, which seem to confirm our hypothesis stating that pairwise networks are sensitive the quality of the input mask.
4.5 Evaluating InstaFormer
Experimental settings. For all experiments, we train 4 NVIDIA RTX A6000 for 120,000 iterations using AdamW [30] with learning rate 10.5 and reduce it to 1 and 10 at iterations 80,000 and 1 110,000 respectively, as suggested by [15]. We use batch size of 16 and use the BCE an CE losses for occlusion and depth order respectively. We resize the input image to 1024x 1024 and use RandomFlip during training, following [9]. For evaluation, we follow the default R-CNN bascline,

---

Ooclusion
Occlusion
Dkylaad
Depti
Depth
Penset
Penced
Elepbanti
Dlephanc
Occlusion
Occlusion
DasetS
Specn
Spece
Daxa
Doxatt
Door
Pack
Katfe
Dumazt
DheLd
Kail
Dog
Fask
Depth
Dorre
Beeuth
Speen
Donaet
Hot Dox
Pack
Keste
Dog
Tike
Bunto
Occlusion
STOP
Occlusion
Penes
Bett
P
Skuz
Dooe
Deptis
Depth
BucTi
Figure 4: Qualitative results obtained using our holistic InstaFormer network. The first row showcases in-the-wild examples from images gathered on the web. The remaining extracted from the validation set of INSTAORDER. We show the segmentation predictions and represent their corresponding predicted occlusion and depth matrices as ordering graphs juxtaposed to them.
which consists of resizing the smaller size the image to 800 pixels and the longer one to 1333. We match the segmentation predictions their respective ground truth segments using Hungarian matching [22] before evaluating depth and occlusion order recovery.
We use 8 heads, 512-dimensional linear projections for all the attention layers, and 2-layered FFNs with 2,048 hidden nodes in all the transformer layers. The encoder consists of a single transformer layer that simply creates global descriptor for each mask. On the other hand, the decoder comprises eight transformer layers. add auxiliary losses on transformer layers of the transformer decoder. This results in 34M parameter geometrical ordering predictor. We initialize the entire ordering module using Xavier initialization [12]. We set the dimension of all the adapters to 64 and initialize their weights with Kaiming uniform [13] and their biases with zero initialization following [6].
Occlusion order recovery. We evaluate InstaFormer in Tab. It achieves the best Fl score of the benchmark, even overcoming InstaOrderNet" and InstaOrderNerdd which were both trained using a simpler objective and on fine-grained ground truth binary masks. We ablate the segmentation backbone size of our InstaFormer", while keeping the geometric predictor untouched (cf. Tab. 4a (Appendix E). The more parameters, the better the results. This seems to imply that the ordering prediction results of the geometric predictor depend on the embedding space of the segmentation backbone. In all settings, InstaFormer" achieves notable results in comparison to Mask2Order". While holistic order prediction a challenging task, methods can be developed to outperform specialized pairwise models while alleviating the need for input binary masks and reducing the number of forward passes to Most impressively, InstaFormer" slightly outperforms Nd the strongest pairwise network using fine-graine ground truth masks inputs.

---

Inference speed (s.)
InstaOrderNetdd
40
InstaOrderNestd
2
InstaFormerdd
Monnay (Gh.)
30
InstaPormerdd
InstaOrderreed (batchedd
InstaOrdetNes" (batched)
Average InstaFormer
Reg. InstaFormerd
2
3
9
10
1211201
13
11
15
16
17
18
19
20
2
3
4
5
10
11
12
15
14
15
16
17
18
19
Number Instances
Number instanoss
(a) Runtime comparison (in seconds).
(b) Memory comparison (in Gb.).
Figure 5: Inference cost. We benchmark the runtime and memory cost of InstaOrderNet and our InstaFormer All measures are recorded on single NVIDIA RTX A6000.
Depth order recovery. We present evaluation results of InstaFormer in Tab. InstaFormer substantially outperforms all baselines on all WHDR (distinct, overlap, all). Similarly to InstaFormer', we also conduct an ablation study related to the backbone size of the segmentation network, while we keep the size of the geometric predictor fixed (Tab. 4b, Appendix E). Surprisingly, the backbone choice has little impact on the final results, which plateaus around the 13.70 mark. Contrary to occlusion order prediction, we hypothesize that depth order prediction does not require edge information and can still operate properly on rougher masks. Nonetheless, InstaFormer" exhibits remarkable results compared to InstaOrderNet and Mask2Order", respectively being par and beating them even though it operates in a much more challenging setting (i.e. holistic framework: without input masks and by predicting the full depth matrix at once).
Joint occlusion and depth order recovery. We report occlusion order and depth order evaluation for InstaFormer in Tab. InstaFormer is competitive with all baselines and sometimes surpasses them, although not using strongest depth order prediction network. Contrary to the Mask2Order family, joint occlusion and depth ordering prediction on the InstaFormer family performs similarly to order-specific networks. InstaFormer" marginally outperforms InstaFormer (Tab. 1). We suspect the optimization process to be harder for joint occlusion and depth order prediction. In both cases, InstaFormer obtains competitive results with InstaOrderNets, especially on depth prediction, even though it was not trained on fine-grained ground truth masks. Interestingly, slightly differently from what was reported in [24], we observe a substantial improvement in depth performances but not occlusion when performing joint occlusion and depth prediction. Thus, still conclude that there are benefits to using a joint training scheme both from the computational efficiency and performance standpoints.
ground truth masks as input. It effectively ranks as the
Inference cost. We benchmark the inference cost of our approach com- pared to InstaOrderNets, representa- tive of pairwise approaches, in Fig. 5. InstaFormer compares favorably, as its runtime appears constant and mem- ory linear with respect to the number of instances in the image. Details can be found in Appendix F.
Table 2:
Ablation study. Occlusion and depth order perfor- mances on INSTAORDER using InstaFormer Swin-T.
Ablation on the input modality.
Occlusion acc. 1
WHDR
Input modality
Precision Recall Fl
Distinct Overlap
Queries/Queries Descriptors/Desripoors
73.56
88.45 77.92
15.85
39.83
23.21
75.68
88.97
79.39
16.75
38.11
22.56
Queries/Descripors
88.64
75.56
79.74
8.43
25.36
14.03
(b) Ablation on the pooling strategy.
4.6 Ablation studies
Occlusion acc. t
WHDR J
Transformer layers
Pooling
Input modality. The main idea behind our method is to generate instance-wise interactions between la- tent object representations. Naïve in- teraction from object queries to them- selves results in low occlusion accu- racy and poor depth ordering performances (Tab. 2a). When latent mask descriptors interact with themselves, results for occlusion orders rise, but not for depth orders. Best performances are obtained when combining information from both object queries and latent mask descriptors, as they carry complementary information while referring the same objects.
Precision Recall
F1
Distinct Overlap
All
Max
0
74.86
87.87
79.01
16.08
37.87
21.88
Max
88.64
75.56
79.74
8.43
25.36
14.03
Max
2
87.62
78.30
79.80
15.67
34.85
21.48
Max
3.
87.39
78.25
79.81
15.67
37.72
21.57

---

Occlusion
Occlusion
Heg
Dryhart
Depth
Dopth
Fowel
Ansee
Heylact
(a) Instance-mixup-
(b) Segmentation failure.
Figure 6: Failure cases of InstaFormerdd, We highlight two failure cases of our network: the "instance-mixup" (Fig. 6a) and the "segmentation failure" (Fig. 6b).
Pooling strategy. Naïvely constructing the latent mask descriptor does not enhance the perfor- mances as much as it could in Tab. 2b. In fact, directly max-pooling the latent masks does not yield good performance in the depth order prediction task. Instead, simply adding transformer layer before pooling produces a more comprehensive descriptor, enabling competitive results for occlusion and depth order prediction. As the performances saturate with single layer, do scale more.
4.7 Qualitative results
InstaFormer can accurately determine geometrical orderings complex scenes, even for small objects (donuts, Fig. 1). Our network still predicts reasonable relations even under sparse clue inputs (K-Pop example). After text conversion, InstaFormer's outputs can geometrically ground VLMs and enhance their reasoning in zero-shot prompting (Fig. 1, right). We provide diverse qualitative results of our model on INSTAORDER in Fig. 10, in Appendix G. We observe that our model predicts sane orderings in various scene types. Ground truths for all the predictions are available in Fig. 9 and Fig. 11 in the Appendices.
in Fig. 4) under in-the-wild inference (K-pop in Fig. 4, table tennis in
4.8
Failure cases
We highlight two cases in which InstaFormer struggles to predict accurate relationships between instances Fig. 6. We name these situation "instance mixup" and "segmentation failure". In the "instance-mixup" case, the "Person" instances related to the "Ski" are shared instead of being targeted solely "Person In the "segmentation failure" case, a singular instance "Elephant" wrongfully segmented multiple times, leading to multiple false positive nodes the geometrical graphs.
Conclusion
We benchmark diverse range of model types from foundation models VLM, passing through visual experts on the tasks of instance-wis wiie occlusion and depth order prediction. We propose InstaFormer loosen the input-output constraints of existing methods to a bare RGB image while reducing the inference cost of the network to a single forward pass thanks to its holistic nature. InstaFormer matches or surpasses all baselines on the tasks of interest.
Limitations and future work. INSTAORDER annotations are limited to 10 instances per image. It would be interesting to the generality of our approach by obtaining GT annotations larger number of objects. We also plan to extend our work to VLMs and layer-aware generative modeling.
Acknowledgments
This work was supported by the IITP grant (RS-2021-I121 1343:
Graduate School Program at Seoul National University (5%). RSS2002--00509257: Global AI Frontier Lab (40%), and RS- 2025-25442338: AT Star Fellowship Support Program (55%)) funded by the Korean government (MSIT).
10

---

References
[1] Sean Bell, Kavita Bala, and Noah Snavely, Intrinsic images in the wild. ACM Transactions Graphics (SIGGRAPH), 2014.
[2] Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ihrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, al. Paligemma:
A versatile 3b vlm for transfer. arXiv preprint arXiv:2407.07726 2024. 3.3
[3] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Çelikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics (ACL), 2019. 1
[4] Nicolas Carion, Francisco Massa, Gabricl Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020. 2, 4, 5,
[5] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialvIm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognittio,, pages 14455-14465, 2024.
[6] Shoufa Chen, Chongjian Ge. Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adapt- former: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35:1 66644-6678, 2022. 5, 8, 25
[7] An-Chieh Cheng, Hongxu Yin, Yang Fu, Qiushan Guo, Ruihan Yang, Jan Kautz, Xiaolong Wang, and Sifei Liu. Spatialrgpt: Grounded spatial reasoning in vision-language models. In NeurIPS, 2024. 1
[8] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Prucessing Systems, 34:17864-1787, 2021. 5.5
[9] Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-12999 2022. 2, 23, 25
[10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph Gonzalez, et al. Vicuna: open-source chatbot impressing gpt-4 with 90% chatgpt quality. See ht : // vicuna. Lmsys. org (accessed 14 April 2023), 2(3);6, 2023.
3, 23
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL Id-YicbFaNTTy. 5,5
[12] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings. 2010.
[13] Kaiming He, Xiangyu Zhang, Shanqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE international conference on computer vision, pages 1026-10344 2015. 8
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778 2016.
23
[15] Kaiming Hc, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4918-4927, 2019. 7
[16] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzehski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019. 5
[17] Jie Hu, Linyan Huang. Tianhe Ren, Shengchuan Zhang. Rongrong Ji, and Liujuan Cao. You only segment once: Towards real-time panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 17819- 17829, 2023, 4,5

---

[18] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-basee image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9492-9502, 2024.
[19] Seoyeon Kim, Minguk Kang. Dongwon Kim, Jaesik Park, and Suha Kwak. Extending CLIP's image-text alignment to referring image segmentation. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies /Volume 1: Long Papers), pages 461 1-4628, Mexico City. Mexico, June 2024. Association for Computational LiniiinnnnLiiiiiiiiiiii Linguistics. doi: 10. 111118/112202444.40 2a0.:////22..mmacooooog.... 15ng.258. org/2024 2024.. naacl-long 258. 1, 5,
URL https: /ttp://aaaalaaat
[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023. 2, 3,
[21] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[22] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 (1-2):83-97 97, 1955. 4, 8
[23] Xin Lai. Zhuotao Tian. Yukang Chen, Yanwei Li. Yuhui Yuan. Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9558,, 2024. 1,3
[24] Hyunmin and Jaesik Park. instance-wis occlusion and depth orders natural scenes. Proceedings of the IEFE/CVF Conference on Computer Vision and Pottern Recognition, pages 21210-21221, 2022. 1, 2, 2, 6, 7, 9, 23, 25, 27, 28, 29
[25] Liting Lin, Heng Fan, Zhipeng Zhang, Yong Xu, and Haibin Ling. Swintrack: A simple and strong baseline for transformer tracking. Advances in Neurul Information Processing Systems, 35: 15:16443166754, 2022. 5
[26] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014, 24, 27, 28, 29
[27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). pages 26296-26306, June 2024. 1, 3, 5, 7, 20, 21, 22, 23
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1, 6, 21, 22,
[29] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021. 25
[30] Ilya Loshchilov Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL
7
[31] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacidin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnacve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: http////ppeeeeieeeeeeeeeeeeeeeee net/forumm diaimmmuddassSSUuee
[32] Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with kins dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014-3023,
2019. 2, 7,23
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021. 2.
12

---

[34] René Ranful Katrin Lasinger. David Hafner. Konrad Schindler, and Vladlen Koltun. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2020. 2, 7.
[35] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Ahdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13009-13018, 2024. 1,3
[36] Nikhila Ravi, Valentin Gabeur, Than-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint 2rXiv:2408.007/1 00714, 2024. 23
[37] N Dinesh Reddy, Robert Tamburo, and Srinivasa G Narasimhan. Walt: Watch and learn 2d amodal representation from time-lapse imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9356 9366, 2022. 2
[38] Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks, arXiv preprint arXiv: 14159, 2024. 3,5, 23, 24
[39] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Citrrentiion-MICCO 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015. 23
[40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ane Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[42] Bo Wan, Michael Tschannen, Yongqin Xian, Filip Pavetic. Ibrahim Alabdulmohsin, Xiao Wang, An- dré Susano Pinto, Andreas Peter Steiner, Lucas Beyer, and Xiaohua Zhai. Locca: Visual pretraining with location-aware captioners. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL http:::///ppeeeeiii
wwttffeummmaaeee id- J.k.Akgw.......
[43] Monika Wysoczańska, Oriane Siméoni, Michačl Ramamonjisoa, Andrei Bursuc, Tomasz Trzciński, and Patrick Pérez. Clip-dinoiser: Teaching clip a few dino tricks for open-vocabulary semantic segmentation. ECCV, 2024.3 3
[44] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024, 2,3 3
[45] Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiangang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406. 09414, 2024. 2, 3, 5, 23, 24
[46] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere any granularity. In The Tweifth International Conference on Learning Representations, 2024. URL https: http://///ppeeeeeeeeeeee/ net/ net/forum? id=2mabbX3ydD.
[47] Xiaohang Zhan, Xingang Pan, Dai, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self-supervieed scene de-occlusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3784-378777 3792, 2020. 1, 3, 6, 7,
[48] Ziyu Zhang, Alexander Schwing, Sanja Fidler, and Raquel Urtasun. Monocular Object Instance Segmentation and Depth Ordering with CNNs. In ICCV, pages 2614-2622. IEEE Computer Society, 2015.
[49] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiangang Wang, and lifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021. URL https: / http////ppeevveeeeeee net/ aeee/foommmiddgg2hhieeeeeee
5
[50] Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Dollár. Semantic amodal segmentation. In
Proceedings of the IEEE conference on computer vision and patiern recognition, pages 1464-1472, 2017.
1, 3,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose modernize instance-wise order prediction network with a novel approach surpassing or matching state-of-the art models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We wrote paragraph on that matter and also provide failure cases of our approach in the in the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
We do not prove any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We extensively describe both our method and experimental setup, even for base- lines that are not our main contribution. Moreover, a link to our open source implementation can be found in the abstract of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our implementation is open-source and its link is provided in the abstract of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c
oniioonooooooooo icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc ggwii/gs///////eeeeee /Coeee/dmdSsbmii /ctoe/eummmsssioooooooo.ooooo ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We deseribe the full experimental setting our experiment section and our baselines' settings in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We simply benchmark our approach and compare it to existing approaches on the same benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detailed the GPUs and number of iterations used during our training. We also display inference cost of our approach (memory and runtime speed) benchmark figure.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / /nwuriiurip... cc httpo//iiitthiiiiiiiieooo
Answer: [Yes]
Justification:
We understand and fully adhere to the ethics guidelines of NeurlPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
We do discuss broader societal impact of our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
We do not describe safeguards against misuse our work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are available in the supplementary material section.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release model checkpoints and InstaOrderVQA datasets in open-source repository.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not perform research on or involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not perform research on involving human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We use LLaVA [27] as baseline and extensively detail our setup.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp///errpp/////ee for what should or should be described.
/LLM)
20

---

Appendix
A
Task definition
Since the tasks of occlusion and depth order prediction are not commonly tackled in the computer vision community, we describe the format of these two challenges in details.
Let I E #x be an image. Given that the image contains n E N instances, first discriminate the cases where n = 0 and n = 1. When 7. = O, there cannot be any geometrical relations since there no instances. When itself is undefined. Thus, it only makes sense of trying to compute occlusion and depth orderings in the event that 2. Hence, suppose it is the case.
the geometrical relations between the only instance in the image and
Occlusion and depth orders are to be understood as relations a graph that can itself be represented adjacency matrix G {O..... k "xm, Here, each entry correspond possible relation type. Generally, we assume E N possible relations.
Specifically, The occlusion order matrix Go contains k 2 possible valid entries. Namely, for a pair (i.j) E , nx]]:
G = 0 indicates that instance does not occlude instance j.
G 1 indicates that instance occludes instance j.
call the case G = GJ = bidirectional occlusion.
Similarly, the depth order matrix Gd contains k 3 possible valid entries. Namely, for pair (i.j) , ni]?:
Gd 0 indicates that instance is not in front instance j.
G 1 indicates that instance is in front of instance j.
G 2 indicates that instances and j share similar overlapping depth ranges.
The upper triangle of the depth order matrix conditions the lower part the matrix since G then G 0 and vice versa. Additionally, if G = 2, then G 2.
Note that in both the occlusion and depth order prediction task, computing Gi does not make sense, thus we simply manually fill these entries with -1
Predicting occlusion and depth order relations thus consist in predicting each entry in G. Traditional pairwise networks model this problem the edge-level, meaning that a forward pass of the model predicts a single Gi Such paradigm enforces a quadratic constraint of forward passes with respect the number of objects since the size of the matrix grows quadratically for increasing n's.
On the other hand, we propose to reformulate this problem from and edge-level prediction task to an adjacency matrix-level prediction problem, meaning predicting the full matrix G single forward pass. We call this task holistic order prediction.
B About INSTAORDER-VOL
To evaluate wide range of models, propose baseline for occlusion and depth order prediction based on LLaVA VA [27, 28] (see Tab. in the main paper). In order to evaluate LLaVA on INSTAORDER, first need to convert the occlusion and depth matrices into visual question answering (VQA) format.
B.1 Visual question answering
We convert every pairwise annotation in INSTAORDER to a textual annotation ready to be fed into LLaVA. Specifically, given two instances A and an image, ask LLaVA the following question:
the (Aas) (REL.) the (Bax) word.
Answer the question in single
21

---

Grounding Image
LLaVA
Yes
(i. ÿ-th order
"Is the (CLS1) at (POS1) (REL) the (CLS2) (POS2)
a(n-1) comparisons
INSTAORDER-
VQA
Figure 7: Overview of our VLM pipeline. We use LLaVA [27] to obtain geometrical orderings and evaluate data from our converted INSTAORDER- VQA dataset. We refer the reader to Appendix C.2 for more information.
Here, (REL) indicates the ordering relation, specifically:
For occlusion orderings, (REL] "obstructing".
For depth orderings, (REL) "closer
than".
{Acis) and [Bas} respectively indicate the ground truth class of and the ground truth class B. They are replaced dynamically for each question based on the currently compared instances. This strategy is sufficient to obtain a binary yes-no answer from LLaVA that can further be converted back to its respective element in the occlusion or depth matrix simply by mapping yes to and no to
B.2 Instance-wise disambiguation
However, there still lies ambiguity when two instances in the scene share the same class since then Asts Bass Thus, it is unclear for the model know which string refers which instance in the image. To mitigate that issue, we leverage the fact that LLaVA can process bounding box coordinates and input the coordinates of the instances to clarify the position of the object we refer (Fig. 7).
Therefore, following LLaVA [27, 28]. we encode the bounding box of an ambiguous INSTAORDER instance 4-tuple (aw, Oh, buy bu). Here, represents the top left corner while b represents the bottom right corner of the bounding box. We then normalize the coordinates to fall in the [o, 1] range. If objects be identified without ambiguity, do specify their bounding boxes in the prompt.
Converting the annotations of the 4,071 images of the validation set INSTAORDER to INSTAORDER- VQA yields total of 178,539 VQA ordering prompts, along with their respective ground truths.
C Baselines
C.1 Benchmark specifications
We specify the backbones used for our main benchmark (Tab. 1) the main paper in Tab. 3. Note that Mask2Order and InstaFormer's backbones frozen training time.
22

---

Table
Backbone specifications for benchmarked methods. We refer the reader table for the
actual results.
Method
Backbone
LLaVA [28]
CLIP [33] + Vicuna [10]
Area
Y-axis
PCNet-M [47]
U-Net [39]
OrderNetM I (ext.) [32]
ResNet-50 [14]
InstaOrderNets [24]
ResNet-50 [14]
MiDaS [34]
ResNet-50 [14]
Foundation
DAv2 [45] + SAM [36]
Mask2Order
Mask2Former [9]
InstaFormer
Mask2Former [9]
C.2 LLaVA
Benchmark. For our benchmark (Tab. 1, in the main paper), we use LLaVA 1.5 [27], which is composed of CLIP VÝT-L 336 pixels vision encoder and Vicuna [10] token generator.
We provide two settings for this experiment. the first setup, we simply use the off-the-shelf LLaVV model and perform zero-shot evaluation. Since the results lower than expert baselines, we propose a stronger setting in which we finetune LLaVA onto INSTAORDERVQ.
In both cases, prompt the model with the INSTAORDER- VQQA evaluation set and report the results using the same metries as with the other methods. A description of our pipeline is detailed in Fig. of the supplementary material.
Finetuning. We observed that finetuning the model directly after the pre-training stage results in mode collapse, in which the VIM constantly outputs no for any ordering prompt. Instead, we take visual instruction-tun model and fine-tune it from there on our custom INSTAORDERV QQA data.
We start by splitting the dataset into four categories: occlusion prompts to which the answer is Yes, occlusion prompts which the answer is no, depth prompts to which the answer is yes and depth prompt to which the answer is No. We then randomly shuffle the samples in each category.
Since the number of parameters of LLaVA VA is order of magnitudes above InstaFoormer, opt not to train on the full dataset to obtain a more reasonable comparison. Specifically, we compute the ratio between the number of parameters of LLaVA and our model. We find that our model has 2% of the parameters contained a LLaVA model. Thus, we subsample the training set INSTAORDERV QA to 2% of the annotations. This means that sample 58K annotations for training in total. We evenly select those annotations from the four categories mentioned in the previous paragraph to obtain balanced dataset. We finetune LLaVA for single epoch 8 NVIDIA RTX A6000 with batch size of 16 per GPU using the finetuning scripts from the official repository. We use all the default hyperparameters.
C.3 Foundation models
Foundation pipeline. A depth order prediction can be viewed as a comparison between two instance- level depth maps. Considering this idea, instance-level depth estimator.
compose foundation models together to obtain
We use Grounded SAM [20, 38] and Depth Anything V2 [45] to obtain instance-level depth esti- mations that we can then compare together to obtain depth ordering predictions. diagram of the overall pipeline is depicted in Fig. 8. To propose baseline competitive possible, we select
23

---

Object Detection
Seg. Masks
Grounded DINO
SAM
Depth map
Depth Anything
Heuristic
(6. f-thh depth order
Figure 8: Overview of our foundation model pipeline. We create instance-level depth orderings from composing the predictions of Grounded SAM [38] and Depth Anything V2 [45]. We create the instance-wiss depth predictions by masking the depth prediction using the instance segments generated from grounded SAM. Finally, we extract the prediction using heuristic (min-max, mean, median). We refer the reader to Appendix C.3 for more details.
the best-performing networks available at the time of writing, i.e., Grounding DINO Swin-T', SAM V and Depth Anything V2 VIT-L..
We denote Grounded SAM [38] as fe. segmentation model parameterized by and Depth Anything V2 as 9s monocular depth estimation model parameterized by 6. We start by listing all the instance classes of COCO [26], denoted C {clss..... clak] } in a textual format and use them as text conditioning for Grounded SAM [38] along with the image of interest E IR Hxxxxxx
fo(I:C),
(8)
where S = {s: s E {0. NXx N I that fall in any class of C. Ji= are the binary segmentation masks for N found instances in
We concurrently forward the image into the monocular depth predictor Jø:
D = (I),
(9)
where D R #xW represents the depth estimation of I. We now obtain the segment-level depth maps by masking D using each sy;
Dorder
D
2 S
{D @ si, Vs E S} c [0,1)NxHXW
(10)
where the Hadamard product. We further derive the final depth orderings from Dorder using
simple heuristic.
Depth order heuristics. There are several ways to obtain instance-wise depth orderings given segment-level depth maps. our work and benchmark, we report three heuristics that yield different results (reported in Tab. 1, in the main paper): min-max mean, and median.
The min-max heuristic revolves around extracting the minimum and maximum of each D to obtain a spanned depth range for every instance:
plant min D Drnder) gemil max max(D Damker) : Vi E N1.
(11)
'The Grounded DINO model for Grounded SAM is available at this URI..
2The SAM model zoo available this URI
The Depth Anything V2 model zoo is available at this URL.
24

---

Table 4: Ablation study related the backbone size. We benchmark Mask2Order and InstaFormer for depth order prediction on INSTAORDER [24]. We experiment with various common sizes of Swin transformers [29]. We subseript the number queries for every backbone size and specify IN-21K pre-trained backbones using f. The best results for a model family are yellow and the best overall results are in bold.
(a) Occlusion order prediction results.
(b) Depth order prediction results.
Occlusion acc.
WHDR +
Method
Backbone
Method
Backbone
Precision
Recall
Distinet
Overlap
All
Swin-T
83.23
77.96
76.74
Swin-Tjo
14.73
27.48
19.02
Swin-S nn
83.08
78.17
76.66
Swin-Sjop
14.48
27.66
18.81
Mask2Order"
Swin-B you
83.62
78.06
76.89
Mask2Order
Swin-Buoo
14.31
27.73
18.75
Swin-B lon
83.94
78.39
77.24
Swin-Bjor
14.54
27.39
18.75
Swin- Auo
84.37
78.48
77.49
Swin-Looo
14.16
27.15
18.52
Swin- 100 76.20
84,46
76.11
Swin-Tioo 12.96
27.43
17.78
Swin-S 76.23
85.15
76.36
Swin-Sun
12.67
27.62
17.60
Ma
Swin-B 77.03
84.91
76.78
Mask2Order
Swin-B1o0
12.49
27.29
17.35
Swin-B 76.93
85.06
76.77
Swin-Bjoo
12.70
27.35
17.48
Swin-Lioe 77.51
85,50
77.17
Swin-L 12.29
27.03
17.09
Swin-T 89.06
75.69
79.63
Swin-Tror 8.10
25.43
13.75
Swin-Sjoo 88.91
77.31
80.53
Swin-S100 8.44
26.04
14,48
InstaFormer"
Swin-Buue 89.02
76.95
80.64
InstaFormer
Swin-Bun
8.28
25.05
13.88
Swin-Bjoo 89.53
77.34
80.99
Swin-Bioo
8.15
25.19
13.72
Swin-Lho 89.82
78.10
81.89
Swin-L2o 8.47
24.91
13.73
Swin-Th o
88.64
75.56
79.74
Swin-Tia
8.43
25.36
14.03
Swin-S n0
88 20
75.98
79.57
Swin-S
8.54
25.42
13.96
InstaFormer
Swin-Bren
88.47
75.96
79.72
Swin-Bnon
8.84
25.77
14.39
Swin-Br Jon
89.24
76.66
80.34
Swin-Bios
8.15
25.79
14,06
Swin-LAo
89.57
78.07
81.37
Swin-Lão
7.90
24.68
13.30
Subsequently, sorting the resulting ranges provides the desired depth orderings. In practice, the min-max heuristic is slightly flawed the instance masks are not always perfectly aligned with the instance-level depth maps. This can result in obtaining minimum or maximum values slightly outside the actual instances, generating bias in the sorting process. Instead, we find the other two heuristics simpler and more robust to noise (as results reveal in Tab. 1 of the main paper).
The mean heuristic relies obtaining the mean of each instance-level depth map:
dm mean (nean(Deree) Vi (I....,NN)
(12)
Then, using these scalar values as proxies for the distance of each mask from the camera, we perform sorting obtain the final instance- wise orderings.
The median heuristic works the same way as the mean heuristic, simply differing by using the median instead of the mean:
ameedian median(D.rury Vi (1,
(13)
In practice, we find the median heuristic to perform the best (ef. Tab. 1, in the main paper).
D Adapters
Since we use frozen Mask2Former networks for our experiments, we use adapters to improve the quality of the trainable geometrical ordering predictors of InstaFormer. Specifically, attach adapters from the attention mechanism to the FFNs (feed forward networks) of every transformer layer in the segmentation transformer and dexooedecoder modules [6, 19].
We supervise the adapters using all losses from Mask2Former [9] as well as the losses of the geometrical module, i.e., L. for InstaFormer". La for InstaFormer L. La for InstaFormer and weight them by factor X, = Xa = 5.
We conduct experiments to study the effects of these components on InstaFormer in Appendix E.2.
25

---

Table 5: Ablation study related to using adapters in the segmentation backbone of our In- staFormer networks. We report the results on the occlusion and depth order prediction performances on INSTAORDER for the InstaFormer family of networks. The best results for model family are in yellow and the best overall results are in bold.
Occlusion acc. 1
WHDR
J
Method
Backbone
Adapters
Precision
Recall
F1
Distinct
Overlap
All
89.00 89.82
75.21
79.71
InstaFormer
Swin-L200
78.10
81.89
17.10
39.37
23.52
InstaFormer
Swin-L200
8.47
24.91
13.73
88.96
75.81
80.05
17.34
39.58
23.40
InstaFormer
Swin-L200
89.57
78.07
81.37
7.90
24.68
13.30
E
Ablations studies
E.1
Segmentation backbone size
We perform an ablation study on the size of the segmentation backbone with respect to the perfor- mance for occlusion and depth ordering prediction for both Mask2Order and InstaFormer. In all experiments, the backhones are frozen and we only train the geometrical predictor on top of them.
Occlusion order recovery. We report the results when varying the backbone of the segmentation network from Swin-T to Swin-Ljoo in Tab. 4a. For Mask2Order", we observe a noticeable performance increase when using IN-21K pre-trained backbones. We observe similar trend for Mask2Order'd.
This is similar to both InstaFormer and InstaFormer where the most noticeable performance improvement comes from increasing the queries to 200 and the backbone size Swin-L, which in both cases results in a full point of improvement over their Swin-Bimo counterparts. While pre-training the backbones ImageNet-21K for InstaFormer seems beneficial, it only yields half point of F1 improvement, whereas increasing the queries almost leads to an approximate I point of F1 increase.
Depth order recovery. We report the results when varying the backbone of the segmentation back- bone from Swin-T to Swin-L in Tab. 4b. These results show very different trends with respect to the occlusion order backbone ablation. For Mask2Order" networks, they seem hastily plateau around 18.75 WHDR (all) with slight improvement from Swin-T to Swin-Biwe Increasing the number of queries from 100 to 200 slightly fosters performance. We note the positive effect of simul- taneously performing both the task of occlusion and depth predictions as the results of Mask2Order Swin-Tion Too already overcomes those of Mask2Order Swin-L Moreover, Mask2Order Swin-L-po beats its Mask2Order counterpart by 1.43 points of WHDR (all).
Task difficulty. We observe that the holistic task of depth order prediction seems to be casier to perform than the task of occlusion order prediction since the WHDR discrepancy from InstaOrderNets and InstaFormer is larger than their Fl difference (cf. Tab. of the main paper) and since changing from a weaker to a stronger backbone mostly benefits the occlusion order prediction networks (from Tab. 4a and Tab. 4b).
A key property of depth matrices is that they always contain (=(==1) positive values since objects must be one behind the others. This is not the case in occlusion order prediction, where no prior on instance layout exists. Moreover, depth is consistent across instances. Consider A and B two instances of the image. If we have -> B, then, in front of B and thus we know that we can transfer all the depth relations of A to B. On the other hand, this property does not apply to occlusion order prediction.
Note that this property only emerges when switching to the holistic paradigm since we can draw information from all instances at once. We believe this property important explaining the strong results for depth order prediction and saturation across backbone sizes.
26

---

E.2 Adapters
We ablate the adapters of InstaFormer models. The results of these experiments are reported Tab. 5.
Occlusion order recovery. The main observation is that adapters are important for obtaining strong performances on the occlusion order prediction benchmark, Most notably, InstaFormer" benefits from an Fl improvement of 2.18. We observe that most of that result is explained by the increase in recall.
Depth order recovery. The depth ordering benefits the most from the use of adapters. As matter of fact, InstaFormer shows spectacular 10.10 improvment of WHDR all. This result is explained by a sharp reduction in both distinct and overlap WHDR. We observe similar trends for InstaFormer'.
F
Inference cost
F.1 Runtime comparison
We benchmark the runtime inference of our holistic InstaFormer and compare it against pairwise InstaOrderNet, in Fig. 5a the main paper. Runtime for our proposed InstaFormer appears constant with respect to the number of instances in the image, while InstaOrderNet is exponential. Moreover, InstaFormer's s efficiency amortized for any image containing more than instances and up to 12 times faster than the baseline when the image contains 20 instances. This makes our approach suitable for prediction on natural images since such scenes usually contain many objects.
We also provide an analysis on a batched version of InstaOrdenNet where the pairwise relations are sent the network single batch. While this approach is the fastest, we highlight its high memory requirements, which makes it impractical for users with limited resources or embedded systems.
F.2 Memory consumption
We further benchmark the memory consumption of our InstaFormer network and compare to InstaFormer in Fig. 5b of the main paper. The memory consumption of our InstaFormer grows linearly with the number instances on the image, while InstaOrdenNet's remains constant. This highlights the trade-off between inference speed and memory consumption. We argue that InstaFormer still wins that trade-off since its runtime is constant while InstaOrderNet's exponential.
We also provide an analysis on batched version of InstaOrderNet, where the pairwise relations are sent to the network in a single batch. We observe that the memory requirements quickly become prohibitive. In fact, we encounter CUDA out-of-memont error for any image containing more than 20 instances. We thus stop the benchmarking at this threshold. This a critical drawback, as natural scenes usually depict a large number of instances at once.
F.3 Scalability
In Tab. 6, we compute the number of average instances detected in images as well as the number of maximum predicted instances in single sample across the validation set of COCO [26]. We vary the number of object queries, but we observe both cases that the number of instances predicted in a sample is close above to 11. Comparing this result to Fig. 5a and Fig. 5b, conclude that it is on average better to use InstaFormer than other baselines since natural images contain, on average, close above 11 instances.
The second column of Tab. emphasizes the fact that our InstaFormer network can scale way above 10 instances although the training set of INSTAORDER is limited to annotations containing 10 objects [24]. As matter fact, InstaForrmer with 200 object queries detects 63 instances on an image of the COCO validation set. In both 100 and 200 object queries settings, the value of maximum instances detected far below the total number of object queries. This leads us conclude that InstaFormer is practical for general inferences in natural scenes.
27

---

Num. Queries Avg. Predicted Instances (per sample)
Max. Predicted Instances (in single sample)
100
10.91
56
200
11.91
63
Table 6: Inference statistics of InstaFormer. We compute the number of average predicted instance per samples and the number of max predicted instances in a single sample on the full COCO validation for model with different number of object queries.
G Qualitative results
G.1 Comments on results from the main paper
In this section, we comment on qualitative results from the main paper in more detail.
In Fig. 1 of the main paper. we display two in-the-wild results. In the "athletics" example, we note the occlusion from the hand of "PersonI" to "Person3" that was effectively captured from our model. Note that this occlusion is difficult to notice without carefully observing the input RGB image.
Still in that figure, the "table tennis" example shows interesting bidirectional occlusion examples where the handling of the rackets properly related to their players in the occlusion graph. Note that in both cases, the depth graph, while crowded on the second example, also depicts accurately the layout of the instances in the scene. This is obvious for the "athletics" example. For the "table tennis" example, careful inspection reveals no sign of unrealistic depth ordering.
Analyzing Fig. 4, in the main paper, the first contains "clephants" and "K-Pop". which are both in-the-wild predictions from our network. We observe an ambiguous occlusion case in the "elephants" example, where "Elephant3" does not on the occlusion graph. While it is ambiguous whether "Elephant3" is closer to the camera than "Elephantl", the network still appears to predict coherent order.
to occlude "Elephant2" even though reported as such
The second image of the first row of Fig. 4, namely "K-Pop" an impressive example of accurate depth order prediction in complex scene. While the input RGB image solely depicts silhouettes, our InstaFormer network still predicts an accurate and coherent depth layout for the scene. In that case, the network shows difficulties understanding the occlusion orderings for some elements since there are little clues to rely on. For example, "Personó" and "Person4" are linked with a bidirectional occlusion even though it seems that it should be unidirectional from "Person4" to "Person6", The same holds for "Person 1" and "Person3".
All the remaining images of Fig. 4 are extracted from the validation set of INSTAORDER [24]. We specifically highlight the "donuts" example emphasize that the model can still predict accurate orderings even crowded scenes.
On the second and second column, the "plate" example highlights perfect prediction in simple scenario.
On the last of Fig. 4, the "cooking cat" example shows that our model can predict sane relations even if the "Cat" is blurred. As a matter of fact, the cat indeed occludes the "Oven", yet the "Oven" does not. Careful inspection reveals that the blurriness of the cat lets appear notch of the "Oven" that visually occludes the "Cat" even though it is physically impossible.
Finally, the last example on the last row and last column of Fig. 4, called "stop sign". shows neat case of a prediction of perfect occlusion and depth orders, even though the size of the cars is noticeably small. We also emphasize this result as no occlusion between elements occurs. We recall that in COCO [26], the stop sign class only annotated on the sign itself, and not the pole.
G.2 Ground-truth visualizations
We visualize the ground- -tunddtruutssaae for the INSTAORDER validation set prediction of Fig. 4 in Fig. 9 and Fig. 10 on Fig. 11. Note that, in some rare cases, our network was unable to detect some of the instances in the image, leading to no ordering prediction (Fig. 9, "donuts", "cat in the oven"). However, more frequently, it predicted more instances than the ground-truth annotation, for example, "sheep2" is accurately detected along with its proper occlusion ordering (first row, last column of
28

---

Occlusion
Occlusion
Spezss
Space
DOURRT
Fark
Knifé
Darcrt
Kintf
Domail
lie Dugg
Deeaso
Depth
Depth
Doour
Perun DenasS
liblic
Spaci
Denut
Danatk
Doygi
Dlg
Fork
Kaife
Hox Dog
Tanks
Occlusion
Occlusion
STOP
Clair
Seep
Renunn
STOP
CHl
Depth
Depth
Ovei
Renunn
Step
G
Figure 9: Ground-truth visualization of the predictions from Fig. 4. We visualize the order graphs and the masks from the INSTAORDER dataset and overlap their respective COCO labels on top of them. However, we re-use the same colors as the predicted classes from InstaFormer in Fig. for better comparison with the predictions. We use a white box with colored border to indicate ground-truth that have not been matched from our network's predictions. Annotated objects on the image not present the graphs symbolizes that our network predicted these nodes even though not present in the ground truth.
Fig. 10 and Fig. 11). This is also the case for the teddy bears (see Fig. 10 and Fig. 11), Sometimes, network predicts even better orderings than the ground-truth, for example, in Fig. 10, in the cake picture (third row, second column), the network predicts a bidirectional occlusion between "Person3" and "Cake". This seems more reasonable than the ground-trute -tuth annotating unidirectional order (Fig. 11). In some cases, the objects are too thin to be detected, in which case the predicted orderings still sane, although they could probably be more precise ("kite", 4th row, 2nd column on Fig. 10 and Fig. 11)
G.3 More qualitative results
We provide even more qualitative results obtained from InstaFormmr on INSTAORDER in Fig. 10. And report their ground-truths in Fig. 11.
H Web image sources
We provide the link all the web images used this paper.
Figure the main paper contains two images, both of which were extracted from the web. Left image "table tennis". Right image "athletics".
Figure of the main paper is composed of rows, out of which only the first one is composed of web images. First row, left image "elephants". First row, right image "K-Pop".
All the remaining images present in our paper originate from the dataset [24, 26] (specifically from the validation set), which is licensed under a Creative Commons Atribution 4.0 License.
INSTAORDER
29

---

Lip
How
Sundwrdh
Applil
Exp
Start
Occlosion
Depth
Occlusion
Depth
Occlusion
Depth
Apple
Arple
Shappl
Steppe
Steapd
Saagpi
Shappi
Saedlanch
Appiei
Applel
Sharplil
Sihear
Staeeti
Tidddy Rizer
Tedds Baai
Plaxao
Big
Prront
sppup
Occlusion
Depth
Occlusion
Depth
Occlusion
Depth
Teddy Beal
Tesbly Beal
Dresodd
Preura
Parood
Talll Boat
Plannd
Nencel
Tesdy Hears
Hears
Paosee
Posce2
Ridd
Teptop
Tapop
Pinool
Beg?
Seal
Bagg
Dos
Paper2
Deronn
Cake
Pasezi
Occlusion
Depth
Occlusion
Depth
Occlasion
Dopth
Panson
Prnood
Reya
Beg
SEIT
Fanweiti
Tlog
Kiol
Buttle
Pirora
Kiel
Call
Occlusion
Depth
Occlusion
Depth
Occlusion
Depth
Kike
Kiaa
Cinel
Cosa
Kixa
Knoel
Low2
Cou?
Penso
Posse
Kiel
Koel
Figure 10: More qualitative results for o All the images from the validation set of INSTAORDER.
30

---

Cup
Buwl
Sheert
Sandwach
Apsiel
Sleapd
Occlusion
Depth
Occlusion
Depth
Occlusion
Depti
Bowill
Booll
Apricl
Arpiel
Talle
Spaeni
Talle
Space
Shegg
Streer
Toddy Boaz
Blgt
Couch
Perouzl
Occlusion
Depth
Occlusion
Depth
Occlusion
Depth
Froood
Taddy
Tadty
Teadl Hears
Tecdy Hears
Renson?
Paneell
Pansel
Hozz
Seri
Patsont
Pomuee
Ponooz
Cake
Occlusion
Depth
Occlusion
Depth
Occlasion
Depth
Hepc
Pieeer
OA
Cake
Frroort
Kikaa
Perzus
Fall
Occlusion
Depth
Occlusion
Depth
Occlusion
Depth
KIN'S
KIDER
Kieck
Coo2
Cow2
Penzz
Figure 11: Ground-truth visualization of the predictions from Fig. 10. We visualize the order graphs and the masks from the INSTAORDEE dataset and overlap their respective COCO labels top of them. However, we re-use the same colors as the predicted classes from InstaFormer in Fig. 10 for better comparison with the predictions. We use white box with colored border indicate ground-truth that have not been matched from our network's 's predictions.
31


Reviews:
[{"review_id": "mMaW5GIQTN", "reviewer": "Reviewer_SiaZ", "summary": "The authors present a method for predicting object order (occlusion relations and depth ordering). The nice new aspect of the method is that it generates order in a single forward pass by modeling the interaction between object queries and latent mask descriptors. The experimental results are comprehensive, by comparing the method to past works, including three parallel works. Results are promising.", "questions": "- Motivation is a bit weak. Highlighting an important use case could make a reader more excited about the area of work.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Modeling order in a single forward pass with good results.\n- Motivating why order is important by linking it with the VLM which improves performance in depth based reasoning.\n- Clear write up\n- Comprehensive experiments\n\nWeaknesses:\n- The related work section could benefit from the authors reviewing some older works on the subject. For example, \"Monocular Object Instance Segmentation and Depth Ordering with CNNs\" by Zhang et al plus references within could be a good reference for older works. \n- Motivating why depth ordering is important could be done more convincingly. I'm still not sure where someone would actually use this method in practice.", "was_revised": false, "final_justification": "", "rebuttal": "# Authors' response to reviewer SiaZ\n\nWe deeply appreciate reviewer SiaZ's thoughtful comments and suggestions. We are glad they found\nour method interesting, our benchmark comprehensive and our writing clear. We address each of\ntheir comments below:\n\n## Weaknesses\n\n> The related section could benefit from the authors reviewing older works on the subject.\n> For example, \"Monocular Object Instance Segmentation and Depth Ordering with CNNs\" by Zhang\n> et al. and references therein.\n\nWe thank reviewer SiaZ for pointing this relevant work to us. We agree that this paper falls in\nthe range of our work by proposing a pioneering approach in the field of depth order prediction.\nWhile the suggested work did not provide open-source implementation of their method, **we plan\non citing it in the related work section of the final version of our paper.**\n\nThanks to reviewer SiaZ's suggestion, we also found the paper \"A Learning-Based Framework for\nDepth Ordering\" by Z. Jia et al. to be a relevant depth order art and will also cite\nit in the final version of our paper.\n\nWe will continue to look for other relevant works and make sure to include them.\n\n> Motivating why depth order is important could be done more convincingly. I'm still not sure why\n> someone would actually use this method in practice.\n\nWe thank the reviewer for offering us the opportunity to clarify the motivations behind predicting\ndepth order relations in a scene. We currently think that there are several compelling areas in\nwhich depth order relations are crucial and where our method can be applied effectively:\n\n1. **Real world applications**\n   - **Autonomous driving**: understanding the depth order of objects in a scene is crucial for\n     safe navigation and obstacle avoidance.\n   - **Augmented reality (AR)**: depth ordering is essential for rendering virtual objects in a\n     way that they appear naturally integrated into the real world.\n   - **Positional verification in sports**: in sports, depth order awareness can become crucial to\n     assess if a rule has been violated. We give a two examples here:\n     - In athletics: depth orders can determine the finish order of runners (as illustrated\n       in Fig. 1 of the main paper).\n     - In football: depth orders can be used to determine if a player is offside or not.\n2. **Vision language models (VLMs)**\n   - We show that predicting order graphs before VLM inference can improve the spatial\n     understanding of VLMs in a zero-shot manner (Fig. 1 of the main paper). This is particularly\n     useful since VLMs often struggle with spatial reasoning, while fine-tuning them is usually\n     expensive in terms of data and compute resources.\n3. **Robotics**\n   - Recently, VLMs have been used in robotics in the form of perception modules.\n     Thus, obtaining fine-grained spatial relations of objects in the surrounding world becomes\n     crucial for grounded and safe interactions between the robot and its environment. In some\n     cases the description of an object to pick up or manipulate are given by its location in space\n     (e.g. \"the object behind the red box\"), which can be poorly interpreted by VLMs. As shown in\n     Fig. 1 of the main paper, providing depth order relations can help the robot to identify the\n     object to be manipulated accurately.\n4. **Image editing**:\n   - Recent works such as \"LooseControl: Lifting ControlNet for Generalized Depth Conditioning\"\n     S. F. Bhat et al., or \"Build-A-Scene: Interactive 3D Layout Control for Diffusion-based\n     Image Generation\" by A. Eldesokey & P. Wonka, use monocular depth estimation as a mean\n     to edit images efficiently. Although this requires further research, we believe that our\n     method could be used to build similar models without relying on a full depth map of the scene.\n     Generally speaking, the field of layered image editing, where each object is considered as a\n     different plane on the 3D space, is a promising area where our approach has chances to be\n     applied effectively.\n\nAs suggested by the reviewer, the paper \"Monocular Object Instance Segmentation and Depth\nOrdering with CNNs\" also provides examples of cases where our approach could be used in practice:\ndriver assistance, image captioning with spatial arrangements, Q&A retrieval systems, and 3D scene\ndescription.\n\n**We will rewrite some part the introduction to include some of these examples and motivate\nour work more effectively.**\n\nWe hope that these examples clarify the motivations behind our work and its potential applications.\nWe thank reviewer SiaZ again for giving us the opportunity to clarify these important points."}, {"review_id": "RJgIghQn0v", "reviewer": "Reviewer_iz2C", "summary": "The authors introduce a nicer design for order/occlusion prediction by internally combining a segmentation architecture (Mask2Former) and its resulting object queries to predict orderings/occlusion in a full attentional approach. This is easy to train end to end, and can rely more on object-centric context in order to make its predictions. Previous work was less integrated or relies on specific pairwise networks to produce these predictions. The authors show a variety of good baselines (including VLMs, and a more basic Mask2Former approach), while their method generally outperforms existing methods.", "questions": "1. More information on how Mask2Order was trained. Is it trained (always) on results from Mask2Former or is this only at inference time?\n2. The final prediction is a dot product which is just binary classification. How does this determine the {1, 0, -1} results needed for occlusion ordering? Is it separate MLP for each \"class\"?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Everything here makes _sense_ and seems superior to previous designs. It's a solid design.\n2. This appears to, in turn, show solid results on the ordering/occlusion benchmarks.\n\nWeaknesses:\n\n1. While the authors appear to slight existing methods for requiring masks, the resulting approach of InstaFormer is still lacking in that it requires a full decode of an instance segmentation mask. The main innovation comes the idea that full Transformer attention of mask/image-derived features can be used to perform the pairwise prediction tasks in one shot (e.g. LightGlue, etc). Their ablation shows that it appears to heavily depend on this _explicit_ spatial information which is somewhat disappointing.\n2. The authors could spend less prose on comments like \"making their real-world inference impractical as the masks might be unavailable to the user.\" This detracts from the paper since it makes it seem like the idea of using predicted masks instead of the ground-truth masks is a heroic leap of faith.\n3. The idea that one can use the dot product here rather than separate classification steps is good, but it seems a little far fetched to call this \"holistic\". Is this implied from the idea of self attention? Is it from using the dot product for prediction (but this doesn't seem particularly holistic?).", "was_revised": true, "final_justification": "My questions have generally been answered regarding missing ablations and comparisons. I think this paper is still worthy of acceptance, however, I would not necessarily champion it if need be since I think the impact and novelty of the method remains marginal, but this is likely a solid paper in the framework of the task the author's have chosen (i.e., I do not necessarily think the authors can do much to improve this paper in itself).", "rebuttal": "# Authors' response to reviewer iz2C\n\nWe would first like to thank reviewer iz2C for its detailed and positive feedback.\nWe strongly appreciate reviewer iz2C's acknowledgment of the soundness, efficiency and performance\nof our design. We are pleased to hear that they found our baselines comprehensive and benchmark\nconvincing. We reply to their considerations as follows:\n\n## Weaknesses\n\n> InstaFormer requires a full decode of an instance segmentation mask.\n\nWe agree that InstaFormer relies on the output obtained from the Mask2Former segmentation\nbackbone. **This means that the detection of the instances influences the order prediction of the\ngeometrical order module**. We observed cases where instance segmentation failure resulted in order\nprediction mismatch.\n\nNonetheless, we would like to highlight that **we are not using the masks as raw inputs to our\ngeometric order prediction head. Instead, we just use the masks to mask the features of the\nper-pixel decoder.** We believe that this is a substantial nuance that is worth keeping in mind.\n**Our network input's remains in the latent space of the segmentation backbone.**\n\nMoreover, we would like to emphasize that obtaining the segmentation masks as the output of the\nnetwork is something desirable. The segmentation masks are what \"ground\" the order predictions in\nall cases. Supposing we would not decode the masks, it would be hard to interpret the entries of\nthe order matrices since we would not be able to tell which row and column correspond to each\ninstance in the image. In the case where the instance label is unique in the scene, one could argue\nthat the predicted label would be sufficient to tell the entries apart. However, this solution is\nnot robust in the case where multiple instances of the same class are present in the image (which\nis frequent on most natural scenes). **In that case, it is mandatory to obtain the masks from the\nsegmentation backbone to be able to interpret the output order graphs properly.**\n\n> About \"making real-world inference impractical as the masks might be unavailable to the user\".\n\nWe hope that this repetition did not interfere with the reading experience in a negative way.\nWe believe that mitigating this limitation constitutes an important contribution of our work, yet,\n**we will make sure to only mention it once in the final version.** We agree that this idea does\nis an \"heroic leap of faith\", yet, we still believe that it constitutes a significant and natural\nevolution in the paradigm of order prediction.\n\n> The use of the dot product rather than separate classification is good but does not seem\n> sufficient to be called \"holistic\". Is this implied from the idea of self attention?\n> Is it from using the dot product for prediction ?\n\n**The reason why we called our network \"holistic\" is because it can obtain all the order relations\nin a single forward pass as opposed to previous baselines which only behave at the pair level.**\nThis property naturally emerges from formulating the order prediction problem as a full graph\ninference and solving it using a transformer architecture where each token represents a single\ninstance. Thus, **reviewer iz2C is right in thinking that this comes from the idea of self\nattention. Since self attention behaves at the token level and since each token has been carefully\ncrafted to represent an instance, we naturally draw the order relations for _all_ the object pairs\nin a single transformer pass.** The final dot product only acts as some aggregation operation to\nfall back in the $N \\times N$ matrix of $N$ detected instances in the image. One can think of it\nas some object-wise relational compatibility in the latent space.\n\n## Questions\n\n> More information on how Mask2Order was trained. Is it trained (always) on results from\n> Mask2Former or is this at inference time ?\n\nWe thank reviewer iz2C for this question which gives us the opportunity to clarify our setup and\npropose a follow up experiment.\n\nResults reported for Mask2Order in table 1 of the main paper were obtained as follows.\nWe trained InstaOrderNet from scratch using the same recipe as in the original InstaOrderNet paper.\nOnce training done, we concatenated InstaOrderNet and Mask2Former and conducted evaluation of order\npredictions on the Mask2Former generated masks. We train on InstaOrder and perform inference\non Mask2Former generated masks. **Results of Mask2Order are at inference time**.\n\nReviewer iz2C's question gives us the opportunity to train a new version of Mask2Order, which\nuses the **Mask2Former masks as inputs to the order prediction head even during training**. **We\nfollow the same training recipe as in the original InstaOrderNets** with a single exception.\nAs we empirically found this task harder than order prediction on GT masks, we modernize the\noptimization routine by using AdamW with a learning rate of 0.001 and a cosine decay schedule\ninstead of the SGD optimizer with step decay from the original InstaOrderNet paper.\n\nWe report the results of this new version of Mask2Order in the following table:\n\n|Method|Precision|Recall|F1|WHDR (distinct)|WHDR (overlap)|WHDR (all)|\n| :- | :-: | :-: | :-: | :-: | :-: | :-: |\n|Mask2Order^od|79.81| 86.86|79.09|12.44|28.44|18.19|\n\nCompared to the original Mask2Order results, we observe a significant accuracy boost for the\nocclusion order prediction task (+1.92 points in F1 score for Mask2Former^od), while the\ndepth order performance slightly decreases (+1.1 points in WHDR all for Mask2Former^od).\n\nYet, these results still all fall short when comparing them to the original InstaOrderNet results,\nhighlighting the difficulty of end-to-end order prediction while strengthening the efficiency and\nthe importance of our InstaFormer architecture. **Indeed, we observe that the InstaFormer models\nall outperform both frozen _and_ trained Mask2Order models.**\n\nWe lacked time to perform the occlusion and depth only experiments in this rebuttal, but will\nadd them during the conversation phase (if we have the opportunity) and in the final revision of our manuscript\nalong with an analysis. We thank reviewer iz2C for providing this experimental suggestion to us.\n\n> The final prediction is a dot product which is just binary classification. How does this\n> determine the {1, 0, -1} results needed for occlusion ordering? Is it a separate MLP for\n> each \"class\"?\n\nWe encourage the reviewer to follow along Fig. 3 of the main paper to understand the following\nexplanations more easily.\n\n**One can think of the final dot product as an object-level relational compatibility**. The\nrepresentations of the object queries features $Q^\\*$ and object descriptors $D^\\*$ are both\n$N \\times C$ and $N \\times C$. Thus, $G = Q^\\*D^{\\*T}$ results in a $N \\times N$ matrix which is\nsent to a final MLP (MLP$_{\\omega}$ in figure 3 of the main paper. **Note that the final MLP is\ntask dependent, meaning there are 1 MLP for occlusion and 1 MLP for depth**).\n\n**The output dimension of the final MLP is either 2 for the occlusion task or 3 for the depth\nordering task.** Indeed, the occlusion task requires to predict whether an object is in\noccluding/hiding, with label 1, or has no occlusion relation with another object, with label 0.\nThe depth order prediction task requires to predict whether an object is in front of another,\nwith label 1, or behind with label 0. There is a third label, label 2, which corresponds to\nobjects sharing the same depth range.\n\nWe manually remove the diagonal of the matrix since it does not make sense for an object to\nocclude itself or be in front/behind itself and compute the loss on all the remaining elements of\nthe matrix. **Thus, we confirm to reviewer iz2C that there are no separate MLPs for each class.**\n**The final MLP treats all entries in the matrix uniformly and its goal is simply to output proper\ndimensions for each of the tasks.**\n\n**We use BCE for the occlusion order prediction task and CE for the depth order prediction task.**\nAt inference, we argmax the predictions on both the occlusion and depth order logits and manually\nfill the diagonal with -1 (this value is arbitrary).\n\nIf more questions arise, we kindly refer you to our answer to reviewer i3cT where we provide\nexplanations to other details of the tasks. If your questions remain, we would be pleased to\nanswer them during the discussion phase."}, {"review_id": "KwffHUWDDZ", "reviewer": "Reviewer_NnW7", "summary": "This paper proposes a solution for simultaneously predicting occlusion and depth orders of all instances in a scene from a single RGB image in a single forward pass. The key novelty lies in predicting the ordering matrix ​**G**​ in a single forward pass, thereby eliminating the need for pairwise inference.", "questions": "I must acknowledge that I am not an expert in this field; however, after reading this paper, I have the following questions:\n\n- **Scalability:** Can your method handle more than 10 instances? Handling fewer than 10 instances appears impractical. The maximum number of instances might be constrained by the value of N. Perhaps incorporating more tokens could address this.\n\n- **​Inference Efficiency:** How does the speed of your solution compare to pairwise methods? For instance, reporting Frames Per Second (FPS) on 1024×1024 images would be beneficial. Pairwise methods might achieve faster inference through batch processing, potentially outperforming the original approach. Therefore, it's essential to ensure that comparisons are fair and that both implementations are optimized appropriately.\n\n- **​Readability of Figure 3:** The rationale behind designing the method as depicted in Figure 3 is not entirely clear. For example, could you elaborate on the rationale behind using a transformer with only one layer? There may be underlying motivations that make your design unique and theoretically superior to alternatives. It would be helpful to elaborate on these motivations before delving into the specifics of the design.\n\n- **Theoretical Advantages of Predicting ​G​ in One Pass:** Does predicting the ordering matrix ​G​ in a single forward pass offer any theoretical guarantees of superiority over previous methods in terms of accuracy, independent of empirical results? It would be beneficial to understand if there are inherent theoretical benefits that position this approach as fundamentally more accurate or robust compared to traditional pairwise inference techniques.", "limitations": "The paper does mention certain limitations of their approach. However, it would be beneficial if the method could handle an unlimited number of instances, thereby increasing its practicality and effectiveness in more complex scenarios.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\nThe paper show state-of-the-art (SOTA) performance relative to pairwise methods.\nIt presents seemingly comprehensive experimental results and ablation studies.\n​\n\n**Weaknesses:**\nThe computational efficiency claims lack empirical validation, such as frame per second (FPS) metrics or memory usage, which are essential for assessing practical applicability. The scalability of the method beyond the dataset's 10-instance limit remains unverified, raising questions about its real-world applicability in scenarios with a larger number of instances.", "was_revised": true, "final_justification": "The authors have effectively addressed all of my concerns.", "rebuttal": "# Authors' response to reviewer NnW7\n\nWe are pleased to hear that reviewer NnW7 appreciates our work and points out the strong and\ncomprehensive results of our approach while adopting a novel design. We thank reviewer NnW7\nfor its precious comments and we address them as follows:\n\n## Weaknesses\n\n> The computational efficiency claims lack empirical validation such as FPS or memory usage,\n> essential for assessing practical applicability.\n\n**We ran a computational efficiency comparison experiment in section F of the supplementary materials**.\nWe kindly refer reviewer NnW7 to the aforementioned section and figure 7 of the supplementary\nmaterials for a detailed analysis of the computational efficiency of our approach compared to\nthe InstaOrderNet baseline.\n\nNonetheless, we will quickly summarize the results. We compared InstaFormer against the\nInstaOrderNet baseline since this is the most recent art and competitive approach. We used the\nofficial implementation of InstaOrderNet but also realized that there is a more efficient\ninference procedure that consists in batchifying all the pairs of instances together and\nperforming inference on them in a single batch. We refer to this as InstaOrderNet (batched).\nFor all InstaOrderNet variants, we resize the image to 256x256 pixels. For InstaFormer, we\nfollow Mask2Former's inference script and do not resize the images. Please, note that this\nresults in a disadvantage for InstaFormer since the images have a larger resolution.\n\nWe compute inference speed in seconds and memory usage in Gb with respect to the number of\ninstances in the order graph.\n\n**For inference speed, we observe that InstaFormer appears with a constant inference time**, while\nInstaOrderNet appears exponential with respect to the number of instances. Our approach's average\ninference time lies below 1 second. We emphasize the fact that we have not yet optimized the method\nfor speed. Above 11 instances, InstaFormer becomes faster than the default InstaOrderNet. On the\nother hand, the batched InstaOrderNet is consistently faster than both InstaOrderNet and\nInstaFormer.\n\nHowever, this batched performance comes at a cost. Comparing the memory usage, we observe that inference\non the batched InstaOrderNet rapidly becomes prohibitive. As a matter of fact, its memory\ncomplexity grows exponentially with respect to the number of instances in the scene. **On the\nother hand, InstaFormer shows a linear memory growth. Above 10 instances, InstaFormer's cost\nfalls below this of InstaOrder (batched). At 20 instances it saves almost 30Gb of memory compared\nto the batched version of InstaOrderNet**. We encounter CUDA OOM errors for any inference above\n20 instances using InstaOrderNet (batched), thus we stop the benchmark here. Naturally, the\ndefault InstaOrderNet memory usage remains constant with respect to the number of instances.\n\nWe believe that these results show that **InstaFormer obtains the best trade-off between runtime\nspeed and memory usage**, especially when instances in the scene grow beyond 10. Further\noptimizations of the inference method could lead to even better results.\n\n> The scalability of the approach beyond the dataset's 10 instances limit remains unverified.\n\nWhen benchmarking our approach in figure 7 of the supplementary materials, we benchmark results\nup to 20 instances in the scene which supports the scalability of our approach.\n\nWe also empirically observe that **our approach scales well largely above 10 instances by\npredicting the order graphs on the whole validation set of COCO**. We report the average and\nmaximum number of instances returned by InstaFormer^o,d in the following table:\n\n|Num. Queries|Avg. Predicted Instances (per sample)|Max. Predicted Instances (in a single sample)|\n|:-|:-:|:-:|\n|100|10.91|56|\n|200 |11.91|63|\n\nIn all cases, **we observe that we are not bounded by the amount of queries we use to predict the\norder graphs**. Moreover, we showed the computational advantages of using InstaFormer over\nbaselines when the number of instances in the scene grows beyond 10 in the previous question.\nThis is on average the case in COCO's natural scenes.\n\n## Questions\n\nWe thank reviewer NnW7 for their important questions. We address them as follows:\n\n> Scalability.\n\n> Inference efficiency.\n\nPlease, refer to the previous section in which we address both these items.\n\n> Readability of Fig. 3. Could you elaborate on the rationale behind using a transformer with only\n> one layer? There may be underlying motivations that make your design unique and theoretically\n> superior to alternatives. It would be helpful to elaborate on these motivations before delving\n> into the specifics of the design.\n\nWe thank reviewer NnW7 for offering us the opportunity to clarify the design choices of our\nproposed InstaFormer network.\n\n### Task formulation\n\nThe main motivation behind our method is to **reformulate the problem of pairwise order prediction\nas a full adjacency matrix prediction**. We can now think about the problem in reverse order:\nif we start from the adjacency matrix of the order graph $G$, we can think of each element $G_{ij}$\nas the result of an **order compatibility score between the instances $i$ and $j$**. We can then\nmodel this score by performing a simple dot product over tokenized representations of $i$ and $j$.\n\nSince the transformer inherently behaves at the token level, the choice of using a transformer\nnaturally emerges at this point. Yet, there remains a question: \"how to encode the instances into\ntokens?\".\n\n### Instance encoding\n\nAblation in table 2a of the main paper shows that simply using the queries of Mask2Former as\ninput is not an optimal choice. However, we Mask2Former's per-pixel decoder latent features are\naligned with the query instances of the transformer (i.e. represent the same instances) while\ncontaining complementary information. These features are crucial determining delicate order\npredictions since they are pixel-aligned. Thus, we create descriptors from the per-pixel latent\nspace of Mask2Former and observe (still in table 2a of the main paper) that this formulation of\nthe compatibility score results in the best performance.\n\nWe would like to mitigate any ambiguity by recalling that the main transformer decoder of InstaFormer\ncontains 8 layers. However, reviewer NnW7 is right in pointing out that the encoder for the\ninstance-aligned per-pixel features of our geometrical order head contains a single transformer\nlayer.\n\nWe empirically observed that simply max-pooling the features results in lower performances (cf.\ntable 2b of the main paper). **Adding a transformer layer dramatically increases the performance\nof the model**. Our motivation for adding this layer resides in the fact that **the\ninstance-aligned per-pixel features of Mask2Former have not been able to share critical spatial\ninformation used by InstaFormer to determine the order relations between instances.**\nSelf-attention at the instance level allows the model to generate much more comprehensive and\nrobust instance descriptors that, in turn, lead to better performance across occlusion and depth\norder accuracy.\n\nThanks to reviewer NnW7's comment, we extended this ablation and increased the number of\ntransformer layers before the max-pooling operation. We present the results in the following table:\n\n|Pooling strategy|Precision|Recall|F1|WHDR (distinct)|WHDR (overlap)|WHDR (all)|\n|:-|:-:|:-:|:-:|:-:|:-:|:-:|\n|Max-Pool|74.86|87.87|79.01|16.08|37.87|21.88|\n|Transformer-1L+Max-Pool|**88.64**|75.56|79.74|**8.43**|**25.36**|**14.03**|\n|Transformer-2L+Max-Pool|87.62|**78.30**|79.80|15.67|34.85|21.48|\n|Transfomrer-3L+Max-Pool|87.39|78.25|**79.81**|15.67|37.72|21.57|\n\nThis experiment is performed using the same setup as in table 2 of the main paper.\n\nWe observe that the occlusion order performance **marginally improves when increasing the number of\ntransformer layers above 1. Worse, it degrades the performance of the model for the depth order\nprediction task.** We believe that this is due to the fact that adding more layers makes the\noptimization procedure harder. It is also better to minimize the number of layers to reduce\ninference time and memory usage. Thus, we opt for a single transformer layer before max-pooling.\n\n> Theoretical advantages of predicting $G$ in one pass.\n\nThe main theoretical advantage of predicting $G$ in one pass is that **we achieve $O(1)$ inference\ncomplexity with respect to the number of instances in the scene**. This claim is empirically \nsupported by our runtime/memory experiments in section F figure 7 of the supplementary materials.\n\nWe propose an interpretation of why **depth order prediction seems to be easier than occlusion\nprediction in the holistic setting in section E.1 (task difficulty) of the supplementary\nmaterials.** We kindly refer the reader to the aforementioned section for details. Note that this\nproperty is unique to the holistic setting and does not emerge in pairwise order prediction\nmethods.\n\n## Limitations\n\n> The paper mentions certain limitations, but it would be beneficial if the method could handle\n> an unlimited number of instances, increasing its practicality and effectiveness in more complex\n> scenarios.\n\nWe hope that the previous sections have clarified that our approach can handle easily more than 10\ninstances. InstaFormer is, in theory, bounded by the number of queries $N$, **but the table we\nprovide in this answer shows that this is not a limitation in practice**. We also **provide failure\ncases of our approach in section G.3 and fig. 9 of the supplementary materials**.\n\nWe are planning on **extending this section in the final version of the paper to include more\nfailure cases such as the \"segmentation failure\"** in which the segmentation backbone creates\nmultiple masks for the same instance, resulting in multiple nodes for the same object in the\norder graph.\n\nWe thank again reviewer NnW7 for their constructive comments. We will integrate them to our\nfinal version."}, {"review_id": "gRoR2J0F7X", "reviewer": "Reviewer_i3cT", "summary": "The paper proposes a novel approach for occlusion and depth ordering of objects in a scene. In particular, while most approaches do this by evaluating pairs of objects in each forward pass, the authors propose a method for \"holistic\" ordering, meaning that a single forward pass is able to construct the whole ordering matrix. The approach is validated on a common dataset, and the authors test it against several baselines, from specialised to naive ones, and perform a number of ablation studies to prove the effectiveness of the approach.", "questions": "Major:\n* More details on the task in Sec. 3.1. For example, on how the matrix G is usually constructed (differences between g_ij and g_ji? How does g_ij look like if object i and object j are completely unrelated? etc.)\n* More training details (loss functions unspecified, mentioned some auxiliary ones in line 252 without specifying which one and for what purpose, etc.), as the paper should allow people to reproduce the approach in its entirety, and it's currently not the case.\n* Quantitative analysis on efficiency of \"holisticity\": report inference time and/or GFLOPs of proposed approach and baselines.\n\nMinor:\n* Some figures/tables are not hooked in the text (Fig.1, Fig. 2b, etc.)\n* Contributions in the introduction unclear (Contrib #2 is unclear and should be reformulated, Contrib #3 is arguably not a contribution at all)\n* Include appendix at the end of the paper?\n* Why are the baselines in Sec. 3.3 discussed in the main section dedicated to the proposed approach?", "limitations": "Limitations are not really addressed, as something is mentioned in the introduction but it's mainly a limitation of the existing dataset to evaluate the approach. A proper evaluation of the limitations of the proposed approach is necessary. A (rather short) sentence on possible future works in, instead, present.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The paper tackles an interesting topic, relevant and relatively under-explored so far. The solutions the authors come up with are interesting and mostly well-motivated, and the experimental evaluation is convincing.\n\nThe main limitation of the paper is that it is unclear. Many parts are hard to understand, and overall the approach is not clear once a reader goes over Sec. 3. Additionally, more details on the task itself could be shared, as it is a relatively uncommon task (for now), and not as easily-recognizable as other CV tasks. Also, the authors repeat several times that their holistic approach is more efficient as it predicts everything in a single forward pass, but its efficiency compared to the other baselines is not tested quantitatively.", "was_revised": true, "final_justification": "First, I want to thank the authors for a thorough response to all reviewers, including me.\nThe authors did a good job in addressing all reviews and convincingly motivated their choices. Additionally, they promise to better specify all parts I marked as unclear.\nMy opinion on the paper (even after a thorough re-read) is positive, as it tackles an interesting topic and proposes a novel approach that can be relevant for the community, and was mainly hindered, as I specified, by the lack of clarity.\nGiven the rebuttal phase and the authors' responses, I'm willing to increase my score from 3 to 4. I strongly recommend the authors to integrate all changes from all discussion in the final version of the paper.", "rebuttal": "# Authors' response to reviewer i3cT\n\nWe deeply thank reviewer i3cT for their detailed and constructive comments. We are grateful towards\nthe reviewer for recognizing the comprehensiveness of our benchmark and the efficiency of our\napproach. We appreciate the reviewer mentioning that while this task is under-addressed and\nconstitutes an interesting topic of research. We address the reviewer's remarks as follows:\n\n## Weaknesses\n\n> Many parts are hard to understand.\n\nWe thank reviewer i3cT for their honest feedback. We would be happy if they could point out precise\nelements that need clarification (phrasing of sentences, the terminology, the notation, etc.) so\nthat we can incorporate it into the final version of the paper.\n\n> Approach is not clear after reading section 3.\n\nSimilarly, we are willing to rephrase, re-write, re-explain or add more details in\nsection 3 upon specific requests. Before we release the final version, we will make sure to read\nand re-write this section to make it as clear as possible.\n\n> More details on the task could be shared as it is relatively uncommon in the CV community.\n\nAlthough our baseline papers:\n\n- Self-supervised scene de-occlusion. Xiaohang Zhan et al.\n- Amodal instance segmentation with kins dataset. Lu Qi et al.\n- Instance-wise occlusion and depth orders in natural scenes. Hyunmin Lee et al.\n\nare all published in CV conferences, we agree that the task has not been yet deeply explored.\n**We are willing to polish our paper and will include more details on the task and the dataset\nformat in the supplementary materials** of our final version (cf. \"questions\" section for more\ndetails).\n\n> The authors mention that their holistic approach is more efficient as it predicts everything in\n> a single forward pass, but its efficiency compared to other baselines is not tested quantitatively.\n\n**We report the inference time and the memory consumption of our proposed proposed InstaFormer\nand compare it against baselines in section F and figure 7 of the supplementary materials**.\nWe choose InstaOrderNet as the baseline for comparison since it constitutes the most recent\nand competitive approach while being representative of all pairwise approaches. We benchmark the\nofficial implementation of InstaOrderNet, but also implement a batched version of it that\nconsists in stacking all the pairs of instances in a single batch and performing inference on them\nall at once. We refer to this as InstaOrderNet (batched). Note that InstaFormer is not\noptimized for speed.\n\nIn figure 7a, **InstaFormer's inference time appears constant**, while\nInstaOrderNet's grows exponentially with respect to the number of predicted instances in the\nscene. InstaFormer's inference time averages below 1 second and becomes advantageous over\nInstaOrderNet's default inference anywhere above 11 instances. The batched version of\nInstaOrderNet consistently outperforms InstaOrderNet and InstaFormer.\n\n**Yet, in practice, it is hard to use the batched version of InstaOrderNet since it requires an\nexponential amount of memory with respect to the number of predicted instances.** As a matter of\nfact, while InstaOrder's default implementation runs in a constant memory size, its batched\ninference quickly becomes prohibitive. Inference for more than 20 instances results in more than\n40Gb of memory usage. On the other hand, **InstaFormer's memory usage grows linearly** with respect\nto the number of instances. It proves more affordable than InstaOrderNet's batched inference for\nmore than 10 instances and **saves almost 30Gb of memory for 20 instances**. We encounter CUDA OOM\nerror for any inference containing more than 20 instances using InstaOrderNet (batched), point\nat which we stop the experiment.\n\n**We believe that these results place InstaFormer in a favorable position considering the runtime\nspeed and memory trade-off**. Further optimization of the inference procedure can decrease\nits runtime even further.\n\n## Questions\n\n### Major\n\n> How is $G$ usually constructed?\n\nFor both occlusion and depth order prediction $G$ represents the adjacency matrix of the order\nrelations between the instances in the scene. The rows and columns each represent an instance\nwhile the entries represent the order relations (occlusion or depth) between them.\n\nGiven $I \\in \\[2, \\dots, N\\]$ instances in the scene, we consider a pair of instances\n$(i, j) \\in I \\times I$ such that $i \\neq j$ and consider their entry in the order matrix\n$G_{i,j}$. We never consider the case where $i = j$, i.e. the diagonal of the matrix, since it\ndoes not make sense for an instance to occlude (resp. be in front/behind) itself. Thus, we always\nmanually fill the matrix diagonal with -1 (this value is arbitrary).\n\nThe occlusion order task predicts binary labels $G_{i,j} \\in \\lbrace0,1\\rbrace$ for all $(i, j)\n\\in I \\times I$, such that $i \\neq j$, symbolizing whether:\n\n1. instance $i$ does not occludes instance $j$ (label 0).\n2. instance $i$ occludes instance $j$ (label 1).\n\nIn the case where $G_{i,j} = G_{j,i} = 1$, we say that there is a bidirectional occlusion relation\nbetween $i$ and $j$.\n\nThe depth order task predicts a ternary label $G_{i,j} \\in \\lbrace0,1,2\\rbrace$ for all $(i, j) \n\\in I \\times I$, such that $i \\neq j$, symbolizing whether:\n\n1. instance $i$ is behind instance $j$ (label 0).\n2. instance $i$ is in front of instance $j$ (label 1).\n3. instance $i$ and $j$ share the same depth region (label 2). This is the case when the two\n   objects span in a large overlapping portion of the 3D space.\n\n**We will complement the paper with this explanation in the final version.**\n\n> What is the difference between $G_{i,j}$ and $G_{j,i}$?\n\n$G_{i,j}$ represents the order relation from $i$ to $j$ while $G_{j,i}$ represents the order\nrelation from $j$ to $i$.\n\nFor occlusion, this is because the occlusion relation is not always symmetric:\nbidirectional occlusion relations are possible, but not always present. Sometimes, only $i$ might\nocclude $j$ and sometimes, only $j$ might occlude $i$.\n\nDepth on the other hand is symmetric. An instance if $i$ appears in front of $j$,\nthen $j$ must necessarily appear behind $i$, unless they share the same depth range, in which case\nthey are both annotated using label 2.\n\n> How does $G_{i,j}$ and $G_{j,i}$ look like if object $i$ and $j$ are completely unrelated?\n\nNaturally speaking, the notion of \"unrelated\" only applies to the occlusion order task. In the\ndepth order task, all instances are \"related\" since they all necessarily appear one behind the\nothers.\n\nFor the occlusion order task, we understand the notion of \"unrelated\" as a case where $i$ does not\nocclude $j$ and vice-versa, thus, their entries in the order matrix would simply be $G_{i,j} =\nG_{j,i} = 0$.\n\n> More training details (loss, etc.).\n\nWe have overlooked the details of our training recipe and we thank reviewer i3cT for offering us\nthe opportunity to clarify it.\n\n- Losses\n  - Occlusion order optimization: we use the **binary cross-entropy (BCE)** loss.\n  - Depth order optimization: we use the **cross-entropy (CE)** loss.\n  - Occlusion + depth order optimization: we use both BCE and CE losses, equally weighted.\n\nWe add auxiliary losses to help convergence of our networks. We detail them below:\n\n- Auxiliary losses\n  - Occlusion order: we add a BCE loss on the occlusion order logits of all transformer\n    decoder layers.\n  - Depth order: we add a CE loss on all the transformer decoder layers.\n  - Occlusion + depth: we add both BCE and CE losses on all the transformer decoder layers.\n\n- Batch size\n  - 16 in all cases.\n\nWe hope that these clarifications are sufficient to help the reviewer understand our training setup\nand full reproduction of our experiments. **We will include all those details in the revised version\nof our paper**. We will also **provide a link to our repository upon acceptance, allowing to\nre-train all InstaFormer networks in a single command.**\n\n> Quantitative analysis on efficiency of \"holisticity\", i.e. inference time and memory of proposed\n> approach and baselines.\n\nWe kindly refer reviewer i3cT to our answer in the \"weaknesses\" section above and to the\nsupplementary section F along with figure 7.\n\n### Minor\n\n> Some figures are not hooked in the text.\n\nWe thank reviewer i3cT for pointing out this issue. We will hook all figures to the text in the\nfinal version of the paper.\n\n> Contributions in the introduction are unclear (\\#2 is unclear, \\#3 is arguably not a\n> contribution).\n\n**We will drop both contribution \\#2 and \\#3 in the final version of the paper**. We will\n**polish the first item of the contributions** to reflect the nature of our submission.\n\n> Include appendix at the end of the paper?\n\nWe agree with reviewer i3cT. **We will move the appendix to the end of the paper in the final\nversion.** We do not see any reason not to merge it with the main paper now that it has been\nfinalized.\n\n> Why are the baselines in Sec. 3.3 discussed in the main section dedicated to the proposed\n> approach?\n\n**We agree with the reviewer after reading the paper again. We propose to move the baseline\ndiscussion to the experiment section (sec. 4) of the paper to keep the flow focused on our\nInstaFormer approach in Sec 3.** We welcome any further suggestions on how to improve the flow\nof the paper.\n\n## Limitations\n\n> Limitations are not really addressed. A proper evaluation is necessary.\n\n**We highlight a failure case of our InstaFormer network in section G.3 and figure 9 of the\nsupplementary materials**. We call this failure case \"instance mix-up\". We kindly refer the\nreviewer to it. **We also observed another failure case which we call \"segmentation failure\"**\nwhere mispredicted masks lead to creating nodes in the graph that belong twice to the same\ninstance, resulting in partially wrong order predictions. **We will include this failure case in\nthe final version of the paper.**"}]

Author rebuttal:

Meta-review: This paper proposes a feed-forward framework for predicting object ordering in single-view images. The central component is a transformer-based model that takes an image as input and predicts two relational matrices, capturing pairwise occlusion and depth ordering among objects. The model is trained end-to-end and evaluated against both heuristic and learning-based baselines.

All reviewers consistently recommend acceptance. They find the task to be well-motivated and of broad interest, and agree that the proposed solution is both effective and practically relevant. After carefully considering the paper, the reviews, and the rebuttal, AC concurs with the reviewers’ assessment: the paper addresses an important problem and demonstrates state-of-the-art performance. I therefore recommend acceptance.

As emphasized by the reviewers and AC, the authors are encouraged to incorporate the clarifications, discussions, and improvements raised during the rebuttal into the final version.
Decision: Accept (poster)