SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Network architecture. Fig. 3 right shows the overall network architecture for segmentation label updates. Given an input pair  $(\mathbf{X},\mathbf{y})$ , we first extract a per-point SE(3)-equivariant local feature  $\mathbf{V}\in \mathbb{R}^{N\times C\times 3}$  within each part masked by  $\mathbf{y}$ .  $\mathbf{V}$  is then passed through a sequence of weighted convolutions with the massage passing defined in Eq. 14. To enable efficient inter-part information exchange, after each message-passing layer, we compute a global invariant feature through a perpoint VN-invariant layer, followed by a global pooling and a concatenation between per-point and global features. After the point convolutions, the per-point equivariant features  $\mathbf{V}_n$  are converted to invariant features  $\mathbf{S}_n$  with inter-part poses canceled. We then apply a global pooling on  $\mathbf{S}_n$  to obtain a global shape code, which is decomposed into  $P$  part codes  $\mathbf{Q}_1,\dots ,\mathbf{Q}_P$  as in [13]. Finally, we compute a point-part assignment score for all  $(\mathbf{S}_n,\mathbf{Q}_p)$  pairs (resulting in an  $N\times P$  tensor) and apply a softmax activation along its  $P$  dimension to obtain the updated segmentation  $\mathbf{y}'$ . For instance segmentation without part permutations (Sec. 3.4), we modify the computation of the part feature  $\mathbf{Q}$  by replacing the global feature decomposition with point feature grouping based on  $\mathbf{y}$ .
For pointcloud networks with set-invariance across all points, directly restricting the upper bound of the network Lipschitz using weight truncations [72, 24, 87] will greatly harm network expressivity as it limits the output range on each point. On the other hand, the space of all possible segmentations  $[0,1]^{N\times P}$  has extremely high dimensionality, making the computation of Lipschitz regularization losses [69] rather inefficient. Therefore, we do not explicitly constrain the Lipschitz constant of our network. Nevertheless, we confine all operations involving  $\mathbf{y}$  to local neighborhoods per point, as shown in Eq. 14. This practical approach generally ensures a small Lipschitz constant for the network in most scenarios. The stability of fixed-point convergence will be studied in Sec. 5.3.
5.1 Articulated-object part segmentation
We begin by evaluating our method on articulated-object part segmentation using four categories of the Shape2Motion dataset [74]: washing machine, oven, eyeglasses, and refrigerator. To demon-
Setting	Unseen states	Unseen states + unseen instances
Category	Washing machine	Oven	Eye-glasses	Refrigerator	Washing machine	Oven	Eye-glasses	Refrigerator
PointNet [57]	46.18 ± 3.84	44.08 ± 8.97	38.96 ± 18.41	38.37 ± 12,32	46.15 ± 3.18	45.29 ± 9.54	39.21 ± 17.14	39.00 ± 12.13
DGCNN [75]	46.78 ± 4.37	44.30 ± 10.84	33.35 ± 20.96	39.70 ± 14.05	46.60 ± 4.03	46.69 ± 12.76	34.96 ± 22.14	40.13 ± 13.81
VNN [15]	47.35 ± 1.93	53.64 ± 13.27	57.08 ± 15.52	52.36 ± 11.39	47.09 ± 1.84	51.01 ± 11.09	62.98 ± 11.09	48.49 ± 11.58
VNN-Inv	46.58 ± 1.61	62.32 ± 14.49	37.24 ± 7.76	60.78 ± 15.57	46.66 ± 1.15	67.35 ± 16.50	38.19 ± 10.26	63.18 ± 14.98
Ours	82.32 ± 15.08	81.91 ± 10.81	77.78 ± 14.45	77.26 ± 7.79	84.99 ± 11.76	82.84 ± 8.13	78.65 ± 10.36	73.93 ± 14.08
Table 1: Shape2Motion results. Numbers are segmentation IoU multiplied by 100.
Figure 4: Banach iterations on Shape2Motion. The network is trained on rest-state objects (left) and tested on novel articulation states.
strate the generalizability of our approach, we train our model on objects in a single rest state, such as ovens with closed doors (as depicted in Fig. 5, left). This training setup aligns with many synthetic datasets featuring static shapes [9, 52]. Subsequently, we assess the performance of our model on articulated states, where global and inter-part pose transformations are applied, replicating real-world scenarios. We evaluate our model on both unseen articulation states of the training instances (Tab. 1, left) and on unseen instances (Tab. 1, right). In both cases, the joint angles are uniformly sampled from the motion range for each category. We compare our network to the most widely adopted object-level part segmentation networks PointNet [57] and DGCNN [75] without equivariance, plus VNN [15] with global SE(3)-equivariance. We also compare to a VNN variant, named VNN-Inv, which first computes per-point local SE(3)-equivariant features and then converts them to invariant features and applies invariant message passing. This is a strategy adopted by [18] for human segmentation. All IoUs are computed for semantic parts. In the case of eyeglasses and refrigerators, which have two parts sharing the same semantic label, we train our network with three motion part inputs but output two semantic part labels. This is feasible because our weighted message passing (Eq. 14) is agnostic to the number of parts.
Fig. 4 demonstrates our iterative inference process on novel states after training on the rest states. Fig. 5 shows our segmentation predictions on the four categories. The inter-part pose transformations can cause significant changes in pointcloud geometry and topology, yet our method can robustly generalize to the unseen articulations with inter-part equivariance.
5.2 Multi-Object Scans
We also test our instance segmentation framework (without part orders) on multi-object scans.
Segmentation transfer on DynLab. DynLab [30] is a collection of scanned laboratory scenes, each with 2-3 rigidly moving solid objects captured under 8 different configurations with object positions randomly changed. We overfit our model to the first configuration of each scene and apply it to the 7 novel configurations, transferring the segmentation from the first scan to the others via inter-object equivariance.
We compare our method to a variety of multi-body segmentation methods, including motion-based co-segmentation methods (DeepPart [85], NPP [25], MultiBodySync [30]), direct segmentation prediction models (PointNet++ [58], MeteorNet [49]), geometry-based point grouping (Ward-linkage [77]), and pre-trained indoor instance segmentation modules (PointGroup [33]). The baseline setups follow [30]. Tab. 2 shows the segmentation IoUs. One thing to note is that motion-based co-segmentation can also be viewed as learning inter-object equivariance via Siamese training, yet they cannot reach the performance of equivariance by construction.
Figure 5: Shape2Motion part segmentation results. Global poses are aligned here for visualization but we do not assume it in our inference.
Method	Seg. IoU
PointNet++ [58]	39.4±7.1
MeteorNet [49]	71.8±9.7
DeepPart [85]	66.3±17.2
NPP [25]	71.6±7.7
Ward-linkage [77]	88.6±5.8
PointGroup [33]	72.4±12.5
MultiBodySync [30]	94.0±3.1
Ours	95.5±4.6
Table 2: DynLab segmentation.
Figure 6: Chair scan segmentation.
Synthetic to real chair scans. We train our model using a synthetic dataset constructed from clean ShapeNet chair models [9] with all instances lined up and facing the same direction (Fig. 6 left). We then test our model on the real chair scans from [41] with diverse scene configurations (Fig. 6 right). The configurations range from easy to hard, including: Z (all chairs standing on the floor), SO(3) (chairs laid down), and Pile (chairs piled into clutters). Remarkably, our model with interobject equivariance demonstrates successful generalization across all these scenarios, even in the most challenging Pile setting with cluttered objects.
Network operators and convergence. We show the importance of our locally confined message passing and SE(3)-features (Eq. 14) to the convergence of test-time iterations. As opposed to extracting part-level equivariant features, one can also apply pose canonicalizations to each part individually and employ non-equivariant learning methods in the canonical space. For SE(3)-transformations with a translation component in  $\mathbb{R}^3$  and a rotation component in SO(3), the former can be canonicalized by subtracting the part centers and the latter by per-part PCA. Thus we compare our full model with two ablated versions with part-level equivariance replaced by canonicalization: PCA, where we canonically organize both the part translations and rotations and simply employ a non-equivariant DGCNN [75] for the weighted message passing; VNN, where we only canonize the part translations but preserve the SO(3)-equivariant structures in the network.
Tab. 3 shows the segmentation IoUs of the full model and the ablated versions on the Shape2Motion oven category. Despite canonicalizations being agnostic to per-part SE(3) transformations, they rely on absolute poses and positions, which breaks the locality of network operations, resulting in a significant increase in the Lipschitz constant of the network. Consequently, when any local equivariant operator is replaced by canonicalization, the network performance experiences a drastic drop.
To further examine the convergence ranges, we test the three models under different  $\mathbf{y}^{(0)}$  initializations by adding random noises  $\xi \sim \mathcal{U}(0,1)$  to  $\mathbf{y}_{\mathrm{gt}}$  according to  $\mathbf{y}^{(0)} = (1 - \alpha)\mathbf{y}_{\mathrm{gt}} + \alpha \xi$ ,  $\alpha \in [0,1]$ . Fig. 7 shows the performance changes of the three models with gradually increased noise levels
Method	Equiv.	Pose cano.	Seg. IoU
PCA	-	rot.+trans.	31.22 ± 7.65
VNN	SO(3)	trans.	27.86±13.07
Ours	SE(3)	-	82.84 ± 8.13
Figure 7: Model performances with initializations of different noise levels.
Table 3: Ablations of per-part equivariant operators replaced by pose canonicalization. Numbers are IoU multiplied by 100.
Lipschitz constraint	Weight trunc. [72]	Reg. loss (rand.)	Reg. loss (rand.)	Reg. loss (adv.) [69]	None
Norm	l∞	l∞	l2	l2	-
IoU (×102)	43.99 ±2.43	31.32 ±21.56	71.29 ±15.97	43.99 ±2.43	80.68 ±10.96
Table 4: Network performances under different Lipschitz constraints.
on  $\mathbf{y}^{(0)}$ . The PCA rotation canonicalization has a theoretically unbounded Lipschitz constant, making the network unable to converge stably even within a small neighborhood of  $\mathbf{y}_{\mathrm{gt}}$ . The SO(3)-equivariant VNN with translation canonicalization perfectly overfits to the ground-truth fixed point but exhibits a rapid decline in performance as the initial  $\mathbf{y}^{(0)}$  deviates from  $\mathbf{y}_{\mathrm{gt}}$ . In contrast, our SE(3)-equivariant network demonstrates stable performance across different noise levels.
Lipschitz constraints. We also provide a study of different Lipschitz constraining methods under different norms in Tab. 4. For weight truncation [72], we use the  $l_{\infty}$ -Lipschitz as it is the loosest constraint and set the per-layer Lipschitz upper bound to be 0.99999. For regularization losses, we set the overall network Lipschitz threshold to 0.99. The adversarial sampling [69] loss only works with  $l_{2}$ -norm. We reduce the total number of points from 2048 to 1024 and the GNN neighbors from 40 to 20 due to the large memory consumption for the loss computations. Based on our observations, not having an explicit Lipschitz constraint and using the SE(3)-equivariant message passing work best for the current situation, yet none of these existing Lipschitz constraining methods are helpful to the network performance. We also plot the  $l_{2}$ -norm regularization losses in the supplementary material, showing that they are zero almost everywhere.
In this work, we propose *Banana*, which provides both theoretical insights and experimental analysis of inter-part equivariance. While equivariance is typically examined from an algebraic perspective, we approach it from an analysis standpoint and obtain a strict formulation of inter-part equivariance at the convergent point of an iteration sequence. Based on our theoretical formulation, we propose a novel framework that co-evolves segmentation labels and per-part SE(3)-equivariance, showing strong generalization under both global and per-part transformations even when these transformations result in subsequent changes to the pointcloud geometry or topology. Experimentally, we show that our model can generalize from rest-state articulated objects to unseen articulations, and from synthetic toy multi-object scenes to real scans with diverse object configurations.
Limitations and future work. While our local segmentation-weighted message passing reduces the Lipschitz constant of the network, practically providing stability during test-time iterations, it is important to note that it is not explicitly bounded or regularized. Lipschitz bounds and regularizations for set-invariant networks without compromising network expressivity would be an interesting and important problem for future study. In addition, we study the full  $\mathrm{SE}(3)^P$  action on multi-body systems, but in many real-world scenarios, different parts or objects may not move independently due to physical constraints. Limiting the  $\mathrm{SE}(3)^P$ -action to its subset of physically plausible motions can potentially increase our feature-embedding conciseness and learning efficiency.
Broader Impacts. Our study focuses on the general 3D geometry and its equivariance properties. Specifically, we investigate everyday appliances in common scenes, for which we do not anticipate any direct negative social impact. However, we acknowledge that our work could potentially be misused for harmful purposes. On the other hand, our research also has potential applications in areas that can benefit society, such as parsing proteins in medical science and building home robots for senior care.
We gratefully acknowledge the following grants: a TRI University 2.0 grant, a Vannevar Bush Faculty Fellowship, and a gift from the Adobe Corporation awarded to Stanford University; and NSF FRR 2220868, NSF IIS-RI 2212433, NSF TRIPODS 1934960, NSF CPS 2038873 awarded to the University of Pennsylvania.