SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Owen Howell *1, David Klee2, Ondrej Biza2, Linfeng Zhao2, and Robin Walters2
$^{1}$  Department of Electrical and Computer Engineering, Northeastern University  $^{2}$  Khoury College of Computer Sciences, Northeastern University
Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three-dimensions to make predictions about novel images. However, imposing SO(3)-equivariance on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have a natural action on the two-dimensional plane. Specifically, it is possible that an element of SO(3) will rotate an image out of the plane. We show that an algorithm that learns a three-dimensional representation of the world from two-dimensional images must satisfy certain consistency properties which we formulate as SO(2)-equivariance constraints. We use the induced and restricted representations of SO(2) on SO(3) to construct and classify architectures that satisfy these consistency constraints. We prove our construction realizes all possible architectures that respect these constraints. We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose a new model that generalizes previously considered methods and contains additional trainable parameters. We test our architecture on three pose prediction tasks and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks.
One of the fundamental problems in computer vision is learning representations of 3D objects from 2D images [1-3]. By understanding how image features correspond to a physical object, a model can generalize better to novel views of the object, for instance, when estimating the pose of an object. In general, neural networks that respect the symmetries of a problem are more noise robust and data efficient, while also less prone to over-fitting [4]. Three-dimensional space has a natural symmetry group of three-dimensional rotations and three-dimensional translations, SE(3). While we would like to leverage this symmetry to design improved neural architectures, serious challenges exist in incorporating 3D symmetry when applied to image data. Specifically, a projection of a three-dimensional scene into a two-dimensional plane does not transform equivariantly under all elements of SE(3). This is because there is no a priori model for how two-dimensional images transform under out-of-plane object rotations. Cohen and Welling [5] showed how to design neural networks that are explicitly SO(2)-equivariant and accept images as inputs. However, SO(2)-equivariant methods ignore the fact that the group SO(3) acts on the space of pose configurations.
Equivariant neural networks are much more constrained than general multi-layer perceptrons. The requirement of equivariance to a group  $G$  places strict restrictions on the allowed linear maps and the
*howell.o@northeastern.edu
allowed non-linear functions in each network layer [5, 6]. Because of this, the possible structures of  $G$ -equivariant neural networks can be completely classified based on the representation theory of the group  $G$  [4, 7, 8]. For compact groups, it is possible to characterize the structure of all possible kernels of  $G$ -equivariant networks [8].
Figure 1: A map  $\Phi : \mathcal{F} \to \mathcal{F}^{\uparrow}$  from signals on  $\mathbb{R}^2$  to signals on  $S^2$ . Let  $\mathrm{SO}(2)$  be the subgroup that consists of all in-plane rotations (i.e. about the axis defined by the red arrow). The map  $\Phi$  must be equivariant with respect to this  $\mathrm{SO}(2) \subseteq \mathrm{SO}(3)$  subgroup.
We argue that any equivariant machine learning algorithm that builds a three-dimensional model of the world from two-dimensional images must satisfy a natural geometric consistency property. This consistency property can be stated as  $\mathrm{SO}(2)$ -equivariance with respect to only the  $\mathrm{SO}(2)$  subgroup acting along the camera viewing orientation (see Figure 1). We give a complete characterization of maps that satisfy this property. This derivation uses the so-called restricted representation of  $\mathrm{SO}(2)$  since the group action is restricted from the full  $\mathrm{SO}(3)$ . Using the Frobenius Reciprocity theorem, we show that this geometric constraint can also be derived using induced representations, a type of representation of  $\mathrm{SO}(3)$  constructed from representations of  $\mathrm{SO}(2)$ . The classification theorems derived in [5,8,4] assume that both the input and output layers are  $G$ -equivariant. The construction presented in [4] is different; we map  $H$ -equivariant functions to  $G$ -equivariant functions. Our arguments using induced and
restricted representations give a natural generalization of equivariant maps between different groups. We derive analogies of the theorems presented in [9, 8] for the induced and restricted representations.
1.1 Importance and Contribution
In this work, we will show how induced and restricted representations can be used to construct neural architectures that accept image data and leverage SO(3)-equivariant methods to avoid learning nuisance transformations in three-dimensional space.
We show that our construction satisfies two desirable theoretical properties, completeness and universality. Let  $H \subseteq G$ . We focus on the case  $G = SO(3)$  and let  $H = SO(2)$  but we give a theoretical analysis for general groups. Specifically, the induced representation construction is complete in that all group-valued functions on  $G$  can be induced from a set of group-valued functions on  $H$ . The construction is universal in that all multi-linear maps that map  $H$ -equivariant functions to  $G$ -equivariant functions are specific cases of the induced representation, modulo isomorphism. Furthermore, we show that the architectures proposed in [10, 11] are special cases of our construction for the icosahedral group  $G = A_{5}$  and the construction proposed in [12] is a special case of our construction for the three-dimensional rotation group  $G = \mathrm{SO}(3)$ . Our method achieves state of the art performance for orientation prediction on PASCAL3D+ [13] and SYMSOL [14] datasets.
Contributions:
- We propose a unified theory for learning three-dimensional representations from two-dimensional images. We show that algorithms that learn three-dimensional representations from two-dimensional images must satisfy certain consistency properties, which are equivalent to SO(2)-steerability constraints.
- We introduce a fully differentiable layer called an induction/restiction layer that maps signals on the plane into signals on the sphere. We show that the induction/restiction layer satisfies a natural consistency constraint and prove both a completeness and universal property for our construction.
- Our method achieves SOTA performance for orientation prediction on PASCAL3D+ and SYMSOL datasets.
**Equivariant Learning** Incorporating task symmetry into the design of neural networks has been effective in domains such as computer vision [15, 16], point cloud processing [17, 18], and robotics [19]. Cohen and Welling [9] introduced steerable kernels which are a trainable layer that can be used to build networks that are equivariant to 2D [5, 20] and 3D transformations [21, 22]. The majority of past works have studied end-to-end equivariant models, where the input can be transformed by the action of the group and all layers are equivariant, in this work, we explore how to 'fuse'  $SO(2)$  and  $SO(3)$  equivariant layers.
There has been growing interest in leveraging 3D symmetry from 2D inputs. [23, 24] learned a 3D transformable latent space from images of a single object. [25] trained a convolutional network to predict pre-trained SO(3) equivariant embeddings, while [11, 10, 12] mapped image features onto elements of the discrete group of SO(3), using structured viewpoints or a hand-coded projections, respectively. In contrast to prior work, we provide a theoretical foundation for learned equivariant mappings from 2D to 3D, which additionally guides us to introduce a more general and effective trainable operation.
Object Pose Estimation Predicting the 3D orientation of objects is an important problem in fields like autonomous driving [26], robotics [27] and cryogenic electron microscopy [28]. Many works [29, 30] have used a regression approach, and others [31-33] have identified ways to mitigate the discontinuities along the SO(3) manifold. More recent works have explored ways to model orientation as a distribution over 3D rotations, which handles object symmetries and captures uncertainty. [34], [35] and [36] predict parameters for Bingham, von Mises and Laplace distributions, respectively. These families of distributions can have limited expressivity, so other work explored using implicit networks [14] or the Fourier basis [12] to model more complex pose distributions. Inspired by [12, 23], we parameterize the latent object pose as a distribution on  $SO(3)$  and then ask what constraints need to be imposed to enforce  $\mathrm{SO}(2) \subseteq \mathrm{SO}(3)$ -equivariance [A.0.4], which is a generalization of SO(2)-equivariance.
We introduce the induced and restricted representations. For a more extensive review of representation theory, see A
Let  $G$  be a group and  $V$  be a vector space over  $\mathbb{C}$ . A representation  $(\rho, V)$  of  $G$  is a map  $\rho : G \to \operatorname{Hom}[V, V]$  such that
$$
\forall g, g ^ {\prime} \in G, \forall v \in V, \rho (g \cdot g ^ {\prime}) v = \rho (g) \cdot \rho (g ^ {\prime}) v
$$
Concisely, a group representation is an embedding of a group into a set of matrices. The matrix embedding must obey the multiplication rule of the group. We now introduce the restricted representation and induced representation.
Restricted Representation Let  $H \subseteq G$  be a subgroup. Let  $(\rho, V)$  be a representation of  $G$ . The restricted representation of  $(\rho, V)$  from  $G$  to  $H$  is denoted as  $\operatorname{Res}_H^G[(\rho, V)]$ . Intuitively,  $\operatorname{Res}_H^G[(\rho, V)]$  can be viewed as  $(\rho, V)$  evaluated on the subgroup  $H$  of  $G$ . Specifically,
$$
\forall h \in H, \forall v \in V, \quad \operatorname {R e s} _ {H} ^ {G} [ \rho ] (h) v = \rho (h) v
$$
For a more in depth discussion of the restricted representation, please see A
Induced Representation The induced representation is a way to construct representations of a larger group  $G$  out of representations of a subgroup  $H \subseteq G$ . Let  $(\rho, V)$  be a representation of  $H$ . The induced representation of  $(\rho, V)$  from  $H$  to  $G$  is denoted as  $\operatorname{Ind}_H^G[(\rho, V)]$ . Define the space of functions
$$
\mathcal {F} = \left\{f \mid f: G \rightarrow V, \forall h \in H, f (g h) = \rho \left(h ^ {- 1}\right) f (g) \right\}
$$
Then the induced representation is defined as  $(\pi, \mathcal{F}) = \operatorname{Ind}_H^G[(\rho, V)]$  where the induced action  $\pi$  acts on the function space  $\mathcal{F}$  via
$$
\forall g, g ^ {\prime} \in G, \forall f \in \mathcal {F}, (\pi (g) \cdot f) (g ^ {\prime}) = f (g ^ {- 1} g ^ {\prime})
$$
Please see A for an in depth discussion of the induced representation. The induced and restricted representations are adjoint functors [37].
Convolutional networks or vision transformers are typically used to extract spatial feature maps from 2D images. For convenience we ignore discretization and treat the feature maps as having continuous inputs  $f: \mathbb{R}^2 \to \mathbb{R}^d$ . To leverage spatial symmetries in 3D, we would like to map our features  $f$  from a plane onto a sphere:  $g: S^2 \to \mathbb{R}^D$ . Klee et al. [12] proposed one such mapping, where the planar feature map is stretched over a hemisphere, but other possible mappings exist.
We formalize the equivariance property that every projection should have through the theory of induced and restricted representations. The constraints that we impose have an intuitive geometric interpretation. We give a complete characterization of all possible linear and equivariant projections  $\Phi$  from planar features to a spherical representation. Our general formulation includes [12] as a special case, and we show that a learnable equivariant projection leads to better predictive models.
4.1 Equivariant 2D to 3D Projection by Induced and Restricted Representations
We first derive the SO(2)-equivariance constraint for the most general linear mapping from images to spherical signals.
Image inputs We first describe  $\mathcal{F}$  the space of image input signals. Let  $V$  and  $V^{\dagger}$  be vector spaces. Let  $\mathcal{F}$  be the vector space of all  $V$ -valued signals defined on the plane
$$
\mathcal {F} = \{f \mid f: \mathbb {R} ^ {2} \to V \}.
$$
Elements of  $\mathcal{F}$  are sometimes called  $\operatorname{SE}(2)$ -steerable feature fields [20]. The group  $\operatorname{SE}(2) = \mathbb{R}^2 \rtimes \operatorname{SO}(2)$  of 2D translations and rotations acts on  $\mathcal{F}$  via representation  $\pi$ . Each  $h \in \operatorname{SE}(2)$  has a unique factorization  $h = h h_c$  where  $h \in \mathbb{R}^2$  is a translation and  $h_c \in \operatorname{SO}(2)$  is a rotation. Let  $(\rho, V)$  be an  $\operatorname{SO}(2)$ -representation describing the transformation of the fibers of the features  $f$ . Then the action  $\pi$  is defined
$$
\forall f \in \mathcal {F}, r \in \mathbb {R} ^ {2}, h \in \operatorname {S E} (2), \quad \pi (h) \cdot f (r) = \rho \left(h _ {c}\right) f \left(h ^ {- 1} r\right)
$$
so that  $(\pi, \mathcal{F}) = \mathrm{Ind}_{\mathrm{SO}(2)}^{\mathrm{SE}(2)}[(\rho, V)]$  and  $(\pi, \mathcal{F})$  gives a representation of the group  $\mathrm{SE}(2)$  [9].
Spherical outputs We would like to map signals in  $\mathcal{F}$  to functions from  $S^2$  into the vector space  $V^{\uparrow}$ . Let  $\mathcal{F}^{\uparrow}$  be the vector space of all such outputs defined as
$$
\mathcal {F} ^ {\uparrow} = \{\textit {f} | f: S ^ {2} \to V ^ {\uparrow} \}
$$
The group  $\mathrm{SO}(3)$  acts on the vector space  $\mathcal{F}^{\uparrow}$  via
$$
\forall f ^ {\uparrow} \in \mathcal {F} ^ {\uparrow}, \hat {n} \in S ^ {2}, g \in \mathrm {S O} (3), \pi^ {\uparrow} (g) \cdot f ^ {\uparrow} (\hat {n}) = \rho^ {\uparrow} (g) f ^ {\uparrow} (g ^ {- 1} \hat {n})
$$
where  $\rho^{\uparrow}(g)$  describes the SO(3) fiber representation.
SO(2)-equivariant image to sphere Let  $H = \mathrm{SO}(2)$  be the SO(2) subgroup of SO(3) that corresponds to in-plane rotations of the image. Our goal is to classify  $H$ -equivariant linear maps  $\Phi : \mathcal{F} \to \mathcal{F}^{\dagger}$ . This is equivalent to the constraint that
$$
\forall h \in H = \operatorname {S O} (2), f \in \mathcal {F}, \Phi (\pi (h) \cdot f) = \pi^ {\uparrow} (h) \cdot \Phi (f) \tag {1}
$$
The constraint enforces equivariance with respect to SO(2) transformations. By definition, the evaluation of  $\pi^{\uparrow}(h)$  at  $h\in \mathrm{SO}(2)$  is the restricted representation  $\pi^{\uparrow}(h) = \operatorname{Res}_{\mathrm{SO}(2)}^{\mathrm{SO}(3)}[\pi^{\uparrow}](h)$ .
Figure 2: Left: Decomposition of the restricted representation  $\mathrm{Res}_{\mathrm{SO}(2)}^{\mathrm{SO}(3)}$  of SO(3)-irreducibles  $(D^{\ell}, W_{\ell}) \in \widehat{\mathrm{SO}(3)}$  into SO(2)-irreducibles  $(\rho_k, V_k) \in \widehat{\mathrm{SO}(2)}$ . Not every SO(2)-representation can be realized as the restriction of a SO(3)-representation. Right: Decomposition of the induced representation  $\mathrm{Ind}_{\mathrm{SO}(2)}^{\mathrm{SO}(3)}$  for SO(2)-irreducibles  $(\rho_k, V_k) \in \widehat{\mathrm{SO}(2)}$  into SO(3)-irreducibles  $(D^{\ell}, W_{\ell}) \in \widehat{\mathrm{SO}(3)}$ . Not every SO(3)-representation can be realized as the induction of an SO(2)-representation.
4.2 Solving the Kernel Constraint
We use tools from [38, 8] to solve for the space of all possible maps satisfying the constraint [1], giving the trainable space for the image to sphere layer.
Our conclusion is that instead of mapping arbitrary SO(2)-input representations to arbitrary SO(2)-output representations, the allowed input and output representations  $(\rho, V)$  and  $(\rho^{\uparrow}, V^{\uparrow})$  must satisfy additional constraints. Specifically, not every representation can be realized as the restriction of an SO(3) to SO(2) representation [2]. Although in this paper we focus on orientation estimation, the equivariant framework in Section C.0.1 is more general. In the Appendix D we formulate and solve analogous equivariance constraints for both 6DoF-pose estimation and monocular volume reconstruction.
Theorem 1. The constraint in Equation can be solved exactly using the results of [38, 8]. The most linear general map  $\Phi : \mathcal{F} \to \mathcal{F}^{\uparrow}$  can be expanded as
$$
[ \Phi (f) ] (\hat {n}) = \int_ {r \in \mathbb {R} ^ {2}} d r \kappa (\hat {n}, r) f (r)
$$
where  $\kappa : \mathbb{R}^2 \times S^2 \to \mathrm{Hom}[V, V^\uparrow]$ . Then, the exact form of  $\kappa$  can be written as
$$
\kappa (\hat {n}, r) = \sum_ {\ell = 0} ^ {\infty} F _ {\ell} (r) ^ {T} Y _ {\ell} (\hat {n}) \tag {2}
$$
where  $Y_{\ell}(\hat{n})$  is the vectorization of the  $\ell$ -type spherical harmonics and each  $F_{\ell}(r)$  is a standard SO(2)-steerable kernel [9 38] that has input SO(2)-representation  $(\rho, V)$  and output SO(2)-representation  $(\rho^{\ell}, V^{\ell}) = (\rho, V) \otimes \mathrm{Res}_{\mathrm{SO(2)}}^{\mathrm{SO(3)}}[(D^{\ell}, V^{\ell})]$ .
The proof of this statement is given in Appendix F. Note that similar to [18, 6] the tensor product structure of the SO(2) and SO(3) irreducible representations determine the allowed input and output representations of the matrix valued harmonic coefficients  $F_{\ell}(r)$ .
4.3 Including Non-Linearities
In Section 4.2, we considered the most general linear maps that satisfied the generalized equivariance constraint. Adding non-linearities should allow for more expressiveness. Understanding nonlinearities between equivariant layers is still an active area of research [39-42].
One way to include non-linearity is to apply standard SO(3) non-linearities after the linear induction layer. After applying the linear mapping described in C we apply an additional spherical non-linearity [43] to the signal on  $S^2$ . This is the method we employ for the results presented in [6,2]. As shown in G it is also possible to include tensor-product based non-linearity analogous to the results of [18, 6].
5 Theory
5.1 Universal Property
In section 4 we showed how the restriction representation arises naturally when constructing  $\mathrm{SO}(3)$ -equivariant architectures for image data. However, there is no a priori choice of the hidden  $\mathrm{SO}(3)$  representation. We show that with this choice, our construction satisfies a universal property and is unique up to isomorphism [44].
A standard result in group theory establishes the following universal property of induced representations, as stated in [37]:
Theorem 2. Let  $H \subseteq G$ . Let  $(\rho, V)$  be any  $H$ -representation. Let  $\operatorname{Ind}_H^G(\rho, V)$  be the induced representation of  $(\rho, V)$  from  $H$  to  $G$ . Then, there exists a unique  $H$ -equivariant linear map  $\Phi_\rho : V \to \operatorname{Ind}_H^GV$  such that for any  $G$ -representation  $(\sigma, W)$  and any  $H$ -equivariant linear map  $\Psi : V \to W$ , there is a unique  $G$ -equivariant map  $\Psi^\uparrow : \operatorname{Ind}_H^GV \to W$  such that the diagram is commutative.
This theorem can be applied to the construction proposed in [1] to prove a universality property, similar to the results of [5] for  $G$ -equivariant neural networks.
Figure 3: Commutative Diagram for Uniqueness Property of Induced Representations.
Let  $(\rho, V)$  be a  $H$ -representation and let  $(\sigma, W)$  be a  $G$ -representation. Let  $\Psi: V \to W$  where  $\Psi$  is an intertwiner of a the  $H$ -representation and the restriction of the  $G$ -representation to an  $H$ -representation so that
$$
\forall h \in H, \quad \Psi \rho (h) = \operatorname {R e s} _ {H} ^ {G} [ \sigma ] (h) \Psi
$$
so that  $\Psi \in \mathrm{Hom}_H[(\rho, V), \mathrm{Res}_H^G(\sigma, W)]$ . The universal property of the induced representation allows us to write any such  $\Psi$  in a canonical form. Specifically, as illustrated in Figure 5.1, we can always uniquely decompose  $\Psi = \Psi^\uparrow \circ \Phi_\rho$  where  $\Psi^\uparrow \in$
$\operatorname{Hom}_G[\operatorname{Ind}_H^G(\rho, V), (\sigma, W)]$  and  $\Psi_\rho : V \to \operatorname{Ind}_H^GV$  is  $(\sigma, W)$  independent.
Figure 4: Factorization Identity for Universal Property of Induced Representations
Factorization Property of  $H \subseteq G$  Neural Networks We use the universal property of induced representations to show that all possible latent  $G$ -equivariant architectures can be written in terms of the induced representation. At each layer of a equivariant neural network, we have a set of functions from a homogeneous space of a group into some vector space [6]. Let  $X_{i}^{H}$  be a set of homogeneous spaces of the group  $H$  and let  $X_{j}^{G}$  be a set of homogeneous spaces of the group  $G$ . Let  $V_{i}^{H}$  and  $W_{j}^{G}$  be a set of vector spaces. Then, consider the function spaces
$$
\mathcal {F} _ {i} ^ {H} = \left\{f \mid f: X _ {i} ^ {H} \to V _ {i} ^ {H} \right\}, \qquad \mathcal {F} _ {j} ^ {G} = \left\{f ^ {\prime} \mid f ^ {\prime}: X _ {j} ^ {G} \to W _ {j} ^ {G} \right\}
$$
The group  $H$  acts on the homogeneous spaces  $X_{i}^{H}$  and the group  $G$  acts on the homogeneous spaces  $X_{j}^{G}$  so that the function spaces  $\mathcal{F}_i^H$  and  $\mathcal{F}_j^G$  form representations of  $H$  and  $G$ , respectively
Suppose we wish to design a downstream  $G$ -equivariant neural network that accepts as signals functions that live in the vector space  $\mathcal{F}_0^H$  and transform in the  $\rho_0$  representation of  $H$ . Thus,
$(\rho_0,\mathcal{F}_0^H)$  is a  $H$  -representation, but not necessarily a  $G$  -representation. At some point, in the architecture, a layer  $\mathcal{F}_i^H$  must be  $H$  equivariant on the left and both  $H$  and  $G$  -equivariant on the right. Let us call the layer that is both  $H$  and  $G$  -equivariant  $\mathcal{F}_1^G$
$$
\begin{array}{r l r l} \ldots \xrightarrow {\Phi_ {i - 1}} (\rho_ {i}, \mathcal {F} _ {i} ^ {H}) \xrightarrow {\Psi} (\sigma_ {1}, \mathcal {F} _ {1} ^ {G}) \xrightarrow {\Psi_ {1}} \ldots & \ldots \xrightarrow {\Phi_ {i - 1}} (\rho_ {i}, \mathcal {F} _ {i} ^ {H}) \xrightarrow {\Phi_ {\rho_ {i}}} \mathrm {I n d} _ {H} ^ {G} [ (\rho_ {i}, \mathcal {F} _ {i} ^ {H}) ] \xrightarrow {\Psi^ {\uparrow}} (\sigma_ {1}, \mathcal {F} _ {1} ^ {G}) \xrightarrow {\Psi_ {1}} \ldots \\ \Biggl \downarrow \rho_ {i} (h) \qquad \Biggl \downarrow \sigma_ {1} (g) & \cong & \Biggl \downarrow \rho_ {i} (h) \qquad \Biggl \downarrow \mathrm {I n d} _ {H} ^ {G} [ \rho_ {i} ] & \Biggl \downarrow \sigma_ {1} (g) \\ \ldots \xrightarrow {} \Phi_ {i - 1} (\rho_ {i}, \mathcal {F} _ {i} ^ {H}) \xrightarrow {} (\sigma_ {1}, \mathcal {F} _ {1} ^ {G}) \xrightarrow {} \Psi_ {1} \ldots & & \ldots \xrightarrow {} \Phi_ {i - 1} (\rho_ {i}, \mathcal {F} _ {i} ^ {H}) \xrightarrow {} \Phi_ {\rho_ {i}} \mathrm {I n d} _ {H} ^ {G} [ (\rho_ {i}, \mathcal {F} _ {i} ^ {H}) ] \xrightarrow [ \Psi^ {\uparrow} ]{} (\sigma_ {1}, \mathcal {F} _ {1} ^ {G}) \xrightarrow [ \Psi_ {1} ]{} \ldots \end{array}
$$
Figure 5: Factorization of Generic Architecture Using Universal Property of Induced Representation. Any network that has input layer  $(\rho_{i},\mathcal{F}_{i}^{H})$  that is  $H$ -equivariant and output layer  $(\sigma_1^G,\mathcal{F}_1^G)$  that is  $G$ -equivariant can be factorized in terms of the induced representation. The map  $\Psi = \Psi^{\uparrow}\circ \Phi_{\sigma_i}$  where  $\Psi^{\uparrow}$  is  $G$ -equivariant and  $\Phi_{\sigma_i}$  is  $H$ -equivariant.