SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Joshua J. Horacsek
University of Calgary
j.horacsek@ncdl.ai
Usman R. Alim
ualim@ucalgary.ca
The use of non-Cartesian grids is a niche but important topic in sub-fields of the numerical sciences, such as simulation and scientific visualization. However, non-Cartesian approaches are virtually unexplored in machine learning. This is likely due to the difficulties in the representation of data on non-Cartesian domains and the lack of support for standard machine learning operations on non-Cartesian data. This paper proposes a new data structure called the lattice tensor which generalizes traditional tensor spatio-temporal operations to lattice tensors, enabling the use of standard machine learning algorithms on non-Cartesian data. We introduce a software library that implements this new data structure and demonstrate its effectiveness on various problems. Our method provides a general framework for machine learning on non-Cartesian domains, addressing the challenges mentioned above and filling a gap in the current literature.
Machine learning heavily relies on tensors to represent multi-dimensional data. However, tensors are inherently Cartesian, and representing data solely on Cartesian grids can be restrictive. Certain data may be more naturally suited to alternative grid structures. For example, raw image data from most imaging sensors is not Cartesian; a Bayer filter represents blue and red data with Cartesian structure, but green data has quincunx (checkerboard) structure. There is also abundant literature showing that hexagonal grids are superior to Cartesian grids when representing isotropically band-limited natural images [4, 34, 32, 38, 3]. Representing data in these domains using traditional Cartesian tensors can be inefficient and lead to inaccuracies depending on the distribution the data belongs to. For the distribution of natural images, which have relatively isotropic behaviour in the Fourier domain, it is well known that using alternative grids yields better image representations [38].
Another reason to use exotic grids is that alternative structures can have significantly different approximation capabilities. Again, turning to the hexagonal lattice as an example, which is well known as the optimal sampling lattice in 2D [4], if one samples a 2D signal with a number of samples, a hexagonal sampling reduces the memory consumption by approximately  $14\%$  [32]. This follows from a simple argument in the Fourier domain; sampling a band-limited signal periodizes that function in the Fourier domain. If one's sampling pattern is hexagonal (the optimal circle packing pattern) then less information is lost when the signal is hexagonally periodicized. This effect compounds as dimensionality increases; for example the body-centered-cubic (BCC) lattice is  $30\%$  more efficient than the Cartesian lattice (for isotropically band-limited functions)<sup>1</sup>.
However, though these grid structures show promise in other areas of numerical sciences, they are relatively unexplored in the context of machine learning. There is no user-friendly, general, and
<sup>1</sup>The face-centered-cubic (FCC) lattice is also attractive, as it minimizes aliasing.
(c)
Figure 1: Three lattices used as examples throughout this paper; a) is the Cartesian lattice, b) is the quincunx lattice and c) is the hexagonal lattice. The cells of each lattice have been shaded according to their distance to the origin. We ignore the cell structure of the lattice, and treat a lattice structure as a simple point structure. The quincunx and hexagonal lattices are formed by interleaving two Cartesian grids; this is shown with the white and black filled dots.
efficient way to represent data on non-Cartesian grids within the context of machine learning (or in general). To address this gap, we introduce a new data-structure called a lattice tensor that we use to represent data on any non-Cartesian lattice structure. This structure changes the inherent representation of the data, as such it requires many common operations to be rewritten in terms of lattice tensors. For the most part, this is straightforward. For example, element-wise operations are trivial to extend. Convolution is more involved, but we still may leverage existing fast implementations. We generalize many of the traditional tensor spatio-temporal operations to lattice tensors, enabling the use of these operations on non-Cartesian domains.
Ultimately, we present a software library called Non-Cartesian Deep Learning (NCDL) which is an open source, concrete implementation of the lattice tensor container and the associated spatiotemporal operations defined over lattice tensors. NCDL library is implemented on top of PyTorch [31], and is designed to be relatively easily integrated into existing PyTorch code bases.
This work is the first general data processing library for non-Cartesian data (with the added benefit of being differentiable). In short, our contributions are as follows:
1. We introduce the concept of a lattice tensor, a data-structure for the representation of data on non-Cartesian lattice structures.
2. We generalize the traditional spatio-temporal operations such as convolution and pooling to lattice tensors.
3. We provide an open source software library that implements our methodology.
4. We show a small set of cases in which the non-Cartesian ideology produces a performance benefit in the broader context of machine learning.
The remainder of the paper is organized as follows. In Section 2, we review related work on non-Cartesian grids both in and out of the context of machine learning. In Section 3, we formalize the key operations and concepts that we use in NCDL. In Section 4, we validate the performance of our implementation for convolution against a specialized implementation for hexagonal lattices, and explore non-Cartesian auto-encoding networks. Finally, in Section 5, we conclude and discuss future work. Our implementation is available at https://www.ncdl.ai.
Convolution is a fundamental operation in most machine vision pipelines, and can be traced back to the early 1980s [12]. Convolutional networks gained popularity owing to the fact that convolutions exploit the locality of data while avoiding the curse of dimensionality (in parameter count). The recent surge of success in convolutional neural networks (CNNs) is in large part to hardware advances in the early 2010s. General purpose GPU compute power facilitated new benchmark performance in image classification [25, 1]. Since then, various forms of CNNs have been proposed with various design differences: early efforts, like VGG and ResNet, focused on building deeper convolutional
networks [14, 29] while trying to avoid the vanishing gradient problem; while others attempted to create more computationally/parameter efficient networks [18, 33, 36, 37]. Other works have focused on changing the mechanics of the convolution layer [44, 35, 18]. However, none of these have pushed into the realm of non-Cartesian data.
There has been limited work on the use of non-Cartesian grids in machine learning. The space efficiency of the hexagonal lattice has been attractive, so it has garnered a small amount of attention. NeuroHex was the first to explore hexagonal convolutions in a modern CNN setting; their domain is the game of Hex [43]. The game of Hex is restricted to a parallelpiped shaped region; to support hexagonal convolution, the authors simply transform the problem domain and restrict convolutions to the hexagonal network by zeroing out filter elements. HexaConv explores hexagonal convolution in the context of image classification, and shows a surprising improvement over traditional Cartesian convolution [15]. They resample square Cartesian images on to the hexagonal lattice in an axial coordinate system (i.e. the same parallelpiped as [43], but zero padded). A similar work proposed a nearly identical structure on the isocshedron-based spherical hexagonal grid. These works all suffer from the same problem: wasted space in their representation. There are currently only two works that do not waste space in this manner. The first is HexCNN [47], which stores data in a column format effectively linearizing the hexagonal domain column by column; convolutions then operate on this domain. The second and most similar to our work is HexagDly, which splits the convolution into multiple convolutions over different shifts of the input domain [34]. Our work is more general than all of these works; our primary contribution is the lattice tensor, a simple, waste-free container for data on arbitrary lattices. Figure 2 shows alternative representations from other frameworks, where they waste memory, and how NCDL addresses this.
This generalization forces us to think more broadly about operations on lattice tensors. Take down sampling for example; traditionally, down sampling is uniform in each dimension (usually dyadic, i.e. a factor of 2 in each dimension). However, this is a limitation only imposed by the restriction to Cartesian grids. There are other possible decimation strategies when discarding samples on the Cartesian lattice, one may discard samples from a checkerboard (i.e. quincunx) pattern, thereby producing another quincunx lattice. This type of downsampling is present in non-dyadic downsampling multi-channel wavelet filterbanks [24], but has not been explored in the context of machine learning. In fact, higher dimensions exhibit more interesting cascading structures between grids [19]. This cascading structure is something we explicitly support in NCDL.
The benefit of non-Cartesian grids extends to higher dimensions. The optimal packing lattice in 3D is the body-centered cubic (BCC) lattice, and this is relatively well studied in the context of scientific visualization [32, 4]. However, most research has focused on finding high performing interpolants in these spaces [9, 22, 11, 21, 7, 10, 6], or on extending numerical techniques to exploit the inherent benefit of these approximation spaces. In this sense, we also contribute in this area, by providing an interface to implement these ideas (one is not required to use the differentiability provided by NCDL).
Another related subfield of machine learning is graph learning [42]. Examples of graph learning problems include document classification, where documents are the set of nodes  $V$ , and are related to other nodes/documents through the edge set  $E$ ; this set may represent academic citations, for example [28]. Since non-Cartesian lattices change the inherent structure of the underlying representation of the grid, it is natural to draw connections between non-Cartesian methods and graph based approaches to operations such as convolution and pooling [13, 26, 27]. However, upon closer inspection, these connections break down in practice. Since the relationship between nodes in a graph is not necessarily constant, defining a simple discrete relationship between them (i.e. a discrete finite filter) is not easily achievable. Compounding this problem, the explicit spatial relationship between points is ambiguous, and depends on how a graph is embedded in space (which is not often provided as a parameter). As such, one typically turns to spectral methods or their approximations to compute the convolution of a "filter" with a graph [8, 23]. To this end, graph convolution is not necessarily compatible with the ideology presented by traditional convolutional methods; while compelling, it is difficult to compare deep non-Cartesian methods with graph learning approaches. While the geometry of non-Cartesian structures is different than that of the traditional tensor, it is still a regular structure; and as such, we seek to leverage pre-existing optimized implementations for operations on these domains.
Figure 2: Three possible underlying representations for the memory of an image stored on a rectangular bounded region of the hexagonal lattice. The "Logical Domain" is the domain over which an operation operates, the shaded cells are the cells in which data is stored, and the white cells are empty (i.e. wasted space). The axial representation (a) treats the hexagonal domain as a transformed Cartesian domain, and requires a large amount of wasted space [34]. The offset representation (b) is more compact, but requires additional logic for the different rows of the image and may also waste space. Our coset representation decomposes the input lattice into a number of Cartesian lattices, is more compact than the offset representation in general, and works with arbitrary lattices.
3 Computing on Non-Cartesian Grids
In general, processing data on any non-Cartesian grid is simple, but technically cumbersome. While most algorithms are easy to state on a non-Cartesian discretization, the non-separable nature of these lattices makes it difficult to design efficient structures for storage and computation. We avoid this by simply ignoring the non-Cartesian structure; treating all processing as operations over separate Cartesian components. A similar approach is taken in some recent works in the non-Cartesian world [17, 16]. Figure 1 shows an example of this decomposition. The ideas in this paper are general, and we state them as  $s$ -dimensional theory. However, in practice, we implement this in 2D and 3D in NCDL.
Lattices We define a lattice as a subset of the  $s$ -dimensional integers  $\mathbb{Z}^s$ . Explicitly, we denote this with a full rank  $s\times s$  integer matrix  $\mathbf{L}$  and define a lattice as a set  $\mathcal{L} = \mathbf{L}\mathbb{Z}^s\subseteq \mathbb{Z}^s$ . Practically speaking, we are interested in a bounded subset of these integers. In the rest of this paper we assume this set is bounded within a rectangular region  $\mathcal{R}$  of  $s$ -dimensional Euclidean space; we denote this bounded set as  $\mathcal{L}_{\mathcal{R}}$ . When  $\mathbf{L} = \mathbf{I}$  then all the discussion reduces to the Cartesian lattice, and classical machine learning approaches trivially apply.
The definition of a lattice usually invokes notions of combinatorial problems and theory. While we do require some lattice algorithms to facilitate our implementation, we instead mainly use lattices as data-structures. To each lattice site within  $\mathcal{R}$ , we assign a real (floating point) value; we also explicitly include a batch and
Figure 3: Two lattice tensors on the quincunx lattice. Each contains two tensors and two shifts (the first shift is always the trivial 0 shift). In general, on other lattices there may be more cosets and shifts. These two lattice tensors are compatible, that is, they share the same overall geometry, but are shifted.
channel index to facilitate common operations in deep learning. We combine these ideas in the notation
$$
a _ {b, c, \mathcal {L} _ {\mathcal {R}}} [ i, k, \mathbf {n} ] \in \mathbb {R}, \forall \mathbf {n} \in \mathcal {L} _ {\mathcal {R}}, 0 \leq i <   b, 0 \leq c <   k. \tag {1}
$$
When the channel and batch indices are clear from context, we will write this as  $a_{\mathcal{L}_{\mathcal{R}}}[i,k,\mathbf{n}]$ , or  $a[i,k,\mathbf{n}]$  when all parameters are apparent from context.
The Coset Decomposition The key observation that facilitates the mechanics of this paper is that any (integer) lattice structure can be written as a collection of Cartesian lattices. That is, there exists an integer  $C$ , an integer diagonal matrix  $\mathbf{D} \in \mathbb{Z}^{s \times s}$  and integer shifts  $\mathbf{v}_i$ ,  $0 \leq i < C$  such that
$$
\mathcal {L} _ {\mathcal {R}} = \left(\bigcup_ {i = 0} ^ {C - 1} \mathbf {v} _ {i} + \mathbf {D} (\mathbb {Z} ^ {+}) ^ {s}\right) \cap \mathcal {R}. \tag {2}
$$
This is the Cartesian coset decomposition. There are many possible coset decompositions; in general  $\mathbf{D}$  need not be a diagonal matrix, however, this condition forces the cosets to be Cartesian. This particular representation allows us to store data on a lattice as multiple shifted traditional tensors (a.k.a. multi-dimensional arrays). To ensure that our algorithms maintain a consistent coset representation, we place a restriction on  $\mathbf{v}_i$  and  $\mathcal{R}$ ; we restrict all  $\mathbf{v}_i\in \mathbf{D}[-1,1]^s$ . This puts a lower bound on the boundaries of the region  $\mathcal{R}$ , and helps avoid "coset creep" (in earlier iterations of this work, we noticed that coset vectors tend to uncontrollably walk around space if not somehow restricted).
Lattice Tensor The lattice tensor is the simple, yet fundamental backbone of this work. A lattice tensor is a collection of tensors and integer vector shifts. We define this formally, as well as a small number of additional properties in the following:
Definition 3.1 (Lattice Tensor). A lattice tensor consists of the bounded region  $\mathcal{R}$ , the collection of vectors  $\mathbf{v}_i^{\mathcal{R}} \in \mathbf{D}[-1,1]^s$ , and the associated values on the lattice  $a_{b,c,\mathcal{L}_{\mathcal{R}}} [l,k,\mathbf{n}]$ . When obvious from context, we will simply write  $\mathbf{v}_i$ . We order the  $\mathbf{v}_i$  lexically, according to their values  $\mathbf{v} \mod \mathbf{D}$  (i.e., the vector in their equivalence class in  $\mathbf{D}[0,1]^s$ ). We denote the individual cosets of this lattice tensor as  $\tilde{a}_j[l,k,\mathbf{m}]$ , where  $0 \leq j < C$  is the coset index, and  $\mathbf{m} \in (\mathbb{Z}^{+})^{s}$ .
The ordering of a lattice tensor's coset vectors is a small, but important part of the definition of a lattice tensor; it ensures that between two arbitrary rectangular regions  $\mathcal{P}$  and  $S$  the set  $\mathbf{v}_i^{\mathcal{P}} + \mathbf{D}\mathbb{Z}^s$  is always equivalent to  $\mathbf{v}_i^S +\mathbf{D}\mathbb{Z}^s$ . When working with the coset representation it is important to know which coset lattice points belong to; this is formalized as follows:
Definition 3.2 (Coset Index). Given some  $\mathbf{n} \in \mathcal{L}_{\mathcal{R}}$ , the coset index  $\iota(\mathbf{n})$  is the integer such that  $\mathbf{n} \in \mathbf{v}_{\iota(\mathbf{n})} + \mathbf{D}(\mathbb{Z}^+)$
Proposition 3.1 ( $\kappa$ -index). Given  $i, j$  with  $0 \leq i, j < C$ , for any  $\mathbf{n} \in \mathcal{L}_{\mathcal{R}}$ ,  $\mathbf{m} \in \mathcal{L}_{\mathcal{S}}$  with  $\iota(\mathbf{n}) = i$  and  $\iota(\mathbf{m}) = j$ , it must be that both  $\iota(\mathbf{n} - \mathbf{m})$  and  $\iota(\mathbf{n} + \mathbf{m})$  are constant. We define  $\kappa_{\pm}(i, j) \coloneqq \iota(\mathbf{n} \pm \mathbf{m})$ .
The intuition for the  $\kappa$ -index is simple, if we subtract or add lattice points on two cosets, the  $\kappa$ -index tells us the resulting coset index of the operation. Figure 3 shows an example of these concepts.
Tensor Operations on Lattice Tensors For the most part, tensor arithmetic on lattice tensors is relatively straightforward. If two lattice tensors share the same  $\mathbf{v}_i$  and  $\mathcal{R}$ , then arithmetie can be performed directly on the underlying coset tensors. However, there may be cases in which we should be able to add two lattice tensors, but the differing coset vectors break the correspondence between two lattice tensors' coset. Figure 3 shows a simple case where this happens.
Definition 3.3 (Compatibility). We say two lattice tensors  $x_{b,c,\mathcal{L}_{\mathcal{R}}}$  and  $y_{b',c',\mathcal{L}_S'}$  are compatible if and only if  $b = b', c = c', \mathcal{L} = \mathcal{L}'$  and  $\exists \mathbf{k}_{\mathcal{R},\mathcal{S}} \in \mathbb{Z}^s$  such that  $\mathcal{L}_{\mathcal{R}} = \{\mathbf{n} + \mathbf{k}_{\mathcal{R},\mathcal{S}} : \mathbf{n} \in \mathcal{L}_S'\}$ .
In practice, finding this correspondence is straightforward, we shift the set of coset vectors so that their centroid is the origin, then we check if all shifted coset vectors have a corresponding match. This allows us to define element-wise binary operations as operations over the underlying Cartesian tensors.
Definition 3.4 (Binary Elementwise Operations). Given a binary operation  $\odot$  defined over the elements of  $\mathcal{R}$ , two compatible lattice tensors  $x_{b,c,\mathcal{L}_{\mathcal{R}}}$  and  $y_{b,c,\mathcal{L}_S}$ , we define
$$
\left(x _ {b, c, \mathcal {L} _ {\mathcal {R}}} \odot y _ {b, c, \mathcal {L} _ {\mathcal {S}}}\right) [ i, j, \mathbf {n} ] := x _ {b, c, \mathcal {L} _ {\mathcal {R}}} [ i, j, \mathbf {n} ] \odot y _ {b, c, \mathcal {L} _ {\mathcal {S}}} [ i, j, \mathbf {n} + \mathbf {k} _ {\mathcal {R}, \mathcal {S}} ]. \tag {3}
$$
Note that this always outputs a new lattice tensor on  $\mathcal{L}_{\mathcal{R}}$
A Note About Reflection It is common to flip a tensor about an axis, either for data augmentation or for another common operation, like padding. However, an astute reader may notice that not all lattices have reflective symmetry.
Definition 3.5. A lattice  $\mathcal{L}$  has reflective symmetry about the plane defined by  $\mathbf{e}_d$  if and only if  $\forall \mathbf{n} \in \mathcal{L}$  it is true that  $-\mathbf{e}_d \cdot \mathbf{n} \in \mathcal{L}$  where  $\mathbf{e}_d$  is the  $d^{th}$  unit cardinal vector.
If a lattice tensor has reflective symmetry about an axis, materializing a reflected tensor is straightforward: flip the underlying tensor and correct the coset shift vectors.
Lattice Tensor Padding Padding is a simple but fundamental operation in many data processing schemes. On the Cartesian lattice we simply extend the region of interest, then introduce the appropriate new elements. Padding a lattice tensor on  $\mathcal{L}_{\mathcal{R}}$  is slightly more involved. Fix an axis  $d$ , and assume we wish to pad on the right by a unit of 1. In this case, we extend the support of  $\mathcal{R}$  by the smallest amount on the right so as to introduce new lattice sites to the lattice tensor, we then pad the appropriate underlying tensors on the right to fill this new space. To pad on the left by a unit of 1, we again extend the support of  $\mathcal{R}$  by the smallest amount on the left so as to introduce new lattice sites to the lattice tensor, any underlying tensors are padded to the left, and their corresponding coset vectors become  $\mathbf{v}_i - \mathbf{e}_d$ . If this moves a coset vector outside of the valid range  $\mathbf{D}[-1,1]^s$ , then all vectors are uniformly shifted back within this range. We leave the details to our implementation, provided as supplementary material.
Convolution The convolution operation naturally generalizes as an operation over lattices, we note this in the following definition.
Definition 3.6. For a lattice tensor  $a_{\mathcal{L}_{\mathcal{R}}}$  and filter  $f_{\mathcal{L}_{\mathcal{P}}}$  where the region  $\mathcal{P}$  is strictly positive, we define convolution as
$$
\left(a _ {b, c, \mathcal {L} _ {\mathcal {R}}} \star f _ {c, k, \mathcal {L} _ {\mathcal {P}}}\right) [ n, i, \mathbf {p} ] := \sum_ {j = 0} ^ {c - 1} \sum_ {s \in \operatorname {s u p p} (f)} a [ n, j, \mathbf {p} + \mathbf {s} ] f [ i, j, \mathbf {s} ] \tag {4}
$$
whose output lattice tensor is in  $\mathcal{L}_S$  where  $\mathcal{L}_R = \mathcal{L}_S \oplus \mathcal{L}_P$  (here,  $\oplus$  is the Minkowski-sum of two point-sets).
When  $\mathcal{L}$  is the Cartesian lattice, then this reduces to the classical case. This definition excludes fused operations such as downsampling (strided convolutions) and any form of padding. We do this intentionally, as including these operations is quite complex, and supporting such fused operation would require a large additional implementation effort. Even as such, creating a dedicated implementation of this generalized convolution operation is a large challenge on its own [2]. Instead, we leverage pre-existing implementations on Cartesian lattices. We use the following proposition to achieve this.
Proposition 3.2. For a lattice tensor  $a_{\mathcal{L}_{\mathcal{R}}}$  and filter  $f_{\mathcal{L}_{\mathcal{P}}}$  where the region  $\mathcal{P}$  is strictly positive, the result of the convolution  $o_{b,k,\mathcal{L}_S} \coloneqq (a_{b,c,\mathcal{L}_{\mathcal{R}}} \star f_{c,k,\mathcal{L}_{\mathcal{P}}})$  can be written in terms of its output cosets as
$$
\tilde {o} _ {i} = \sum_ {j = 0} ^ {C - 1} \tilde {a} _ {\kappa_ {+} (i, j)} [ \cdot + \delta (i, j) ] * \tilde {f} _ {j} \tag {5}
$$