Title: NCDL:  A Framework for Deep Learning on non-Cartesian Lattices
Venue: NeurIPS 2023
Accepted: yes
Keywords: Computer Vision and Pattern Recognition
Abstract: The use of non-Cartesian grids is a niche but important topic in sub-fields of the numerical sciences such as simulation and scientific visualization. However, non-Cartesian approaches are virtually unexplored in machine learning. This is likely due to the difficulties in the representation of data on non-Cartesian domains and the lack of support for standard machine learning operations on non-Cartesian data. This paper proposes a new data structure called the lattice tensor which generalizes traditional tensor spatio-temporal operations to lattice tensors, enabling the use of standard machine learning algorithms on non-Cartesian data. However, data need not reside on a non-Cartesian structure, we use non-Dyadic downsampling schemes to bring Cartesian data into a non-Cartesian space for further processing.   We introduce a software library that implements the lattice tensor container (with some common machine learning operations), and   demonstrate its effectiveness. Our method provides a general framework for machine learning on non-Cartesian domains, addressing the challenges mentioned above and filling a gap in the current literature.

NCDL: A Framework for Deep Learning on non-Cartesian Lattices

Abstract

The use of non-Cartesian grids is a niche but important topic in sub-fields of the numerical sciences, such as simulation and scientific visualization. However, nonCartesian approaches are virtually unexplored in machine learning. This is likely due to the difficulties in the representation of data on non-Cartesian domains and the lack of support for standard machine learning operations on non-Cartesian data. This paper proposes a new data structure called the lattice tensor which generalizes traditional tensor spatio-temporal operations to lattice tensors, enabling the use of standard machine learning algorithms on non-Cartesian data. We introduce a software library that implements this new data structure and demonstrate its effectiveness on various problems. Our method provides a general framework for machine learning on non-Cartesian domains, addressing the challenges mentioned above and filling a gap in the current literature.

1 Introduction

Machine learning heavily relies on tensors to represent multi-dimensional data. However, tensors are inherently Cartesian, and representing data solely on Cartesian grids can be restrictive. Certain data may be more naturally suited to alternative grid structures. For example, raw image data from most imaging sensors is not Cartesian; a Bayer filter represents blue and red data with Cartesian structure, but green data has quincunx (checkerboard) structure. There is also abundant literature showing that hexagonal grids are superior to Cartesian grids when representing isotropically band-limited natural images [4, 34, 32, 38, 3]. Representing data in these domains using traditional Cartesian tensors can be inefficient and lead to inaccuracies depending on the distribution the data belongs to. For the distribution of natural images, which have relatively isotropic behaviour in the Fourier domain, it is well known that using alternative grids yields better image representations [38].

Another reason to use exotic grids is that alternative structures can have significantly different approximation capabilities. Again, turning to the hexagonal lattice as an example, which is well known as the optimal sampling lattice in 2D [4], if one samples a 2D signal with a number of samples, a hexagonal sampling reduces the memory consumption by approximately 14% [32]. This follows from a simple argument in the Fourier domain; sampling a band-limited signal periodizes that function in the Fourier domain. If one's sampling pattern is hexagonal (the optimal circle packing pattern) then less information is lost when the signal is hexagonally periodized. This effect compounds as dimensionality increases; for example the body-centered-cubic (BCC) lattice is 30% more efficient than the Cartesian lattice (for isotropically band-limited functions) 1 .

However, though these grid structures show promise in other areas of numerical sciences, they are relatively unexplored in the context of machine learning. There is no user-friendly, general, and

(a)

(b)

(c)

efficient way to represent data on non-Cartesian grids within the context of machine learning (or in general). To address this gap, we introduce a new data-structure called a lattice tensor that we use to represent data on any non-Cartesian lattice structure. This structure changes the inherent representation of the data, as such it requires many common operations to be rewritten in terms of lattice tensors. For the most part, this is straightforward. For example, element-wise operations are trivial to extend. Convolution is more involved, but we still may leverage existing fast implementations. We generalize many of the traditional tensor spatio-temporal operations to lattice tensors, enabling the use of these operations on non-Cartesian domains.

Ultimately, we present a software library called Non-Cartesian Deep Learning (NCDL) which is an open source, concrete implementation of the lattice tensor container and the associated spatiotemporal operations defined over lattice tensors. NCDL library is implemented on top of PyTorch [31], and is designed to be relatively easily integrated into existing PyTorch code bases.

This work is the first general data processing library for non-Cartesian data (with the added benefit of being differentiable). In short, our contributions are as follows:

We introduce the concept of a lattice tensor, a data-structure for the representation of data on non-Cartesian lattice structures.

We generalize the traditional spatio-temporal operations such as convolution and pooling to lattice tensors.

We provide an open source software library that implements our methodology.

We show a small set of cases in which the non-Cartesian ideology produces a performance benefit in the broader context of machine learning.

The remainder of the paper is organized as follows. In Section 2, we review related work on nonCartesian grids both in and out of the context of machine learning. In Section 3, we formalize the key operations and concepts that we use in NCDL. In Section 4, we validate the performance of our implementation for convolution against a specialized implementation for hexagonal lattices, and explore non-Cartesian auto-encoding networks. Finally, in Section 5, we conclude and discuss future work. Our implementation is available at https://www.ncdl.ai .

2 Related Work

Convolution is a fundamental operation in most machine vision pipelines, and can be traced back to the early 1980s [12]. Convolutional networks gained popularity owing to the fact that convolutions exploit the locality of data while avoiding the curse of dimensionality (in parameter count). The recent surge of success in convolutional neural networks (CNNs) is in large part to hardware advances in the early 2010s. General purpose GPU compute power facilitated new benchmark performance in image classification [25, 1]. Since then, various forms of CNNs have been proposed with various design differences: early efforts, like VGG and ResNet, focused on building deeper convolutional

networks [14, 29] while trying to avoid the vanishing gradient problem; while others attempted to create more computationally/parameter efficient networks [18, 33, 36, 37]. Other works have focused on changing the mechanics of the convolution layer [44, 35, 18]. However, none of these have pushed into the realm of non-Cartesian data.

There has been limited work on the use of non-Cartesian grids in machine learning. The space efficiency of the hexagonal lattice has been attractive, so it has garnered a small amount of attention. NeuroHex was the first to explore hexagonal convolutions in a modern CNN setting; their domain is the game of Hex [43]. The game of Hex is restricted to a parallelpiped shaped region; to support hexagonal convolution, the authors simply transform the problem domain and restrict convolutions to the hexagonal network by zeroing out filter elements. HexaConv explores hexagonal convolution in the context of image classification, and shows a surprising improvement over traditional Cartesian convolution [15]. They resample square Cartesian images on to the hexagonal lattice in an axial coordinate system (i.e. the same parallelpiped as [43], but zero padded). A similar work proposed a nearly identical structure on the isocsahedron-based spherical hexagonal grid. These works all suffer from the same problem: wasted space in their representation. There are currently only two works that do not waste space in this manner. The first is HexCNN [47], which stores data in a column format effectively linearizing the hexagonal domain column by column; convolutions then operate on this domain. The second and most similar to our work is HexagDLy, which splits the convolution into multiple convolutions over different shifts of the input domain [34]. Our work is more general than all of these works; our primary contribution is the lattice tensor, a simple, waste-free container for data on arbitrary lattices. Figure 2 shows alternative representations from other frameworks, where they waste memory, and how NCDL addresses this.

This generalization forces us to think more broadly about operations on lattice tensors. Take down sampling for example; traditionally, down sampling is uniform in each dimension (usually dyadic, i.e. a factor of 2 in each dimension). However, this is a limitation only imposed by the restriction to Cartesian grids. There are other possible decimation strategies when discarding samples on the Cartesian lattice, one may discard samples from a checkerboard (i.e. quincunx) pattern, thereby producing another quincunx lattice. This type of downsampling is present in non-dyadic downsampling multi-channel wavelet filterbanks [24], but has not been explored in the context of machine learning. In fact, higher dimensions exhibit more interesting cascading structures between grids [19]. This cascading structure is something we explicitly support in NCDL.

The benefit of non-Cartesian grids extends to higher dimensions. The optimal packing lattice in 3D is the body-centered cubic (BCC) lattice, and this is relatively well studied in the context of scientific visualization [32, 4]. However, most research has focused on finding high performing interpolants in these spaces [9, 22, 11, 21, 7, 10, 6], or on extending numerical techniques to exploit the inherent benefit of these approximation spaces. In this sense, we also contribute in this area, by providing an interface to implement these ideas (one is not required to use the differentiablity provided by NCDL).

Another related subfield of machine learning is graph learning [42]. Examples of graph learning problems include document classification, where documents are the set of nodes V , and are related to other nodes/documents through the edge set E ; this set may represent academic citiations, for example [28]. Since non-Cartesian lattices change the inherent structure of the underlying representation of the grid, it is natural to draw connections between non-Cartesian methods and graph based approaches to operations such as convolution and pooling [13, 26, 27]. However, upon closer inspection, these connections break down in practice. Since the relationship between nodes in a graph is not necessarily constant, defining a simple discrete relationship between them (i.e. a discrete finite filter) is not easily achievable. Compounding this problem, the explicit spatial relationship between points is ambiguous, and depends on how a graph is embedded in space (which is not often provided as a parameter). As such, one typically turns to spectral methods or their approximations to compute the convolution of a 'filter' with a graph [8, 23]. To this end, graph convolution is not necessarily compatible with the ideology presented by traditional convolutional methods; while compelling, it is difficult to compare deep non-Cartesian methods with graph learning approaches. While the geometry of non-Cartesian structures is different than that of the traditional tensor, it is still a regular structure; and as such, we seek to leverage pre-existing optimized implementations for operations on these domains.

(a) Axial Representation

(b) O

ff

set Representation

(c) Coset Representation

Logical Domain

Logical Domain

Memory Layout

Memory Layout

Logical Domain

Memory Layout

3 Computing on Non-Cartesian Grids

In general, processing data on any non-Cartesian grid is simple, but technically cumbersome. While most algorithms are easy to state on a non-Cartesian discretization, the non-separable nature of these lattices makes it difficult to design efficient structures for storage and computation. We avoid this by simply ignoring the non-Cartesian structure; treating all processing as operations over separate Cartesian components. A similar approach is taken in some recent works in the non-Cartesian world [17, 16]. Figure 1 shows an example of this decomposition. The ideas in this paper are general, and we state them as s -dimensional theory. However, in practice, we implement this in 2D and 3D in NCDL.

Lattices We define a lattice as a subset of the s -dimensional integers Z s . Explicitly, we denote this with a full rank s × s integer matrix L , and define a lattice as a set L = L Z s ⊆ Z s . Practically speaking, we are interested in a bounded subset of these integers. In the rest of this paper we assume this set is bounded within a rectangular region R of s -dimensional Euclidean space; we denote this bounded set as L R . When L = I then all the discussion reduces to the Cartesian lattice, and classical machine learning approaches trivially apply.

The definition of a lattice usually invokes notions of combinatorial problems and theory. While we do require some lattice algorithms to facilitate our implementation, we instead mainly use lattices as data-structures. To each lattice site within R , we assign a real (floating point) value; we also explicitly include a batch and

( a)

P

R

v

P

0

= (0,0)

v

P

1

= (1,1)

v

R

0

= (0,0)

v

R

1

= (

-

1,

-

1)

v

R

1

ι

(

) = 0

ι

(

) = 1

v

P

1

κ

+

ι

v

R

1

v

R

0

κ

+

(1,1) =

ι

(

v

R

1

+

v

R

1

) = 0

κ

+

(0,1) =

ι

(

v

R

0

+

v

R

1

) = 1

κ

+

(0,0) =

ι

(

v

R

0

+

v

R

0

) = 0

( b)

(1,0) =

(

+

) = 1

channel index to facilitate common operations in deep learning. We combine these ideas in the notation

When the channel and batch indices are clear from context, we will write this as a L R [ i, k, n ] , or a [ i, k, n ] when all parameters are apparent from context.

The Coset Decomposition The key observation that facilitates the mechanics of this paper is that any (integer) lattice structure can be written as a collection of Cartesian lattices. That is, there exists an integer C , an integer diagonal matrix D ∈ Z s × s and integer shifts v i , 0 ≤ i < C such that

This is the Cartesian coset decomposition. There are many possible coset decompositions; in general D need not be a diagonal matrix, however, this condition forces the cosets to be Cartesian. This particular representation allows us to store data on a lattice as multiple shifted traditional tensors (a.k.a. multi-dimensional arrays). To ensure that our algorithms maintain a consistent coset representation, we place a restriction on v i and R ; we restrict all v i ∈ D [ -1 , 1] s . This puts a lower bound on the boundaries of the region R , and helps avoid "coset creep" (in earlier iterations of this work, we noticed that coset vectors tend to uncontrollably walk around space if not somehow restricted).

Lattice Tensor The lattice tensor is the simple, yet fundamental backbone of this work. A lattice tensor is a collection of tensors and integer vector shifts. We define this formally, as well as a small number of additional properties in the following:

Definition 3.1 (Lattice Tensor) . A lattice tensor consits of the bounded region R , the collection of vectors v R i ∈ D [ -1 , 1] s , and the associated values on the lattice a b,c, L R [ l, k, n ] . When obvious from context, we will simply write v i . We order the v i lexically, according to their values v mod D (i.e. the vector in their equivalence class in D [0 , 1] s ). We denote the individual cosets of this lattice tensor as ˜ a j [ l, k, m ] , where 0 ≤ j < C is the coset index, and m ∈ ( Z + ) s .

The ordering of a lattice tensor's coset vectors is a small, but important part of the definition of a lattice tensor; it ensures that between two arbitrary rectangular regions P and S the set v P i + D Z s is always equivalent to v S i + D Z s . When working with the coset representation it is important to know which coset lattice points belong to; this is formalized as follows:

Definition 3.2 (Coset Index) . Given some n ∈ L R , the coset index ι ( n ) is the integer such that n ∈ v ι ( n ) + D ( Z + ) s .

Proposition 3.1 ( κ -index) . Given i, j with 0 ≤ i, j < C , for any n ∈ L R , m ∈ L S with ι ( n ) = i and ι ( m ) = j , it must be that both ι ( n -m ) and ι ( n + m ) are constant. We define κ ± ( i, j ) := ι ( n ± m ) .

The intuition for the κ -index is simple, if we subtract or add lattice points on two cosets, the κ -index tells us the resulting coset index of the operation. Figure 3 shows an example of these concepts.

Tensor Operations on Lattice Tensors For the most part, tensor arithmetic on lattice tensors is relatively straightforward. If two lattice tensors share the same v i and R , then arithmeteic can be performed directly on the underlying coset tensors. However, there may be cases in which we should be able to add two lattice tensors, but the differing coset vectors break the correspondence between two lattice tensors' coset. Figure 3 shows a simple case where this happens.

Definition 3.3 (Compatibility) . We say two lattice tensors x b,c, L R and y b ′ ,c ′ , L ′ S are compatible if and only if b = b ′ , c = c ′ , L = L ′ and ∃ k R , S ∈ Z s such that L R = { n + k R , S : n ∈ L ′ S } .

In practice, finding this correspondence is straightforward, we shift the set of coset vectors so that their centroid is the origin, then we check if all shifted coset vectors have a corresponding match. This allows us to define element-wise binary operations as operations over the underlying Cartesian tensors.

Definition 3.4 (Binary Elementwise Operations) . Given a binary operation ⊙ defined over the elements of R , two compatible lattice tensors x b,c, L R and y b,c, L S , we define

Note that this always outputs a new lattice tensor on L R .

A Note About Reflection It is common to flip a tensor about an axis, either for data augmentation or for another common operation, like padding. However, an astute reader may notice that not all lattices have reflective symmetry.

Definition 3.5. A lattice L has reflective symmetry about the plane defined by e d if and only if ∀ n ∈ L it is true that -e d · n ∈ L where e d is the d th unit cardinal vector.

If a lattice tensor has reflective symmetry about an axis, materializing a reflected tensor is straightforward: flip the underlying tensor and correct the coset shift vectors.

Lattice Tensor Padding Padding is a simple but fundamental operation in many data processing schemes. On the Cartesian lattice we simply extend the region of interest, then introduce the appropriate new elements. Padding a lattice tensor on L R is slightly more involved. Fix an axis d , and assume we wish to pad on the right by a unit of 1. In this case, we extend the support of R by the smallest amount on the right so as to introduce new lattice sites to the lattice tensor, we then pad the appropriate underlying tensors on the right to fill this new space. To pad on the left by a unit of 1, we again extend the support of R by the smallest amount on the left so as to introduce new lattice sites to the lattice tensor, any underlying tensors are padded to the left, and their corresponding coset vectors become v i -e d . If this moves a coset vector outside of the valid range D [ -1 , 1] s , then all vectors are uniformly shifted back within this range. We leave the details to our implementation, provided as supplementary material.

Convolution The convolution operation naturally generalizes as an operation over lattices, we note this in the following definition.

Definition 3.6. For a lattice tensor a L R and filter f L P where the region P is strictly positive, we define convolution as

whose output lattice tensor is in L S where L R = L S ⊕L P (here, ⊕ is the Minkowski-sum of two point-sets).

When L is the Cartesian lattice, then this reduces to the classical case. This definition excludes fused operations such as downsampling (strided convolutions) and any form of padding. We do this intentionally, as including these operations is quite complex, and supporting such fused operation would require a large additional implementation effort. Even as such, creating a dedicated implementation of this generalized convolution operation is a large challenge on its own [2]. Instead, we leverage pre-existing implementations on Cartesian lattices. We use the following proposition to achieve this.

Proposition 3.2. For a lattice tensor a L R and filter f L P where the region P is strictly positive, the result of the convolution o b,k, L S := ( a b,c, L R ⋆ f c,k, L P ) can be written in terms of its output cosets as

where ∗ is the traditional Cartesian convolution operator, and δ ( i, j ) := D -1 ( v R κ + ( i,j ) -v R i -v S j ) is a constant by which we shift the appropriate coset of a L R .

A practical note about this operation is in order. We are typically limited to rectangular support for supp ( ˜ f j ) in all popular implementations of Cartesian convolution. To address this, we simply take the 'ZeroOut' approch [47], and zero out any inappropriate elements in the square filters' support.

Pooling The pooling operator also naturally extends to lattice tensors, we note this in the following definition.

Definition 3.7. For a lattice tensor a L R and filter geometry L P where the region P is strictly positive, lattice pooling is defined as

whose output lattice tensor is in L S where L R = L S ⊕L P .

We again, re-write this in terms of processing over Cartesian lattices in order to leverage existing frameworks for evaluation

Proposition 3.3. For a lattice tensor a L R and stencil geometry L P where the region P is strictly positive, the result of the lattice pooling operation o b,c, L R [ l, m, n ] = max s ∈L P { a [ l, m, n + s ] } can be written in terms of its output cosets as

where the maximum operation running over the set D -1 ( L P -v j ) ∩ Z s is the traditional max pool operator, restricted to the j th coset of the stencil.

Downsampling Downsampling (i.e. discarding data then reducing resolution) is another basic operation that we generalize. The most general case for downsampling is to systematically remove samples in such a way that the lattice property is preserved. We take the standard path to define down/up sampling about an integer matrix S with 0 < det S ∈ Z . We obtain the sub-sampled lattice by restricting to the new set Q := S L . Loosely, for the new lattice Q we obtain the coset decomposition, then subsample the input lattice tensor according to the new decomposition. This yields a new lattice tensor.

Upsampling Upsampling introduces new datapoints (initialized to zero) at a higher resolution lattice. We denote this with a matrix S -1 L ∈ Z s × s . This employs the same mechanics as downsampling, but rather than removing points, we introduce new points according to the lattice in question. The mechanics of this are similar to downsampling, and we leave the details to the supplementary implementation.

3.1 Derivative Computation

Since gradient computation is an important feature that must be supported by any deep learning framework, it is crucial to discuss gradient computation from the perspective of lattice tensors. For a given loss function h , and the lattice tensor a the gradient ∂h/∂a is simply the lattice tensor consisting of the gradients of the individual cosets, that is, ∂h/∂ ˜ a . Computing the gradient of any element-wise operation is straightforward; we simply operate independently on the coset tensors. However, for the remainder of the operations discussed so far, we require more care. We start with convolution.

Proposition 3.4. For a lattice tensor a L R and filter f L P where the region P is strictly positive, with convolution defined as

whose output lattice tensor is in L S where L R = L S ⊕L P , the derivatives of the loss h with respect to the filter and input lattice tensor are given by

where f mirrors the filter and swaps the channels, i.e. f [ i, j, k ] := f [ j, i, -k ] .

Note that it is common to state the filter gradient as a convolution. We avoid introducing new notation to describe this fact and simply note this here.

Proposition 3.5. For a lattice tensor a L R and filter geometry L P where the region P is strictly positive, lattice pooling

whose output lattice tensor is in L S , has the gradients given by

where

The proofs of these facts are direct consequences of the chain rule, and can be found in the Appendix.

Derivatives of Padding, Downsampling and Upsampling The loss gradient of the downsampling operation is upsampling (taking care to respect the original dimensions of the original lattice tensor) and vice versa. The gradient of the padding operation simply truncates the padded gradient tensor, discarding any gradient signal outside of the original geometry of the tensor from the forward pass. For the most part, we leverage the automatic differentiablity of PyTorch to compute derivatives of these operations [31]. However, more streamlined implementations using the forms above may save memory in future iterations of this work, since less temporary tensors must be saved during the forward pass.

4 Exploring non-Cartesian Networks

In this section, we delineate the experiments we conducted to evaluate NCDL. First, we compare NCDL with the closest existing hexagonal convolution library [34]. Subsequently, we explore the use of non-dyadic down/up sampling within bottlenecked architectures. To the best of our knowledge, this operation has not been explored in machine learning. All experiments are conducted on an AMD Ryzen 9 3900X (3.8GHz) with 128GB of DDR4 RAM operating at 3200 MHz, accompanied by an NVIDIA RTX 3090 with 24GB of RAM.

4.1 Hexagonal Convolution

While NCDL focuses primarily on integer lattices, the hexagonal lattice exhibits the same coset structure as the quincunx lattice. This permits the storage of a hexagonal image on a quincunx lattice -convolution can be implemented via the selection of a suitable filter on the quincux lattice. We assess the efficacy of this method in comparison to HexagDLy [34], which, among all other related works, is conceptually most similar to our approach.

0

2

4

6

8

10

12

14

16

Mean Time (ms)

256

512

1024

2048

Convolution Size v.s. Execution Time

Conv2D

HexagDLy

NCDL

Evaluation Figure 4 illustrates the performance trend as the grid size increases. Here, all grids are 'equivalent' to the Cartesian grid. That is, we choose our hexagonal domain/lattice tensors to consume roughly the same amount of memory as the Cartesian grid. For example, if our Cartesian grid is 256 × 256 , then each coset of our lattice tensor has a dimension of 182 × 182 . We fixed input and output channels at 32 and 64 respectively (simulating an early stage in a convolutional pipeline). We limit our experiment to the case where the convolution kernel is a hexagonal 1-ring/1-neighborhood filter. We set the stride to 1, as NCDL does not support fused strides with convolution. It is important to note that we employ the same filter sizes as HexagDLy, but our convolutions occur over the separate cosets of the lattice, which are smaller than the grids over which HexagDLy operates.

There are a few things to note about this experiment. For smaller grid sizes, NCDL exhibited slower performance compared to HexagDLy. This is likely due to the inherent overhead of managing the coset structure and the overhead of GPU kernel queueing. However, beyond a threshold, NCDL vastly outperforms HexagDLy. This is likely due to the fact that HexagDLy operates over larger grids (even though it employs strided convolutions). This effectively increases the computational load, particularly in processing larger inputs. While the strides in HexagDLy's implementation help in reducing the spatial dimensions of the output, the fundamental operation over larger grids does demand additional computational resources. Interestingly, only in one scenario (2048) did a non-Cartesian implementation outperform the Cartesian 1-ring filter. This finding is particularly significant, considering that the Cartesian convolution employs a larger filter size and is thus expected to involve more computation.

4.2 Non-dyadic Downsampling and the Quincunx Lattice

It is somewhat expected that convolution on hexagonal images yields superior results in aspects such as speed or accuracy. Nonetheless, non-Cartesian lattices may enhance performance without necessitating a change in approximation space through other means. In this second set of experiments, we investigate the application of non-dyadic downsampling. That is, as data flows through the network, it starts as a tensor on a Cartesian lattice; subsequent layers perform a convolution, then a downsample operation according to the following sub-sampling matrix S = [ -1 1; 1 1] . This generates a lattice tensor on the Quincunx lattice. A subsequent convolution (followed by a downsample operation) results in a lattice tensor on the Cartesian lattice. Different levels of convolution alternate between these two grids (analogous to [19]), offering a more gradual reduction in resolution.

3

64

64

128

128

256

256

512

512

1024

1024

1024

1024

1024

1024 1024

1024

512

512

256

256

128

128

64

64

3

3

64

64

128

128

256

256

512

512

1024

1024

1024

1024

1024

1024 1024

1024

512

512

256

256

128

128

64

64

3

Traditional

Quincunx Lattice

Tensor

Geometry

Quincunx Filter

Geometry

CAE

QCAE

L1

↓

0.08804

0.08318

L2

↓

0.01795

0.01639

PSNR

↑

17.53

17.92

SSIM

↑

0.6113

0.6206

LPIPs

↓

0.4808

0.4765

Quincunx Auto-encoder In this experiment, we establish a rudimentary autoencoder structure that progressively downsamples to a latent space before upsampling back to the image space. We first establish a baseline experiment, incorporating two Cartesian convolutions with 3 × 3 filters, followed by a downsample (we maintain two convolutions to preserve parameter parity with the quincunx case). The Quincunx Convolutional Auto-Encoder (QCAE) lattice encompasses one convolution, a downsample onto the Quincunx lattice, one convolution, and a downsample onto the Cartesian lattice. Figure 5 shows the exact structures we design.

We train our models using the CelebA dataset [30]. Employing a straightforward L1 loss, we measure validation L1, L2, PSNR, SSIM [41] and LPIPs metrics [46]. The network is trained with the Adam optimizer, default parameters, and a batch size of 8. We train for 300,000 iterations, as convergence was observed at this point, and take an average of 5 runs. This represents a total of approximately 160 hours of training time for all experiments.

Salient Object Detection with a Quincunx U-Net In this experiment, we extend the rudimentary auto-encoder from the previous experiment by simply adding skip connections. Figure 5 shows the exact structures we design, and the dotted lines show the additional skip connections that we

U-Net

L1

↓

0.07583

0.08268

L2

↓

0.04969

0.05107

BCE

↓

0.19266

0.18706

SSIM

↑

0.2900

0.3074

introduce for this example. We train our models on the DUTS salient object detection dataset [40], and employ a straightforward binary cross-entropy (BCE) loss; we measure validation BCE, L1, L2 and SSIM. The network is trained with the Adam optimizer, with default parameters, and a batch size of 8. We train for 120,000 iterations, as convergence was observed in the validation data at this point.

Discussion In our experiments, introducing a quincunx downsampling shows a noteable difference across our tasks. For the autoencoder experiment, we note a small improvement in performance for no additional cost in parameters (Figure 6); replacing a traditional Cartesian convolution with a downsample operation followed by a Quincunx convolution should also be more computationally efficient, since certain convolutions occur on smaller grids. However, due to implementation overhead, we note that this is currently not true. The U-Net experiment is interesting in that it does not show distinctly better performance in training, but validation performance improves for the BCE metric (Figure 7). It is somewhat surprising that other metrics do not reflect the improved validation entropy; however minimizng BCE does not strictly imply that other pixel-wise metrics will improve, due to the way in which BCE weights pixelwise classification errors.

There are a few potential reasons for the improved quantitative performance of these networks. The downsampling scheme is more gradual in a QCAE. This likely leads to a more gradual reduction of information as it passes through the bottleneck of the network. Important information is less likely to be "missed" by this more gradual reduction. The second, more simple explanation may be that the aliasing introduced by the quincunx downsampling operation is much less strong, thereby providing a stronger signal as data pass through lower resolution layers of the network; aliasing is a well-known enemy of neural networks [45].

5 Future Work and Conclusions

This work has presented a general approach to enable the use of non-Cartesian lattices in machine learning. Doing so has opened a new avenue of research by introducing a new twist on a fundamental concept in machine learning. There are many possible avenues for future work: we plan on exploring this in the context of models at scale, for example in generative approaches like GANs or diffusion models [5, 20]. Additionally, modern components, such as the attention mechanism [39], must be generalized to this new structure. There is also still a good amount of performance left on the table. Decomposing convolution across the different cosets separates the convolution operation into disjoint operations that could, for example, take place on completely separate hardware. These separate operations could be scheduled across different memory chips, for example, to avoid bank conflicts or other complications that arise in GPU memory controllers.

References

Md Zahangir Alom, Tarek M Taha, Christopher Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian C Van Esesn, Abdul A S Awwal, and Vijayan K Asari. The history began from alexnet: A comprehensive survey on deep learning approaches. arXiv preprint arXiv:1803.01164 , 2018.

Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:1410.0759 , 2014.

ACohen and J-M Schlenker. Compactly supported bidimensional wavelet bases with hexagonal symmetry. Constructive approximation , 9(2-3):209-236, 1993.

John Horton Conway and Neil James Alexander Sloane. Sphere packings, lattices and groups , volume 290. Springer Science & Business Media, 2013.

Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Balazs Csebfalvi. Cosine-weighted b-spline interpolation: A fast and high-quality reconstruction scheme for the body-centered cubic lattice. IEEE transactions on visualization and computer graphics , 19(9):1455-1466, 2013.

Balázs Csébfalvi and Gergely Rácz. Retailoring box splines to lattices for highly isotropic volume representations. Computer Graphics Forum , 35(3):411-420, 2016.

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems , 29, 2016.

A. Entezari, D. Van De Ville, and T. Moller. Practical box splines for reconstruction on the body centered cubic lattice. Visualization and Computer Graphics, IEEE Transactions on , 14(2):313-328, March 2008.

Alireza Entezari, Ramsay Dyer, and Torsten Möller. Linear and cubic box splines for the body centered cubic lattice. In Proceedings of the conference on Visualization'04 , pages 11-18. IEEE Computer Society, 2004.

Alireza Entezari and Torsten Möller. Extensions of the Zwart-Powell box spline for volumetric data reconstruction on the cartesian lattice. IEEE Transactions on Visualization and Computer Graphics , 12(5), 2006.

Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological cybernetics , 36(4):193-202, 1980.

Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning , pages 1263-1272. PMLR, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Emiel Hoogeboom, Jorn WT Peters, Taco S Cohen, and Max Welling. Hexaconv. arXiv preprint arXiv:1803.02108 , 2018.

Joshua Horacsek and Usman Alim. A closed pp form of box splines via green's function decomposition. Journal of Approximation Theory , 233:37-57, 2018.

Joshua Horacsek and Usman Alim. Fastspline: Automatic generation of interpolants for lattice samplings. ACM Trans. Math. Softw. , 49(2), jun 2023.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Jiro Inoue and A James Stewart. Multiresolution sphere packing tree: a hierarchical multiresolution 3d data structure. In Proceedings of the 2008 ACM symposium on Solid and physical modeling , pages 367-373, 2008.

Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10124-10134, 2023.

Minho Kim. GPU isosurface raycasting of FCC datasets. Graphical Models , 75(2):90 - 101, 2013.

Minho Kim. Quartic box-spline reconstruction on the BCC lattice. IEEE transactions on visualization and computer graphics , 19(2):319-330, 2013.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Jelena Kovacevic and Martin Vetterli. Nonseparable multidimensional perfect reconstruction filter banks and wavelet bases for r/sup n. IEEE Transactions on Information Theory , 38(2):533555, 1992.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 25. Curran Associates, Inc., 2012.

Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209 , 2022.

Levi Lingsch, Mike Michelis, Sirani M Perera, Robert K Katzschmann, and Siddartha Mishra. Vandermonde neural operators. arXiv preprint arXiv:2305.19663 , 2023.

Jiaying Liu, Jing Ren, Wenqing Zheng, Lianhua Chi, Ivan Lee, and Feng Xia. Web of scholars: A scholar knowledge graph. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 2153-2156, 2020.

Shuying Liu and Weihong Deng. Very deep convolutional neural network based image classification using small training sample size. In 2015 3rd IAPR Asian conference on pattern recognition (ACPR) , pages 730-734. IEEE, 2015.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

Daniel P Petersen and David Middleton. Sampling and reconstruction of wave-number-limited functions in n-dimensional euclidean spaces. Information and control , 5(4):279-323, 1962.

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4510-4520, 2018.

Constantin Steppa and Tim L. Holch. Hexagdly-processing hexagonally sampled data with cnns in pytorch. SoftwareX , 9:193 - 198, 2019.

Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1-9, 2015.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International conference on machine learning , pages 10096-10106. PMLR, 2021.

Dimitri Van De Ville, Thierry Blu, Michael Unser, Wilfried Philips, Ignace Lemahieu, and Rik Van de Walle. Hex-splines: A novel spline family for hexagonal lattices. IEEE Transactions on Image Processing , 13(6):758-772, 2004.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In CVPR , 2017.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Feng Xia, Ke Sun, Shuo Yu, Abdul Aziz, Liangtian Wan, Shirui Pan, and Huan Liu. Graph learning: A survey. IEEE Transactions on Artificial Intelligence , 2(2):109-127, 2021.

Kenny Young, Gautham Vasan, and Ryan Hayward. Neurohex: A deep q-learning hex agent. In Computer Games: 5th Workshop on Computer Games, CGW 2016, and 5th Workshop on General Intelligence in Game-Playing Agents, GIGA 2016, Held in Conjunction with the 25th International Conference on Artificial Intelligence, IJCAI 2016, New York, USA, July 9-10, 2016, Revised Selected Papers , pages 3-18. Springer, 2017.

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4471-4480, 2019.

Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning , pages 7324-7334. PMLR, 2019.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR , 2018.

Yunxiang Zhao, Qiuhong Ke, Flip Korn, Jianzhong Qi, and Rui Zhang. Hexcnn: a framework for native hexagonal convolutional neural networks. In 2020 IEEE International Conference on Data Mining (ICDM) , pages 1424-1429. IEEE, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper generalizes common machine learning operations from Cartesian lattices to other regular lattices, such as the hexagonal lattice. The authors argue that the Cartesian lattice is a sub-optimal representation for important natural signals, and that operating on their non-Cartesian structure natively leads to more efficient implementations and better results.\n\nThe authors further promise to release a software library for non-Cartesian deep learning, and include an experimental section with implementation details and efficiency arguments, as well as experimental results on various computer vision tasks.\n\nSTRENGTHS:\nThe contribution is very novel in the sense that the field of deep learning on lattices is under-explored. The author's work is likely to have a big impact on this area of machine learning, as providing a complete optimized library with non-Cartesian version of common lattice operations with speed-up future research significantly.\n\nWEAKNESSES:\nThe paper is short, with the authors being allowed one more page of content. They could improve the discussion by expanding, for example, on the computation of the derivatives.\n\nAre the backward passes of all operations naturally handled by PyTorch or did it require manual implementations? What about numerical stability?\n\nQUESTIONS:\nThe area of non-Cartesian deep learning could reasonably be considered a sub-field of geometric deep learning, which also includes deep learning on graphs, and models that leverage group symmetry. The authors should consider comparing lattice neural network approaches with GNNs (similar to experiments using sub-pixel graphs), as the graph models should be able to handle the non-Cartesian grid (albeit differently and with less inductive bias).\n\nLIMITATIONS:\nSome aspects are lacking, as explained above:\n- Derivatives and numerical stability\n- Comparison with other non-Cartesian models such as GNNs and group equivariant neural networks", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
This works introduces a framework as well as software for computing convolutions on non-Cartesian lattices. The method is compared to existing software for hexagonal lattices as well as on image data.

STRENGTHS:
The method is put in a strong theoretical framework that also explores the very important up and down sampling operations on non-uniform grids. The authors also make available open-source software that is more general and whose performance seems much better than anything available.

WEAKNESSES:
In the context of scientific machine learning, there have been similar ideas explored on how to make architectures which work on arbitrary grids, for example, https://arxiv.org/abs/2207.05209, https://arxiv.org/abs/2305.19663, and I am sure there are others. Furthermore, graph neural networks can also perform convolutions or arbitrary grids, for example, https://arxiv.org/abs/1704.01212, among many, many others. It would be good to mention some of these and even include some comparisons.

QUESTIONS:
The motivation for images on hexagonal grids are is a bit unclear to me. I always think of images as living on Cartesian grids, so it is somewhat strange to consider them on hexagonal grids instead. Why is this useful? Furthermore, where do you see most of the applications for this? Would be great to include possible large scale applications that lead a vision for future work.

LIMITATIONS:
The authors have adequately addressed limitations and potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a high-quality software extension for PyTorch that enables seamless computations with non-Cartesian lattices for 1D, 2D and 3D images. The key observation made by the authors of this work is that **non-Cartesian lattices can be decomposed as \"sums\" of Cartesian lattices**: up to some clever refactoring (which makes up the core numerical code of the proposed software package), efficient computations on non-Cartesian lattices can directly leverage the standard (and highly optimized) PyTorch/cuDNN implementations of e.g. convolutional layers. \n\nThis implies that **the proposed software package is both efficient and easy to maintain in the long run**.\n\nSTRENGTHS:\n- The authors tackle an interesting and **very original topic**. Non-Cartesian lattices are indeed fundamental to low-level image processing but essentially absent from the machine learning literature.\n\n- The paper is **extremely well written**, with clear figures, attention paid to details and a satisfying evaluation. I haven't tried to run the code provided in the supplementary materials (too many papers to review at once!), but this is **clearly a high-quality software package** with a clean structure, a full test suite and extensive documentation.\n\n- This package targets successfully one specific and interesting operation in image processing, providing a **useful extension to our common toolbox via a neat and clever software package**. This is more than what most papers (never mind submissions) can provide and, in my opinion, clearly warrants publication at NeurIPS.\n\nWEAKNESSES:\nReviewers can always ask for more (outstanding deep learning experiments, better run times, support for all types of attention layers and niche hardware architectures, etc.)... But realistically, I am very happy with the paper as submitted.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nhe authors claim that the concept of tensors, a fundamental cornerstone in machine learning, assumes data are organized on Cartesian grids. They further suggest that alternative non-Cartesian representations may be more beneficial in certain situations. One case is when the data is inherently non-Cartesian — for example, the raw R/G/B image data from most imaging sensors are follows a quincunx (checkerboard) structure in the green channel. Another case is when non-Cartesian grids show superior performance in certain aspects — for example, the hexagonal lattice is known as the optimal sampling lattice in 2D.\nThe authors hereby propose a framework and a software library that introduces standard machine learning operations on non-Cartesian data. The new data structure is called lattice tensor and the software library is called Non-Cartesian Deep Learning (NCDL).\n\nSTRENGTHS:\n1. In the introduction section, the authors provided sufficient context on why non-Cartesian grid structures may be superior to Cartesian counterparts under certain circumstances.\n2. We shall appreciate the effort in designing the memory-efficient coset representation for operations on the hexagonal lattice (shown in Figure 2).\n\nWEAKNESSES:\nWhile custom tensor definitions and operations for non-Cartesian data representation are theoretically pleasing, the authors have not shown clearly the potential applications and impact. It is not obvious which datasets and/or standard machine learning tasks will directly benefit from the non-Cartesian representation. A table detailing some typical use cases will be utterly helpful. \nThe experiments/comparisons performed are not convincing enough. The two main results shown in the submission are (1) runtime of convolution operation at different grid sizes, and (2) loss curves of a Cartesian vs. Quincunx auto-encoder on CelebA (celebrity faces) dataset. \n\nIn the first experiment, while the authors compare the runtime against the standard Cartesian Conv2D and another non-Cartesian baseline (HexagDLy), they have only investigated the convolution operation but skipped the other operations such as pooling, downsampling, upsampling, gradient computation, and back-propagation. Further, no comment has been made on numerical correctness or precision.\nIn the second experiment, the authors aim to show superior performance of Quincunx auto-encoder for image reconstruction. It is a bit weak as the experiment is only performed on one task over one dataset. It may be helpful to include a few more datasets — they don’t even need to be big one, e.g., STL-10, SVHN, LSUN will be sufficient. Besides, the only metrics shown are L1 and SSIM on the validation set. I would recommend including other metrics such as L2, PSNR, FID, and perceptual distance.\n\nQUESTIONS:\n1. Continuing on Weakness #1, it will be very helpful if the authors can construct a table which outlines several typical use cases, with the following information: (1) domain (vision/language/graph/etc), (2) dataset name and description of data format, (3) machine learning task (classification/segmentation/reconstruction/retrieval/etc), (4) preferable non-Cartesian representation, (5) short explanation on why it’s better than Cartesian. Do you think it is a reasonable idea, or is there a better way to demonstrate the applications and impact?\n\n2. Open-ended question: will it be better to embed the proposed method in hardware?\n\nLIMITATIONS:\nWhile it has been covered in previous sections, the main limitations are:\n-Unclear potential application and impact.\n-Insufficient of experiments and comparisons.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}