JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

1

Resolution learning in deep convolutional
networks using scale-space theory
Silvia L. Pintea∗ , Nergis Tömen∗ , Stanley F. Goes, Marco Loog, and Jan C. van Gemert

Abstract—Resolution in deep convolutional neural networks(CNNs) is typically bounded by the receptive field size throughfilter sizes, and subsampling layers or strided convolutions on
feature maps. The optimal resolution may vary significantly
depending on the dataset. Modern CNNs hard-code their resolution hyper-parameters in the network architecture which makestuning such hyper-parameters cumbersome. We propose to doaway with hard-coded resolution hyper-parameters and aim tolearn the appropriate resolution from data. We use scale-space
theory to obtain a self-similar parametrization of filters and makeuse of the N-Jet: a truncated Taylor series to approximate afilter by a learned combination of Gaussian derivative filters. The
parameter σ of the Gaussian basis controls both the amount ofdetail the filter encodes and the spatial extent of the filter. Since σ
is a continuous parameter, we can optimize it with respect to theloss. The proposed N-Jet layer achieves comparable performance
when used in state-of-the art architectures, while learning thecorrect resolution in each layer automatically. We evaluate our
N-Jet layer on both classification and segmentation, and we showthat learning σ is especially beneficial when dealing with inputs
at multiple sizes.
Index Terms—Scale-space theory, Gaussian basis approximation, resolution learning in deep networks.

I. I NTRODUCTION
Resolution defines the inner scale at which objects should
be observed in an image [1]. To control the resolution in
a network, one can change the filter sizes or feature map
sizes. Because there is a maximum frequency that can be
encoded in a limited spatial extent, the filter sizes and feature
map sizes define a lower bound on the resolution encodedin the network. CNNs typically use small filters of 3 × 3
px or 5 × 5 px, where the first layers are forced to look at
detailed, local image neighborhoods such as edges, blobs, andcorners. As the network deepens, each subsequent convolution
increases the receptive field size linearly [2], allowing the
network to combine the detailed responses of the previous
layer to obtain textures, and object parts. Going even deeper,
strategically placed memory-efficient subsampling operations
reduce feature maps to half their size which is equivalent toincreasing the receptive field multiplicatively. At the deepest
layers, the receptive field spans a large portion of the image
and objects emerge as combinations of their parts [3]. The
resolution, as controlled by the sizes of the receptive field andfeature maps, is one of the fundamental aspects of CNNs.
Dr. S.L. Pintea, Dr. N. Tömen, Dr. M. Loog and Dr. J.C. van Gemert
are with the Computer Vision Lab, Delft University of Technology, 2628CD
Delft, Netherlands (e-mail: s.l.pintea@tudelft.nl; n.tomen@tudelft.nl). MSc.
S.F. Goes is with Q.E.F Electronic Innovations.
∗ Shared first authorship with equal contributions.

Fig. 1. Illustration of how an N-Jet Gaussian derivative basis parametrizes
the shape and size of the filters. A linear combination of Gaussian derivativebasis filters (left) weighted by α parameters span a Taylor series to locallyapproximate the shape of image filters. The filters are self-similar: the σparameter can change the size of the filters while keeping its spatial structure
intact. Each of the three filters (right) has a different weighted combinationof basis filters, while their σ is varied on the horizontal axis. Optimizing for
α learns filters shape, optimizing for σ learns their size.

In modern CNN architectures [4], [5], the resolution is a
hyper-parameter which has to be manually tuned using expert
knowledge, by changing the filter sizes or the subsamplinglayers. For example, the popular ResNeXt [5] for the ImageNet
dataset starts with a 7 × 7 px filter, followed by 3 × 3 px and
1×1 px convolutions where the feature maps are subsampled 6times. The same network on the CIFAR-10 dataset exclusively
uses 3×3 px convolutions and the feature maps are subsampled
2 times. Hard-coding the resolution hyper-parameters in the
network for different datasets affects the extent of the receptive
field, and the specific choices made can be restrictive.
In this paper we propose the N-JetNet which can replace
CNN network design choices of filter sizes by learning these.
We make use of scale-space theory [6], where the resolution ismodeled by the σ parameter of the Gaussian function family
and its derivatives. Gaussian derivatives allow a truncated
Taylor series, called the N-Jet [7], to model a convolutional
filter [8] as a linear combination of Gaussian derivative filters,
each weighted by an αi . We optimize these α weights insteadof individual weights for each pixel in the filter, as done in astandard CNN. The choice of the basis cannot be avoided. In
standard CNNs the choice is implicit: an N × N pixel-basis,
whose size cannot be optimized, because it has no well-defined
derivative to the error. In contrast, in the N-Jet model the basis
is a linear combination of Gaussian derivatives where the σparameter controls both the resolution and the filter size, andhas a well-defined derivative to the effective filter and therefore
to the error. This formulation allows the network to learn σ andthus the resolution of convolutional networks. We exemplify
our approach in Fig. 1.

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

To avoid confusions, we make the following naming conventions: throughout the paper we refer to ‘resolution’ as theinner scale as defined in [1]; ‘size’ as the outer scale [1]denoting the number of pixels of a filter or a feature map; and‘scale’ as the parameter controlling the resolution, which isthe standard deviation σ parameter of the Gaussian basis [7].
The scale is different from the size of a filter: one can blura filter and change its scale without necessarily changing its
size. However, they are related as increasing the scale of an
object (i.e. blurring) increases its size in the image (i.e. thenumber of pixels it occupies). Here we tie the filter size to thescale parameter by making it a function of σ.
We make the following contributions. (i) We exploit themulti-scale local jet for automatically learning the scale parameter, σ. (ii) We show both for classification and segmentationthat our proposed N-Jet model automatically learns the appropriate input resolution from the data. (iii) We demonstratethat our approach generalizes over network architectures anddatasets without deteriorating accuracy for both classificationand segmentation.
II. R ELATED WORK
Multiples scales and sizes in the network. Size plays animportant role in CNNs. The highly successful inception
architecture [9] uses two filter sizes per layer. Multiple input
sizes can be weighted per layer [10], integrated at the feature
map level [11], processed at the same time [12], [13], [14],[15] or even made to compete with each other [16]. To process
multiple featuremap sizes, spatial pyramids are used [17],[18], [19], [20], alternatively the best input size and network
resolution can be selected over a validation set [21]. Scaleequivarinat CNNs can be obtained by applying each filter atmultiple sizes [22], or by approximating filters with Gaussianbasis combinations [23] where the set of scale parameters is
not learned, but fixed. Unlike these works, we do not explicitly
process our feature maps over a set of predefined fixed sizes.
We learn a single scale parameter per layer from the data.
Downsampling and upsampling can be modeled as a bijective function [24], or made adaptive using reinforcement
learning [25] and contextual information at the object boundaries [26]. The optimal size for processing an input giving
the maximum classification confidence can be selected among
multiple sizes [27], [28], or learned by mimicking the humanvisual focus [29], or minimizing the entropy over multiple
input sizes at inference time [30]. Network architecture search
can also be used for learning the resolution, at the cost ofincreased computations [31]. Alternatively, the scale distribution can be adapted per image using dynamic gates [32], orby using self-attentive memory gates [33]. The atrous [34],[35] or dilated convolutions [36], [37] design fixed versions oflarger receptive fields without subsampling the image. Theseare extended to adaptive dilation factors learned through a
sub-network [38]. Rather than only learning the filter size, welearn both the filter shape and the size jointly, by relying on
scale-space theory.
Architectures accommodating subsampling. A pooling operation groups features together before subsampling. Popular2

forms of grouping are average pooling [39], and max pooling [40]. Average pooling tends to perform worse than max
pooling [41], [42] which is outperformed by their combination [43], [44]. Other forms include pooling based on ranking [45], spatial pyramid pooling [46], spectral pooling [47]and stochastic pooling [48] and stochastic subsampling [49].
The recent BlurPool [50] avoids aliasing effects when sampling,√ while fractional pooling [51] subsamples with a factorof 2 instead of 2 which allows larger feature maps to be
used in more network layers. All these pooling methods use
hard-coded feature map subsampling. Our work differs, as we
do not use fixed subsampling or strided convolution: we learn
the resolution in each layer.
Fixed basis approximations. Resolution in images is aptly
modeled by scale-space theory [52], [1], [53]. This is achievedby convolving the image with filters of increasing scale,removing finer details at higher scales. Convolving with aGaussian filter has the property of not introducing any artifacts [54], [55] and the differential structure of images canbe probed with Gaussian derivative filters [56], [6] which
form the N-Jet [7]: a complete and stable basis to locallyapproximate any realistic image. Scale-spaces model images
at different resolutions by a continuous one-parameter familyof smoothed images, parametrized by the value of σ of theGaussian filter [1]. In this paper we build on scale-space theory
and exploit the differential structure of images to optimize σand thus learn the resolution.
Various mathematical multi-scale image modeling tools
have been used in convolutional networks. The classical workof Simoncelli et al. [57] proposes the steerable pyramid,
defining a set of wavelets for orientation and scale invariance.
Similarly, the seminal Scattering transform [58], [59] and its
extensions [60], [61] are based on carefully designed complex
wavelet basis filters [62] with pre-defined rotations and scales
giving excellent results on uniform datasets such as MNIST
and textures. Using the Scattering transform as initialization
for the first few layers of a CNN has recently [63], [64] beenshown to also lead to good results on more varied datasets.
Filters can also be approximated as a liner combination overa set of learned low-rank filter basis [65]. Recent work
also starts with a filter basis and use a CNN to learn thefilter weights. Examples include a PCA basis [11], circularharmonics [66], Gabors [67], and Gaussian derivatives [8].
In this paper we build on the Gaussian derivative basis [8]because it directly offers the tools of Gaussian scale-space tolearn CNN resolution.
Learning kernel shape. Current methods investigate inherent
properties of CNN filters. Filters that go beyond convolution
include non-linear Volterra kernels [68], a learned image
adaptive bilateral filter [69] and learned image processing
operations [70]. For convolutional CNN filters, Sun et al. [71]proposes an asymmetric kernel shape, which simulates hexagonal lattices leading to improved results. The active convolution by Jeon and Kim [72] and the deformable CNNs by
Dai et al. [73] offer an elegant approach to learn a spatialoffset for each filter coefficient leading to flexible filters andimproved accuracy. [74] learns continuous filters as functions

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

+

=

+

3

+

+

+

Fig. 2. Representing local image structure with a linear combination of Gaussian basis filters. The patch (left), F (x, y, 0; σ), is modeled by up to second
order Gaussian derivatives, using six α-coefficients.

over sub-pixel coordinates, allowing learnable resizing of thefeature maps. The hierarchical auto-zoom net [75], the scale
proposal network [76], and the recurrent scale approximation
network [77] explicitly predict the object sizes and adapt the
input size accordingly. Our work differs from all these methods
because we learn both the filter shape and the size.
Most similar to us, [78], [79] combine free-form filterswith learned Gaussian kernels that can adapt the receptive
field size. The recent work of Lindeberg et al. [80] uses
Gaussian derivatives for scale-invariance, however the scales
are fixed according to a geometric distribution. Dissimilar tothese works we propose to approximate the complete filter
using a combination of Gaussian derivatives, while adapting
the receptive field size.

define 2D Gaussian derivatives by the product of the partial
derivatives on x and on y:∂ i+j G(x, y; σ)∂xi ∂y ji∂ G(x; σ) ∂ j G(y; σ)
=
.
∂xi
∂y j

Gi,j (x, y; σ) =B. Multi-scale local N-Jet for modeling local image structure
A discrete set of Gaussian derivatives up to nth order,{Gi,j (x, y; σ) | 0 ≤ i + j ≤ n}, can be used in a truncatedlocal Taylor expansion to represent the local scale-space near
any given point with increasing accuracy [7]. This allows us
to approximate a filter F (x) around the point a up to order
N as:III. L EARNING NETWORK RESOLUTION
F (x) =
A. Local image differentials at given scale−x2

by G(x; σ) = σ√12π e 2σ2 which is readily extended to 2D asG(x, y; σ) = G(x; σ) G(y; σ). The local structure learned
in deep networks [1] is linked to the image derivatives.
Image pixels are discretely measured, and do not directlyoffer derivatives. The linearity of the convolution operator
allows [81] to take an exact derivative of a slightly smoothedfunction f with a Gaussian kernel G(.; σ) with scale σ:∂(f (x) ∗ G(x; σ))∂G(x; σ)=∗ f (x),
∂x
∂x

(1)

where ∗ denotes a convolution. This allows taking image
derivatives by convolving the image with Gaussian derivatives.
Gaussian derivatives in 1D at order m and scale σ can be
defined recursively using the Hermite polynomials [82]:∂ m G(x, σ)m ∂x m


−1
x
√
√
=
G(x; σ),
Hm
σ 2
σ 2

Gm (x; σ) =N∂i
Xi F (a)∂xi=0Scale-spaces [56], [6], [53] offer a general framework formodeling image structures at various scales. The resolution, or
the inner scale [1] of an image is modeled by a convolution
with a 2D Gaussian. The 1D Gaussian at scale σ is given(2)

where G(x; σ) is the Gaussian function and Hm (x) the mth order Hermite polynomial, recursively defined as Hi (x) =2xHi−1 (x) − 2(i − 1)Hi−2 (x); H0 (x) = 1; H1 (x) = 2x. We

(3)

i!

(x − a)i +R(a)(x − a)N +1 , (4)(N + 1)!

where R is the residual term that corresponds to the approximation error. By absorbing the polynomial coefficients intoa value α, we arrive at a linear combination of Gaussianderivative basis filters which can be used to approximate image
filters, as illustrated in Fig. 2. For filter F (x, y, c) at position(x, y) and color channel c the approximation is:
F (x, y, c; σ) =

i+jX
≤ N
0 ≤ i, 0 ≤ j+ R(x, y, c; σ),αi,j,c∂ i+j
G (x, y; σ)∂xi ∂y j(5)where R is the residual error, ignored here. Optimizing theα parameters allows us to switch from learning pixel weights
as commonly used in CNNs, to learning the weights of theGaussian basis filters. We show some examples in Fig. 3 where
we optimize the α parameters of an order-3 RGB Gaussianderivative basis with σ = 5 to least squares fit an 11 × 11 px
patch. Results show that the fit can approximate well a slightly
blurred version of the original patch. Because σ = 5 we cannot
recover a perfectly sharp faithful copy of the original patch.
C. Learning receptive field size
We have all ingredients to learn the resolution in a convolutional deep neural network (CNN). Resolution is bounded by
the size of the CNN filters. We can now dynamically adapt
the resolution during training.

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 20204

Original:

Approx:Fig. 3. Illustration that a Gaussian basis can approximate local image structure. Top row: The original cropped 11 × 11 patch. Bottom row: The approximation
by a least-squares fit of the α-coefficients using a third order RGB Gaussian basis with σ = 5. The black border pixels are not evaluated in the least-squares
fit. The approximation captures well a slightly blurred version (σ = 5) of the original.

0.04

Order 0Order 1
Order 2
Order 3
Order 4

0.03

0.125

Order 0Order 1
Order 2
Order 3
Order 4

0.100
0.075
0.050

0.02

0.025
0.000

0.01

0.025
0.050

0.00

0.075
40

20

0

20

40

40

(a) Unnormalized

20

0

20

40

(b) Normalized

Fig. 4. Effect of filter normalization. For unnormalized filters, the higher
order filters are dwarfed by the lower order filters. Normalizing each basis
filter of order i by multiplying with σ i , ensures that the magnitude of eachfilter is approximately in the same range.

Scale-invariant basis normalization. The filter responses
of Gaussian derivatives decay with order, as depicted infigure 4.(a). Following [7], we make the Gaussian derivatives
scale-independent by multiply each i-th order partial derivativeby σ i . This brings the magnitude of basis filters in approximately the same range, as illustrated in Fig. 4.(b).
Learning scale and filter size. The network resolution depends on the parameter σ, determining the inner scale of theGaussian derivative basis. The chain-rule for differentiation
allows to express the derivative of the error J with respect∂J∂F
to σ as the product of two terms: ∂J∂σ = ∂F · ∂σ . The firstterm is the derivative of the error with respect to the filter and
it is found by error-backpropagation, as standardly done. The
second term is the derivative of the filter with respect to σ
and can be found by differentiating Eq. (5) with respect to σ.
Similarly, the value of the Gaussian basis mixing coefficients,
αi,j,c can be found by differentiating the filter F with respect
to the coefficients α.
In practice we cannot work with continuous filters. Therefore, we need to clip the filters to a finite size to performthe convolution. The size s of the filter follows the formula:s = 2 d kσ e + 1, where k determines the extent of the local
N-Jet approximation and is experimentally set. By tying thefilter size to the scale parameter, we only need to change σ
and adapt both the scale controlling the network resolution,and the size defining the spatial extent of the filters.
IV. E XPERIMENTS
We perform extensive experiments to test the added value ofour proposed N-Jet model on both classification and segmen-

Fig. 5. Integrating N-Jet into a backbone architecture: Example using
the NiN backbone [83]. We replace all convolutional layers (except 1 × 1
convolutions) in the backbone architectures with our N-Jet layers defined bya linear combination of Gaussian derivatives where the scale of Gaussians, σ
and the coefficients α are learnable parameters. (In blue: the 1 × 1 standard
convolutions, in gray: pooling and dropout, and in orange: our N-Jet layer).

tation tasks. We demonstrate the ability of our model to automatically adapt the filter size to match the scaling of the input
on: a multi-scale MNIST dataset (for classification) and multiscale Fashion MNIST (for segmentation). Additionally, forclassification we perform experiments on 3 datasets (CIFAR10, CIFAR-100 [84] and SVHN [85]) and using 5 architectures:
NiN [83], ALLCNN [86] and Resnet-32, Resnet-110 [87], as
well as the recent state-of-the-art EfficientNet [88]. While forsegmentation we additionally test on Pascal VOC (SBD) [89],[90] and we consider 2 state-of-the-art architectures: UNet [91]and DeepLabv2 [34]. For both classification and segmentation
tasks we show that our model has competitive performancewith state-of-the-art at reduced number of parameters.
To derive our N-Jet models we start from the baseline
backbone and replace all the non 1 × 1 convolutional layerswith our N-Jet convolutional layers where each filter represents
a linear combination of Gaussian derivatives with learnablecoefficients α and scale, σ. Each kernel in the layer has its
own individual set of α variables, while the σ is shared forthe complete layer. Fig. 5 illustrates this on the smaller NiNarchitecture [83].
A. N-Jet for Image Classification
Safely subsampling for image classification. The receptive field size is also altered through subsampling, pooling,

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

5

1.0x MNIST

1.5x MNIST

2.0x MNIST

3.0
2.5

1.0x MNIST

1.5x MNIST

2.0x MNIST

2.0

structured conv, order 4, filters 16

2.0 = 2.82± 0.04

1.5 = 2.04± 0.04

1.5

batch norm, relu
2x2 max pool
stride 2

3x3 max pool
stride 3

4x4 max pool
stride 41.0

1.0 = 1.11± 0.06

0.5

fully-connected, softmax(b) Learned σ on multi-size MNIST.

(a) Network architecture.

Fig. 6. Exp 1.2(A): (a) Toy architecture used for testing whether we can learn the correct data resolution from the inputs. (b) Estimated Gaussian basis scale,σ, on MNIST resized 1×, 1.5× and 2×. The estimated σ follows the resizing of the data.

Varying filter sizes in layers 1,4,7
94

CIFAR-10 accuracy

92

90

88

86

84

82

80

(5,5,3) default(3,3,3)(5,5,5)(7,7,5)

Fig. 7. Exp 1.1(A): The impact on accuracy when varying filter sizes andfeature map size for the NIN baseline on CIFAR-10. Smaller filter sizes are
more affected by the removal of the subsampling. Setting the resolution hyperparameters wrong can have a detrimental effect on accuracy.

or strided convolution. For classification models we remove
all subsampling operations in the network and add a safesubsampling operation. If the resolution is low (i.e., the σvalue is high) then there is no need to keep the feature map atfull size, and it can safely be subsampled, to improve memoryand speed. For a feature map of size s, we subsample thefeature map to a new size s̄, where we half its current size
σ/r
as a function of σ as: s̄ = s 21
, where r is the safesubsampling hyper-parameter. We apply safe-subsampling for
all models except for the very deep networks: Resnet-110 and
EfficientNet, where it is detrimental by reducing the feature
map sizes too much.
Experimental setup. In all our N-Jet models we set theorder of the Taylor series approximation to 3, unless otherwise
specified. When using safe-subsampling we remove all poolinglayers, and set the stride to 1 in all network layers. For all themodels reported, we add a batch normalization layer after the
convolutional layers, for robustness, and use the momentum
SGD with the momentum set to 0.9. We train for the numberof epoch reported in the literature. For the NIN baseline model
we found the best starting learning rate to be 0.5, while
for the ALLCNN baseline 0.25. We use the same starting
learning rates in our N-Jet models. For our N-Jet-NIN model
we use an L2 regularization weight over the Gaussian mixing
coefficients, α, set to 0.01, while for N-Jet-ALLCNN weregularize the α-s with a weight decay of 0.001. When training
our N-Jet-Resnet models we use an L2 regularization weightover the Gaussian mixing coefficients, α, set to 0.0001, and astarting learning rate of 0.1 as indicated in [87]. We evaluate on

relatively small datasets, and therefore we use the lightweightversion of Resnet where the first block has 16 channels and thelast block 64, while for the deeper Resnet-110 models we use
bottleneck blocks with a 4× channel expansion. For both thebaseline EfficientNet and our N-Jet-EfficientNet we train themodels from scratch, and given the small datasets we use thesmallest model B0 [88]. For the EfficientNet baseline we use
a 0.01 learning rate and a batch size of 32 and we rescale theinputs to 224 × 224 px, since otherwise the model performs
poorly, maybe due to the large subsampling. In our N-JetEfficientNet we keep the input images to their original size.
For our deeper models N-Jet-EfficientNet and N-Jet-Resnet110 we use batches of 16 and a learning rate of 0.001. We
release the N-Jet code1 .
Experiment 1(A): Validation
Experiment 1.1(A): Do resolution hyper-parameters reallymatter? We test our assumption that filter sizes and feature
map sizes affect accuracy. For this we use the NIN baseline
trained on CIFAR-10. We vary the filter sizes in the layersof the NIN which are not 1 × 1 convolutions, and we reset
the strides to 1 in all layers, to remove the feature map
subsampling. Fig. 7 shows the impact of changing the filter
sizes and removing the subsampling. The smaller filter sizes,
as in the case when all filter sizes are set to 3, are affected to agreater degree by the removal of the subsampling because they
have a smaller receptive field. Selecting the correct filter sizes
impacts the overall classification accuracy, and an exhaustive
search over all possible filter size combinations is not feasible.
This validates the need for learning filter sizes.
Experiment 1.2(A): Can the image resolution be learned?
To test resolution learning, we create a toy network architecture depicted in Fig. 6(a). We train the toy architecture on
MNIST when resizing the images 1×, 1.5×, and 2×. Fig. 6.(b)shows the learned Gaussian basis scale, σ, per setup. The σvalues learned for the images resized by 1.5 and 2 do notdirectly correspond to these values because the operations ofsampling and resizing are not commutative: we first discretized
the continuous signal into an image and subsequently subsampled it. However, the relative ratio between the learned scales
is close: (2.0/1.5)σ1.5 = 2.81 ± 0.04 ≈ σ2.0 = 2.82 ± 0.04.
1 https://github.com/SilviaLauraPintea/N-JetNet

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

TABLE IE XP 2.1(A): T HE EFFECT ON CIFAR-10 OF VARYING THE FILTER SCALE ,σ. H AVING A SUB - OPTIMAL FILTER SCALE σ CAN DECREASE ACCURACY
UP TO 3%. T HE SAFE - SUBSAMPLING IS AFFECTED SLIGHTLY MORE BY
THE CHOICE OF σ THAN THE BASELINE SAMPLING .

6

TABLE IIIE XP 3.1(A): DATASET GENERALIZATION . C OMPARISON OF OUR
N-Jet-NIN AND BASELINE NIN ON THE CIFAR-100 AND SVHN DATASETS .
F OR THE BASELINE MODEL WE ONLY REPORT THE BEST PERFORMANCEWE OBTAIN RERUNNING THE MODELS , WHILE FOR OUR N-J ET MODELSWE REPORT MEAN AND STANDARD DEVIATIONS OVER 3 RUNS . W E
ACHIEVE COMPARABLE CLASSIFICATION ACCURACY WITH THE BASELINE .

σSampling

σ = 0.5

σ = 1.0

σ = 2.0

Baseline
Safe-subsampling

88.76%
86.29%

90.25%
89.50%

87.39%
87.26%SVHN
CIFAR-10
CIFAR-100

TABLE IIE XP 2.2(A): T HE IMPORTANCE OF THE HYPER - PARAMETER r OF THE
SAFE - SUBSAMPLING ON THE CIFAR-10 ACCURACY. T HE ACCURACYSLIGHTLY INCREASES AS THE HYPER - PARAMETER r OF THE
SAFE - SUBSAMPLING INCREASES .

NIN [83]

N-Jet-NIN (Ours)

98.17%
90.89%
66.14%

97.55% (±0.08)
91.60% (±.08)
68.42% (±0.31)

NIN feature-map sizes

N-Jet-NIN feature-map sizes

Safe-subsampling hyper-parameter
2
x1

12

0
x2

0

x2

0
x2

20

20

9
x2

29

20

9

9
x2

29

2
x3

32

x2
29

8
8x

6

8
8x

6

x1

x1

16

16

2

Experiment 3.1(A): Generalization to other datasets. We
compare our N-Jet-NIN method with the baseline NIN [83].
We test the generalization properties of our method by also
reporting scores on two other datasets: CIFAR-100 and SVHN.
Tab. III shows the classification results of our N-Jet-NIN whencompared with the baseline NIN. We report mean and standard
deviations over 3 runs for our method. We show in Fig. 8 the

6Experiment 3(A): Generalization ability

x1

Experiment 2.2(A): Safe-subsampling. We test the importance of the hyper-parameter r in the safe-subsampling, withrespect to the classification accuracy on CIFAR-10 using a NIN
backbone. For this experiment we learn the filter scale σ andset k = 2. We fix the hyper-parameter r to one of the values
in the set {2.0, 4.0, 6.0}. Tab. II shows the effect on accuracyof different settings of r. We also show the runtime needed
to train the network for different r settings. As the valueof r increases the accuracy also increases, however also thefeature map sizes in the layers of the network increase, which
affect the overall computational time. For our subsequent
experiments we select r = 4.0 as a trade-off between accuracy
and training speed.

16

Experiment 2.1(A): Learning sigma. We test the effect of σon the performance on the CIFAR-10 dataset, using the NIN
backbone. We fix the spatial extent, k, to 2 and vary sigma
in the set {0.5, 1.0, 2.0}. Tab. I shows that a wrong settingof σ can influence the classification accuracy up to 3%. The
safe-subsampling setting is affected more by the choice of σthan the baseline subsampling as it relies on the value of σ
when deciding how much to subsample the input feature maps.
Overall, we note that σ = 1.0 achieves the best performanceon this setting, therefore we use this value when initializing
σ during the learning in our N-Jet models.

2

Experiment 2(A): Model choicesx3

The learned σ values follow the input resizing, thus the correctfilter scales and sizes can be learned from the input.

32

91.63%≈193.48 min.

2

Accuracy
91.59%
Training time ≈129.91 min.

x3

91.60%≈179.43 min.

x3

r = 6.032

r = 4.032

r = 2.0

Fig. 8. Exp 3.1(A): Dataset generalization. The baseline feature map sizes
compared to the learned feature maps sizes by our N-Jet-NIN on CIFAR-10.
We show in orange the subsampling layers. Safe-subsampling dynamicallyfinds the appropriate feature map size.

hard-coded sizes of the baseline feature maps, versus the sizes
learned by our N-Jet-NIN on CIFAR-10. The performance of
N-Jet-NIN is comparable with the baseline performance, while
dynamically learning the appropriate feature map size.
Experiment 3.2(A): Generalization to other models. To
test the generalization of our N-Jet convolutional layer todifferent network architectures, we use the ALLCNN [86],Resnet [87], and the recent EfficientNet [88] backbone networkarchitectures. For N-Jet-ALLCNN we use safe-subsampling at
every layer, while for N-Jet-Resnet-32 only at the layers where
the original network subsamples. Tab. IV shows the classification accuracy of the baseline models tested by us on CIFAR-10,
CIFAR-100, when compared with our N-Jet models. We reportmean and standard deviation over 3 repetitions for our models,
as well as the number of parameters. Using our proposed NJet layers gives similar accuracy to the standard convolutional
layers, while avoiding the need to hard-code the filter sizes.
For an approximation of order 3 in the N-Jet, there is a small
increase in the number of parameters compared to the baseline,
except for the EfficientNet which uses also kernel sizes larger
than 3 × 3 px. When employing larger models – N-Jet-Resnet110 and N-Jet-EfficientNet – a Gaussian basis combination oforder 2 is sufficient to obtain an accuracy comparable to thebaseline models, while reducing the number of parameters. In
Fig. 9 we show the baseline ALLCNN feature map sizes whencompared to the N-Jet-ALLCNN learned feature map sizes on
CIFAR-10. The N-Jet model has similar classification accuracy
when compared to the ALLCNN baseline, while learning at
every layer the befitting feature map size. When applied
at every layer, the safe-subsampling makes the subsampling
continuous and smooth, compared to the baseline.
1 We use torchinfo (https://github.com/tyleryep/torchinfo) to enumerate all
the parameters.

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

TABLE IV
E XP 3.2(A): A RCHITECTURE GENERALIZATION . T HE CLASSIFICATION
ACCURACY ON CIFAR-10, CIFAR-100 WHEN COMPARING THE BASELINEMODELS WITH OUR PROPOSED N-Jet-ALLCNN, N-Jet-Resnet-32, AND TWO
DEEPER MODELS : N-Jet-Resnet-110 AND N-Jet-EfficientNet. F OR THE
BASELINE MODELS WE REPORT THE BEST PERFORMANCE , WHILE FOR
OUR N-J ET MODELS WE REPORT MEAN AND STANDARD DEVIATIONSOVER 3 RUNS . W E ALSO SHOW THE NUMBER OF PARAMETERS FOR EACH
MODEL . N-Jet NETS OBTAIN COMPARABLE ACCURACY TO THE BASELINES
WHILE REDUCING THE NUMBER OF PARAMETERS FOR ORDER 2.

# params
CIFAR-10
CIFAR-100

ALLCNN [86]N-Jet-ALLCNN (Ours)
Order 3
Order 20.97 M
91.87%
67.24%

1.07 M
92.48% (±0.134)
67.62% (±0.863)

Resnet-32 [87]Order 3# params
CIFAR-10
CIFAR-100# params
CIFAR-10
CIFAR-100# params
CIFAR-10
CIFAR-100

0.66 M
89.91% (±0.032)
65.17% (±0.228)

N-Jet-Resnet-32 (Ours)
Order 20.47 M
92.30%
67.89%

0.52 M
92.28% (±0.260)
67.59% (±0.278)

Resnet-110 [87]N-Jet-Resnet-110 (Ours)
Order 3
Order 26.90 M
92.83%
73.53%

7.29 M
93.71% (±0.337)
71.73% (±0.203)

EfficientNet [88]N-Jet-EfficientNet (Ours)
Order 3
Order 23.60 M
92.64%
76.19%

3.51 M
93.51% (±0.110)
75.22% (±0.163)

ALLCNN feature-map sizes

0.31 M
89.49% (±0.304)
65.14% (±0.619)

5.74 M
93.52% (±0.043)
73.66% (±0.295)

3.48 M
93.71% (±0.029)
76.17% (±0.409)

N-Jet-ALLCNN feature-map sizes

5

5

5x

8

3

5x

8x

x1

13

1

7

x1

17

4

9

x2

x2

21

x2

24

29

8

8

8x

6

8x

6

x1

x1

16

6

2
x3

2
x3

2
x3

x1

16

16

32

32

32

Fig. 9. Exp 3.2(A): Architecture generalization. The baseline feature map
sizes when compared to the learned N-Jet feature map sizes. We show
in different colors the layers at which the size changes. For the standard
architecture the subsampling layers are orange. We can find at every layer theappropriate subsampling level.

Experiment 4(A): Comparison to scale-invariant methods.
We evaluate on the normal sized 28 × 28 px MNIST andon MNIST resized by a factor of 4 with a size of 112 ×
112 px. We compare against a standard CNN with varying
filter sizes, and against the Deformable CNN [73], as well
as Atrous (dilated) convolutions [34]. We consider 2 and 4layer toy architectures containing only convolutional layers
followed by ReLU activations. Results in table V show thatthe standard CNN performs well on MNIST, yet results are
sensitive to the filter size for 4 × MNIST. The Deformable
CNN [73] is also affected by the change in image size. Our
intuition is that the Deformable CNN still relies on the initial
3 × 3 convolutions and optimizing the offsets is difficult underlarge size changes in the input. For Atrous CNN the dilation
factor has to be hard-coded, and we use a dilation factor of2, as using 4 would imply including prior knowledge. The
Atrous performance is also affected by the change in input
size. In contrast, our N-Jet model is able to learn the correct

7

TABLE VE XP 4(A): C OMPARISON WITH SCALE - INVARIANT METHODS . W E
EVALUATE ON MNIST AND MNIST RESIZED 4× OUR N-J ET , STANDARD
CONVOLUTIONS OF VARYING FILTER SIZES , AS WELL AS USING ATROUS
CONVOLUTIONS [34] AND DEFORMABLE CONVOLUTIONS [34]. O UR
N-J ET PERFORMS WELL ON MNIST × 4 DESPITE THE INCREASE IN SIZE .

2-layer architecture
4 × MNIST

CNN

MNIST

Standard 3 × 3Standard 5 × 5Standard 9 × 9Standard 11 × 11

97.04% (± 0.22)
98.62% (± 0.08)
98.93% (± 0.08)
98.72% (± 0.20)

86.27% (± 3.44)
88.67% (± 5.97)
95.84% (± 0.76)
95.75% (± 1.50)

Atrous [34]Deformable [73]N-Jet (Ours)

98.47% (± 0.25)
97.54% (± 0.46)
99.05% (± 0.04)

89.87% (± 3.67)
84.09% (± 0.84)
98.11% (± 0.16)

CNN

MNIST

Standard 3 × 3Standard 5 × 5Standard 9 × 9Standard 11 × 11

98.49% (± 0.20)
98.54% (± 0.37)
98.91% (± 0.18)
98.81% (± 0.20)

86.53% (± 7.15)
97.47% (± 0.41)
94.23% (± 4.01)
96.21% (± 1.40)

Atrous [34]Deformable [73]N-Jet (Ours)

98.40% (± 0.38)
98.91% (± 0.27)
99.37% (± 0.05)

91.28% (± 1.88)
86.45% (± 1.32)
98.87% (± 0.21)

4-layer architecture
4 × MNIST

resolution and is more accurate.
B. N-Jet for Image Segmentation
Learning the receptive field size for segmentation. Multiscale information processing is heavily used in modern segmentation architectures, and seems to be an important performance booster [10]. Here, we focus on two popular mechanisms for multi-scale processing, namely the merging of
information at different scales via skip connections in U-Net
architectures [91], and the pooling of information at different
scales via atrous spatial pyramid pooling (ASPP) layers in
DeepLab architectures [34], [92].
Similar to the classification experiments (Section IV-A), we
replace the fixed-size convolutional filters of baseline networks
with the N-Jet definition, where we learn the size and scaleof the filters in the convolution operations during training.
Experiment 1(B). Segmentation with U-Net
Experiment 1.1(B): Segmentation of multi-scale inputs. We
first evaluate the performance of N-Jet models on a small toy
dataset, where each input image is formed by concatenating
4 images (objects) from the Fashion MNIST [93] dataset(Fig. 10). Each object is assigned a random scale s, which
determines the factor by which we upsample the original
Fashion MNIST image, via bilinear interpolation. The scaleaffects the object sizes — the number of pixels occupied by
the object in the image. We construct four different training
sets: three where the scale of each object is homogeneous:the discrete variable s has the probability mass functions
P (s = 1) = 1, or P (s = 2) = 1, or P (s = 4) = 1; andone where the image contains objects on multiple scales: s
has the probability mass function P (s) = 0.25 for valuess ∈ {1, 2, 3, 4}. After rescaling, each object is placed in8Validation mIoU [%]JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020(a) Input image, homogeneous scales(b) Input image, multi-scaleFig. 10. Exp 1.1(B): Example images from the multi-scale Fashion MNIST
toy dataset for segmentation. The corresponding segmentation masks are
generated by assigning the class label of the corresponding object to pixels
whose input grayscale values are above the threshold hx,y ≥ hθ = 0.2.

one quadrant of the input image, centered at a uniformly
sampled random location. The corresponding ground truth
segmentation masks are created by assigning the class label(1 . . . 10) of the corresponding object to pixels whose input
grayscale values hx,y are above the threshold hθ = 0.2, byassigning the background label (0) to pixel locations where
hx,y = 0, and by assigning an ignore index to undetermined
pixel locations where 0 < hx,y < hθ . The ignored pixel
locations do not contribute to the loss during training and do
not contribute to the accuracy at test time.
Due to the simple nature of the training set, we use asmall U-Net architecture, where the encoding network has
three levels, as opposed to five in the original U-Net [91].
This corresponds to two downsampling layers. Each levelis composed of two convolutional layers, followed by the
ReLU activation layer. The channel dimension is 64 at thefirst level, and doubles with every downsampling, performed
via 2 × 2 max pooling. In the decoding network, we use
bilinear upsampling to increase feature map size, and in all
convolutional layers we use ‘same’ padding. We train all
networks (N-Jet and baseline) for 50 epochs, using the ADAM
optimizer [94] and learning rate 0.0001. To accommodate
input images of different sizes, and keep with the original
U-Net implementation, we use a batch size of 1 and nobatch normalization, but high momentum β = (0.9, 0.999). To
combat class imbalance, given the especially high frequencyof the background class, we weigh the losses with the inverseof class frequencies in the training set. For the N-Jet models,
we use filters with basis order 4 and 2. The scale parameter
σ is shared between all filters in a convolutional layer.
After training, we evaluate segmentation performance on thevalidation set using the mean intersection over union (mIoU)
over all object classes. Each validation set is constructed in thesame way as the corresponding training set, using the Fashion
MNIST validation images. We find that as we increase theaverage scale s of the segmented objects (homogeneous scalecase) or the variance of object scales s (multi-scale case),
N-Jet models successfully optimize the scale parameter σaccordingly. This makes N-Jets capable of adapting to different
object scales without changing the network architecture, depth
or hyper-parameters at all. In contrast, baseline U-Net models
with fixed filter size cannot adapt their receptive field (RF)

100
95
90
85
80
75
70
65
60

baseline
N-jet order 2
N-jet order 4

s=1

s=2

s=4

Multiscale Fashion MNIST

p(s)=0.25

Fig. 11. Exp 1.1(B): mIOU scores averaged over 5 repetitions with different
random seeds on multi-scale Fashion MNIST. Error bars denote the standard
deviation. Validation mIoU decreases dramatically for baseline networks with
fixed filter sizes (orange) for larger objects. In comparison, N-Jet models arerobust against scale changes, as they can learn the filter size and scale, withoutchanging the architecture or hyper-parameters.

size based on the object scales in the training set, and theirsegmentation performance decays for larger objects (Fig. 11).
In addition to the robustness of N-Jet networks againstchanging object scales, we find that N-Jets of only order 2(where each kernel is defined by only 6 free parameters) isenough to obtain good validation accuracy. In fact, N-Jets oforder 4 perform slightly worse for larger s. This is partly
because the reduction of the basis order acts as a regularization
via parameter reduction on our simple toy dataset, and partly
because Fashion MNIST (especially after upscaling) does notcontain many high frequency components, which the higher
order Gaussian derivatives can capture.
Experiment 1.2(B): Learning the receptive field size. Whileσ optimization is successful for different basis orders, we notethat the N-Jet model with basis order 4 has a larger numberof free parameters than the baseline U-Net. Nevertheless, weobserve on our toy dataset that the validation mIoU depends
only weakly on the number of parameters, beyond a certain
network size. For the multi-scale segmentation task with s =4, where the scale of objects are increased by a factor of 4,
we find that the receptive field size at the end of the encoding
network largely determines the validation mIoU (Fig. 12). To
demonstrate this, we vary the number of parameters and thereceptive field size at the end of encoding in the baseline U-Net
models, until we match the N-Jet performance: we increase
the kernel size k from 3 to 4 and 5, and expand the depthof the baseline network by increasing the number of encoding
and decoding levels from 3 (10 convolutional layers) to 4 (14convolutional layers) and 5 (18 convolutional layers). To keep
the number of trainable parameters at a reasonable level, fornetworks with 4 and 5 levels we also decrease the channel
width of the layers (by halving or quartering the number ofchannels in each layer, as given in the legend of Fig. 12).
We find that the N-Jet models can outperform baseline UNet models while using a much smaller number of free parameters, due to σ optimization. In addition, we show that while
the receptive field size is a good predictor of performance,

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

TABLE VI
E XP 2(B): S EGMENTATION M I O U SCORES ON Pascal VOC VALIDATIONSET ALONG WITH THE NUMBER OF PARAMETERS IN THE ASPP LAYER .
N-J ET MODELS WITH WEIGHT SHARING , LOWER NUMBER OF
PARAMETERS , AND NO HYPER - PARAMETER TUNING ARE AS ACCURATE AS
THE BASELINE DeepLabv2 MODEL WITH WEIGHT SHARING . M OREOVER
THE N-J ET NETS SUBSTANTIALLY REDUCE THE NUMBER OF PARAMETERS .

Validation mIoU [%]

85
80
75
70
65 1M

9

2M

U-Net, k=3, 3 levels
U-Net, k=4, 3 levels
U-Net, k=5, 3 levels
U-Net, k=3, 4 levels
U-Net, k=3, 4 levels, 0.5 width
U-Net, k=3, 5 levels, 0.5 width
U-Net, k=3, 5 levels, 0.25 width
N-Jet (ours), 3 levels, order 4
N-Jet (ours), 3 levels, order 2

3M
4M
5M
# of parameters

6M

7M

Fig. 12. Exp 1.2(B): The effect of the number of parameters (x-axis)
and receptive field size (round marker size) on the mIOU scores in thevalidation set of multi-scale Fashion MNIST dataset with s = 4. We observethat the number of model parameters is a weak predictor of segmentation
performance compared to receptive field size which can be learned by N-Jets
during training. We note that N-Jet models (in black squares) can perform
dramatically better than the baseline model with the same architecture (orange
circle), and overall display high mIoU with a low number of parameters.

it cannot be learned during training for the baseline U-Net,
and would need to be optimized via hyper-parameter scans.
This can potentially mean increasing the depth of the network
to match the input resolution, which cannot be parallelized.
Finally, we observe that slightly better validation mIoU can beobtained by baseline models, with almost 7 times the numberof parameters and double the number of layers. We attribute
this slight performance boost to the much larger depth, andthus increased number of nonlinearities in the network.
Experiment 2(B): Segmentation with DeepLabv2. Next, weconsider a more realistic segmentation task on the Pascal
VOC (SBD) dataset [89], [90] using the DeepLabv2 architecture [34]. Modern DeepLab models take advantage of dilated
convolutions to aggregate information from multiple scales
in atrous spatial pyramid pooling (ASPP) layers [34], [92].
However, dilated kernels can only be upsampled discretely,
based on the dilation rate in units of pixels. In addition, it istypically not possible to determine a priori which scales ina dataset contain task-relevant information and the employed
dilation rates need to be optimized using excessive hyperparameter scans. We propose N-Jets as an alternative to
optimizing the scales in a continuous way during training,eliminating the need to excessively search for dilation rates
for each task.
To that end, we employ the DeepLabv2 model with a
ResNet-101 backbone pretrained on the 20 class subset of
the MS COCO dataset [95] corresponding to the Pascal
VOC classes. We retain all the network and training hyperparameters of the original DeepLabv2 model and finetune
the baseline network with an ASPP output layer on Pascal
VOC with the batch normalization layers frozen. For the NJet network, we replace the 4 convolutional layers of the ASPP
layer with different dilation rates with 4 N-Jet layers withindependent, learnable scales σ (during finetuning) and we
impose weight sharing between the different scales (i.e. same

Model

mIoU

# parameters

DeepLabv2
DeepLabv2, weight sharing

76.13
74.73

1,548,372
387,093

N-Jet, order 3
N-Jet, order 2
N-Jet, order 1

75.17
74.58
74.89

430,101
258,069
129,045

α values). On top of eliminating the need to manually tune
the dilation rates, N-Jet models with weight sharing also havethe potential to dramatically reduce the number of parametersin ASPP layers.
We find that DeepLabv2 with N-Jet output layers indeed
allows for parameter reductions (Tab. VI). Using an N-Jet output layer with basis order 3 and weight sharing, we achievevalidation mIoU values within 1% of the baseline network,
while reducing the number of parameters by nearly a factorof 4. As an additional control, we also train a baseline networkwith weight sharing within the ASPP layer. Interestingly,
we observe that our N-Jet models attain on par or better
performance than the weight-tied baseline network, even when
we only use a basis order of 1 (each kernel is defined by only
3 free parameters).
It is worth noting that these validation mIoUs are achieved
with no hyper-parameter tuning for the N-Jet models, and despite not using N-Jet layers in the pretraining of the DeepLabbackbone. As it is, we believe N-Jet output layers may
be used for multi-scale processing applications with further
hyper-parameter tuning of learning rates and regularization
parameters, or can be used out of the box to estimate theoptimal scale or dilation rates for other architectures.
V. D ISCUSSION AND LIMITATIONS
To illustrate the differences between standard convolutional
layers and N-Jet convolutional layers, we visualize a set oftrained baseline filters compared to the equivalent N-Jet filters(Fig. 13). We find that in many models earlier layers will
converge to smaller σ values during training (Fig. 13, top),while deeper layers are prone to learning larger filter sizes(Fig. 13, bottom).
In addition, the strength of the N-Jet representation lies inthat it can learn filter sizes, and thus the receptive field size,
during training. However, recent work has demonstrated thatthe effective receptive field (eRF) size of networks can beconsiderably smaller than what would be expected from thekernel size [96]. We investigate the change in eRF size in our
N-Jet models by visualizing the gradients with respect to the
input image in our models trained on the multiscale FashionMNIST dataset (Fig. 14). We find that, as expected, the eRF
size of N-Jet models grows with the size of the training images,

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020

10

N-Jet-NiN

N-Jet

Deeper layer

NiN

N-Jet-NiN

U-NetFirst layer

NiN

Learned Filters

Fig. 13. Filter visualization. Learned filters in the NiN and N-Jet-NiN
models. In the first layer, N-Jet-NiN converges to a kernel size of 5 × 5,
matching the baseline kernel size (top). In deeper layers, the learned σ value
can be much larger, leading to larger kernel sizes: in this example 11 × 11
(bottom). All filter values are centered around 0.

proportionally to the growth of filter sizes. The baseline U-Net
model with 3×3 kernels cannot learn to adapt its receptive fieldsize during training, its eRF size remains relatively constantas a function of the input image scale.
One of the limitations of our proposed kernels is that they
are typically larger than the standard 3 × 3 px, and therefore
the convolutions take longer to compute. This comes at no
cost in parameters as the size of the N-Jet filters is only
affected by the scale parameters, σ. Additionally, computing
the Gaussian basis is more expensive because it involves
more operations: computing the Hermite polynomials, andobtaining the individual Gaussian basis from these, followedby estimating their linear combinations with the weights α.
For the NiN architecture, our model is ≈ 2× slower thanthe baseline model. As the network depth increases, so do thecomputations. However, manual architecture search takes a lot
longer for finding the appropriate resolution hyper-parameters,
because it requires a grid search over all possible filter sizes
given a specific network depth and sub-sampling strategy.
VI. C ONCLUSION
We learn the resolution in deep convolutional networks.
Learning the resolution frees the network architect from setting
resolution related hyper-parameters such as the receptive fieldsize and subsampling layers, which are dataset and network
dependent. While we learn the receptive field size and the feature map subsampling for classification, the resolution is also
determined by the depth of the network, as each layer increases
the resolution linearly. Network depth is not something welearn, and thus we do not learn all resolution hyper-parameters.
In addition to hard-coded filter sizes and subsampling layers,
current CNN architectures are also designed to share the same
filter size in a single layer. Due to computational restrains, ourimplementation does not make it possible to learn a σ for eachfilter, rather than per layer. We leave this as potential future
work. To conclude, by replacing pixel-weights convolutional

Fig. 14. Effective receptive field sizes. Effective receptive field (eRF) size for
the N-Jet models trained on the multiscale Fashion-MNIST dataset. Columns(s = {1, 2, 4}) denote models trained with input images scaled up by a factor
of s. eRF visualizations are obtained by the gradients back-propagated to input
pixel space [96]. We find that the eRF size of N-Jet models scale up with thesize of the input, while the eRF size of baseline U-Net models stay constant.

layers with our N-Jet convolutional layers we show that we canobtain similar performance as the baseline methods, without
tuning the hyper-parameters controlling the resolution.
Acknowledgements. This publication is part of the project”Pixel-free deep learning” (with project number 612. 001.805
of the research programme TOP which is financed by the
Dutch Research Council (NWO).
R EFERENCES[1] J. J. Koenderink, “The structure of images,” Biological cybernetics,
vol. 50, no. 5, pp. 363–370, 1984.
[2] W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective receptive field in deep convolutional neural networks,” in NIPS,
pp. 4898–4906, 2016.
[3] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in ECCV, pp. 818–833, Springer, 2014.
[4] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten, “Densely
connected convolutional networks,” in CVPR, 2017.
[5] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residualtransf foa deep neural networks,” in CVPR, pp. 5987–5995, IEEE, 2017.
[6] T. Lindeberg, Scale-space theory in computer vision, vol. 256. Springer
Science & Business Media, 2013.
[7] L. Florack, B. T. H. Romeny, M. Viergever, and J. Koenderink, “Thegaussian scale-space paradigm and the multiscale local jet,” IJCV,
vol. 18, no. 1, pp. 61–75, 1996.
[8] J.-H. Jacobsen, J. C. van Gemert, Z. Lou, and A. W. M. Smeulders,“Structured receptive fields in CNNs,” in CVPR, 2016.
[9] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,”in CVPR, 2015.
[10] L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, “Attention toscale: Scale-aware semantic image segmentation,” in CVPR, 2016.
[11] G. Ghiasi and C. C. Fowlkes, “Laplacian pyramid reconstruction andrefinement for semantic segmentation,” in ECCV, 2016.
[12] S. Bai, V. Koltun, and J. Z. Kolter, “Multiscale deep equilibrium
models,” ICLR workshop, 2018.
[13] A. Kanazawa, A. Sharma, and D. Jacobs, “Locally scale-invariant
convolutional neural networks,” NIPS workshop, 2014.
[14] T.-W. Ke, M. Maire, and X. Y. Stella, “Multigrid neural architectures,”in CVPR, 2017.
[15] Y. Li, Z. Kuang, Y. Chen, and W. Zhang, “Data-driven neuron allocation
for scale aggregation networks,” in CVPR, 2019.

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020[16] Z. Liao and G. Carneiro, “A deep convolutional neural network modulethat promotes competition of multiple-size filters,” Pattern Recognition,
2017.
[17] E. Agustsson, D. Minnen, N. Johnston, J. Balle, S. J. Hwang, and
G. Toderici, “Scale-space flow for end-to-end optimized video compression,” in CVPR, pp. 8503–8512, 2020.
[18] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie,“Feature pyramid networks for object detection,” in CVPR, pp. 2117–
2125, 2017.
[19] Z. Liu, G. Gao, L. Sun, and L. Fang, “Ipg-net: Image pyramid guidance
network for small object detection,” in CVPR Workshops, pp. 1026–
1027, 2020.
[20] X. Wang, S. Zhang, Z. Yu, L. Feng, and W. Zhang, “Scale-equalizing
pyramid convolution for object detection,” in CVPR, pp. 13359–13368,
2020.
[21] T. Yang, S. Zhu, C. Chen, S. Yan, M. Zhang, and A. Willis, “Mutualnet: Adaptive convnet via mutual learning from network width andresolution,” in ECCV, 2020.
[22] D. Marcos, B. Kellenberger, S. Lobry, and D. Tuia, “Scale equivariancein cnns with vector fields,” ICML workshop, 2018.
[23] I. Sosnovik, M. Szmaja, and A. Smeulders, “Scale-equivariant steerable
networks,” ICLR, 2020.
[24] M. Xiao, S. Zheng, C. Liu, Y. Wang, D. He, G. Ke, J. Bian, Z. Lin, andT.-Y. Liu, “Invertible image rescaling,” ECCV, 2020.
[25] B. Uzkent and S. Ermon, “Learning when and where to zoom with deep
reinforcement learning,” in CVPR, pp. 12345–12354, 2020.
[26] D. Marin, Z. He, P. Vajda, P. Chatterjee, S. Tsai, F. Yang, and Y. Boykov,“Efficient segmentation: Learning downsampling near semantic boundaries,” in ICCV, pp. 2131–2141, 2019.
[27] Y. Li, D. M. Tax, and M. Loog, “Scale selection for supervised image
segmentation,” Image and Vision Computing, vol. 30, no. 12, pp. 991–
1003, 2012.
[28] Y. Li, D. M. Tax, and M. Loog, “Supervised scale-invariant segmentation (and detection),” in International Conference on Scale Space andVariational Methods in Computer Vision, pp. 350–361, 2011.
[29] G. A. Sigurdsson, A. Gupta, C. Schmid, and K. Alahari, “Beyond thecamera: Neural networks in world coordinates,” CoRR, 2020.
[30] D. Wang, E. Shelhamer, B. Olshausen, and T. Darrell, “Dynamic scaleinference by entropy minimization,” CoRR, 2019.
[31] C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L. Yuille,and L. Fei-Fei, “Auto-deeplab: Hierarchical neural architecture searchfor semantic image segmentation,” in Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 82–92,
2019.
[32] Y. Li, L. Song, Y. Chen, Z. Li, X. Zhang, X. Wang, and J. Sun, “Learning
dynamic routing for semantic segmentation,” in CVPR, pp. 8553–8562,
2020.
[33] G.-J. Qi, “Hierarchically gated deep networks for semantic segmentation,” in CVPR, pp. 2267–2275, 2016.
[34] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille,“Deeplab: Semantic image segmentation with deep convolutional nets,
atrous convolution, and fully connected CRFs,” TPAMI, 2017.
[35] G. Papandreou, I. Kokkinos, and P.-A. Savalle, “Modeling local andglobal deformations in deep learning: Epitomic convolution, multiple
instance learning, and sliding window detection,” in CVPR, 2015.
[36] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated
convolutions,” in ICLR, 2016.
[37] F. Yu, V. Koltun, and T. Funkhouser, “Dilated residual networks,” in
CVPR, 2017.
[38] M. Zhang, J. Zhao, X. Li, L. Zhang, and Q. Li, “Ascnet: Adaptivescale convolutional neural networks for multi-scale feature learning,” inInternational Symposium on Biomedical Imaging, pp. 144–148, 2020.
[39] K. Fukushima, “A neural network model for selective attention in visual
pattern recognition,” Biological Cybernetics, vol. 55, no. 1, pp. 5–15,
1986.
[40] T. Serre, L. Wolf, and T. Poggio, “Object recognition with features
inspired by visual cortex,” in CVPR, vol. 2, pp. 994–1000, Ieee, 2005.
[41] Y.-L. Boureau, F. Bach, Y. LeCun, and J. Ponce, “Learning mid-levelfeatures for recognition,” in CVPR, pp. 2559–2566, IEEE, 2010.
[42] D. Scherer, A. Müller, and S. Behnke, “Evaluation of pooling operations
in convolutional architectures for object recognition,” Artificial Neural
Networks–ICANN 2010, pp. 92–101, 2010.
[43] Y.-L. Boureau, J. Ponce, and Y. LeCun, “A theoretical analysis of feature
pooling in visual recognition,” in ICML, 2010.
[44] C.-Y. Lee, P. Gallagher, and Z. Tu, “Generalizing pooling functions in
cnns: Mixed, gated, and tree,” TPAMI, 2017.

11[45] Z. Shi, Y. Ye, and Y. Wu, “Rank-based pooling for deep convolutional
neural networks,” Neural Networks, vol. 83, pp. 21–31, 2016.
[46] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep
convolutional networks for visual recognition,” in ECCV, pp. 346–361,
Springer, 2014.
[47] O. Rippel, J. Snoek, and R. P. Adams, “Spectral representations forconvolutional neural networks,” in NIPS, pp. 2449–2457, 2015.
[48] M. Zeiler and R. Fergus, “Stochastic pooling for regularization of deep
convolutional neural networks,” in ICLR, 2013.
[49] S. Zhai, H. Wu, A. Kumar, Y. Cheng, Y. Lu, Z. Zhang, and R. Feris,“S3pool: Pooling with stochastic spatial sampling,” in CVPR, 2017.
[50] R. Zhang, “Making convolutional networks shift-invariant again,” ICML,
2019.
[51] B. Graham, “Fractional max-pooling,” CoRR, 2014.
[52] T. Iijima, “Basic theory of pattern observation,” Technical Group on
Automata and Automatic Control, pp. 3–32, 1959.
[53] A. P. Witkin, “Scale-space filtering,” in International Joint Conferenceon Artificial Intelligence, 1983.
[54] J. Babaud, A. P. Witkin, M. Baudin, and R. O. Duda, “Uniqueness ofthe gaussian kernel for scale-space filtering,” TPAMI, vol. 1, pp. 26–33,
1986.
[55] R. Duits, L. Florack, J. De Graaf, and B. ter Haar Romeny, “On theaxioms of scale space theory,” Journal of Mathematical Imaging andVision, vol. 20, no. 3, pp. 267–298, 2004.
[56] L. M. Florack, B. M. ter Haar Romeny, J. J. Koenderink, and M. A.
Viergever, “Scale and the differential structure of images,” Image andVision Computing, vol. 10, no. 6, pp. 376–388, 1992.
[57] E. P. Simoncelli, W. T. Freeman, E. H. Adelson, and D. J. Heeger,“Shiftable multiscale transforms,” IEEE transactions on Information
Theory, vol. 38, no. 2, pp. 587–607, 1992.
[58] J. Bruna and S. Mallat, “Invariant scattering convolution networks,”
TPAMI, vol. 35, no. 8, pp. 1872–1886, 2013.
[59] S. Mallat, “Group invariant scattering,” Communications on Pure andApplied Mathematics, vol. 65, no. 10, pp. 1331–1398, 2012.
[60] F. Cotter and N. Kingsbury, “Visualizing and improving scattering
networks,” in MLSP, 2017.
[61] L. Sifre and S. Mallat, “Rotation, scaling and deformation invariant
scattering for texture discrimination,” in CVPR, 2013.
[62] S. Mallat, A wavelet tour of signal processing. Academic press, 1999.
[63] E. Oyallon, E. Belilovsky, and S. Zagoruyko, “Scaling the scatteringtransform: Deep hybrid networks,” in ICCV, 2017.
[64] A. Singh and N. Kingsbury, “Efficient convolutional network learning
using parametric log based dual-tree wavelet scatternet,” in CVPR
workshop, 2017.
[65] Y. Li, S. Gu, L. V. Gool, and R. Timofte, “Learning filter basis forconvolutional neural network compression,” in ICCV, pp. 5623–5632,
2019.
[66] D. E. Worrall, S. J. Garbin, D. Turmukhambetov, and G. J. Brostow,“Harmonic networks: Deep translation and rotation equivariance,” in
CVPR, July 2017.
[67] S. Luan, B. Zhang, C. Chen, X. Cao, Q. Ye, J. Han, and J. Liu, “Gabor
Convolutional Networks,” CoRR, 2017.
[68] G. Zoumpourlis, A. Doumanoglou, N. Vretos, and P. Daras, “Non-linear
convolution filters for cnn-based learning,” in ICCV, 2017.
[69] V. Jampani, M. Kiefel, and P. V. Gehler, “Learning sparse high dimensional filters: Image filtering, dense crfs and bilateral neural networks,”in CVPR, 2016.
[70] Q. Chen, J. Xu, and V. Koltun, “Fast image processing with fullyconvolutional networks,” in CVPR, 2017.
[71] Z. Sun, M. Ozay, and T. Okatani, “Design of kernels in convolutional
neural networks for image classification,” in ECCV, 2016.
[72] Y. Jeon and J. Kim, “Active convolution: Learning the shape of convolution for image classification,” in CVPR, 2017.
[73] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformableconvolutional networks,” in ICCV, 2017.
[74] A. Shocher, B. Feinstein, N. Haim, and M. Irani, “From discrete tocontinuous convolution layers,” CoRR, 2020.
[75] F. Xia, P. Wang, L.-C. Chen, and A. L. Yuille, “Zoom better to see
clearer: Human and object parsing with hierarchical auto-zoom net,” in
ECCV, pp. 648–663, Springer, 2016.
[76] Z. Hao, Y. Liu, H. Qin, J. Yan, X. Li, and X. Hu, “Scale-aware facedetection,” in CVPR, pp. 6186–6195, 2017.
[77] Y. Liu, H. Li, J. Yan, F. Wei, X. Wang, and X. Tang, “Recurrent scale
approximation for object detection in cnn,” in ICCV, 2017.
[78] E. Shelhamer, D. Wang, and T. Darrell, “Blurring the line between
structure and learning to optimize and adapt receptive fields,” CoRR,
2019.

JOURNAL OF TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. XX, XX 2020[79] Z. Xiong, Y. Yuan, N. Guo, and Q. Wang, “Variational contextdeformable convnets for indoor scene parsing,” in CVPR, pp. 3992–
4002, 2020.
[80] T. Lindeberg, “Scale-covariant and scale-invariant gaussian derivative
networks,” 2020.
[81] R. A. Young, “The gaussian derivative model for spatial vision: I. retinal
mechanisms,” Spatial vision, vol. 2, no. 4, pp. 273–293, 1987.
[82] J.-B. Martens, “The hermite transform-theory,” IEEE Transactions on
Acoustics, Speech, and Signal Processing, vol. 38, no. 9, pp. 1595–1606,
1990.
[83] M. Lin, Q. Chen, and S. Yan, “Network in network,” ICLR, 2014.
[84] A. Krizhevsky and G. Hinton, “Learning multiple layers of features fromtiny images,” Tech Report, 2009.
[85] Y. Netzer, T. Wang, A. C, A. Bissacco, B. Wu, and A. Y. Ng, “Reading
digits in natural images with unsupervised feature learning,” NIPS
workshop, vol. 2011, no. 2, p. 5, 2011.
[86] J. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller, “Striving
for simplicity: The all convolutional net,” in ICLR workshop, 2015.
[87] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in CVPR, pp. 770–778, 2016.
[88] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine
Learning, pp. 6105–6114, PMLR, 2019.
[89] B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik, “Semantic
contours from inverse detectors,” in 2011 International Conference on
Computer Vision, pp. 991–998, IEEE, 2011.
[90] M. Everingham, S. A. Eslami, L. Van Gool, C. K. Williams, J. Winn,
and A. Zisserman, “The pascal visual object classes challenge: A
retrospective,” International journal of computer vision, vol. 111, no. 1,
pp. 98–136, 2015.
[91] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks
for biomedical image segmentation,” in International Conference on
Medical image computing and computer-assisted intervention, pp. 234–
241, Springer, 2015.
[92] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, “Rethinking
atrous convolution for semantic image segmentation,” arXiv preprint
arXiv:1706.05587, 2017.
[93] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image
dataset for benchmarking machine learning algorithms,” arXiv preprint
arXiv:1708.07747, 2017.
[94] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”in 3rd International Conference on Learning Representations, ICLR,
2015.
[95] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollár, and C. L. Zitnick, “Microsoft COCO: common objects incontext,” in ECCV, vol. 8693, pp. 740–755, 2014.
[96] W. Luo, Y. Li, R. Urtasun, and R. Zemel, “Understanding the effective
receptive field in deep convolutional neural networks,” in 30th International Conference on Neural Information Processing Systems (NeurIPS),
pp. 4905–4913, 2016.

12