Top-Down Networks: A coarse-to-fine reimagination of CNNs
Ioannis Lelekas

Nergis Tomen

Silvia L. Pintea

Jan C. van Gemert

Computer Vision Lab
Delft University of Technology, Netherlands
ﬁneAbstract
Biological vision adopts a coarse-to-fine information
processing pathway, from initial visual detection and binding of salient features of a visual scene, to the enhanced
and preferential processing given relevant stimuli. On thecontrary, CNNs employ a fine-to-coarse processing, movingfrom local, edge-detecting filters to more global ones extracting abstract representations of the input. In this paper
we reverse the feature extraction part of standard bottom-up
architectures and turn them upside-down: We propose topdown networks. Our proposed coarse-to-fine pathway, by
blurring higher frequency information and restoring it onlyat later stages, offers a line of defence against adversarial attacks that introduce high frequency noise. Moreover,since we increase image resolution with depth, the high resolution of the feature map in the final convolutional layer
contributes to the explainability of the network’s decision
making process. This favors object-driven decisions overcontext driven ones, and thus provides better localized classactivation maps. This paper offers empirical evidence forthe applicability of the top-down resolution processing tovarious existing architectures on multiple visual tasks.

1. Introduction
In human biological vision, perceptual grouping of visual features is based on Gestalt principles, where factors
such as proximity, similarity or good continuation of features generate a salient percept [42]. Salient objects are
rapidly and robustly detected and segregated from the background in what is termed the “pop-out” effect [7, 22]. Thisinitial detection and grouping of salient features into a coherent percept, leads to preferential processing by the visualsystem, described as stimulus-driven attention [52]. For relevant visual stimuli, the exogenously directed attention issustained, and results in a more detailed visual evaluationof the object. This typical pipeline of perception and attention allocation in biological vision represents an efficient,coarse-to-fine processing of information [14]. In contrast,

Fine-to-coarse
Coarse-to-ﬁnecoarseBlur+subsample

Figure 1. A coarse-to-fine versus fine-to-coarse processing pathway. The conventional fine-to-coarse pathway in a CNN sacrifices localization for semantically richer information. The opposite path, proposed in this paper, starts from the coarsest input andfocuses on the context: given the sky, grass and building, it isclearly a landscape scene of a building. Moving to finer representations of the input, the focus shifts to local information. Architectural aspects of the building, and the cross on the top, are now
the most informative for classifying the image as a church. Our
proposed coarse-to-fine pathway is in line with human biologicalvision, where detection of global features precedes the detection of
local ones, for which further processing of the stimuli is required.

modern CNNs (Convolutional Neural Networks) do not incorporate this perspective [12, 23, 38, 40].
Standard CNNs begin with the high resolution input, andpropagate information in a fine-to-coarse pathway. Early
layers learn to extract local, shareable features, whereas
deeper layers learn semantically rich and increasingly invariant representations. In this paper we propose the reversal of the conventional feature extraction of standard CNNs,
as depicted in Figure 1. More specifically, we suggest the
adoption of a coarse-to-fine processing of the input, whichcan be interpreted as gradual focusing of visual attention.
The top-down hierarchy first extracts the gist of a scene,
starting from a holistic initial representation, and subsequently enhances it with higher frequency information.
A growing body of literature since the seminal workof [10, 41] shows that adversarial perturbations with highfrequency components may cause substantial misclassifications. Suppressing higher frequencies in the input image, as
proposed in our top-down paradigm, can provide a first lineof defence. At the same time, explainability of the decision
making process of CNNs has recently emerged as an important research direction [36, 54]. In this context, our coarseto-fine processing scheme, having feature maps with higher
spatial resolution at deeper layers, favors object-driven decisions over context-driven ones, and provides better localized class activation maps.
We make the following contributions: (i) We propose
biologically inspired top-down network architectures, obtained by reversing the resolution processing of conventional bottom-up CNNs; (ii) We analyze various methodsof building top-down networks based on bottom-up counterparts as well as the difference in resolution-processing
between these models, providing a versatile frameworkthat is directly applicable to existing architectures; (iii)
We compare our proposed model against the baseline ona range of adversarial attacks and demonstrate enhanced
robustness against certain types of attacks. (iv) We findenhanced explainability for our top-down model, withpotential for object localization tasks. Trained models
and source code for our experiments are available online:
https://github.com/giannislelekas/topdown.

2. Related work
Coarse-to-fine processing. Coarse-to-fine processing is an
integral part of efficient algorithms in computer vision. Iterative image registration [30] gradually refines registrationfrom coarser variants of the original images, while in [16] acoarse-to-fine optical flow estimation method is proposed.
Coarse-to-fine face detection is performed by processing
increasingly larger edge arrangements in [8], and coarseto-fine face alignment using stacked auto-encoders is introduced in [50]. Efficient action recognition is achievedin [44] by using coarse and fine features coming fromtwo LSTM (Long Short-Term Memory) modules. In [34]coarse-to-fine kernel networks are proposed, where a cascade of kernel networks are used with increasing complexity. Existing coarse-to-fine methods consider both coarse
input resolution, as well as gradually refined processing.
Here, we also focus on coarse-to-fine image resolution,
however we are the first to do this in a single deep neuralnetwork, trained end-to-end, rather than in an ensemble.

Bottom-up and top-down pathways. Many approaches
exploit high spatial resolution for finer feature localization,
which is crucial for semantic segmentation. The U-net [33]and FPN (Feature Pyramid Networks) [29] merge information from bottom-up and top-down pathways, combining
semantically rich information of the bottom-up with the fine
localization of the top-down stream. Similarly, combinations of a high-resolution and a low-resolution branch were
proposed for efficient action recognition [5], for face hallucination [25], and depth map prediction [3]. Top-down
signals are also used to model neural attention via a backpropagation algorithm [49], and to extract informative localization maps for classification tasks in Grad-CAM [36].
Similarly, we also focus on top-down pathways where we
slowly integrate higher levels of detail, however our goalis biologically-inspired resolution processing, rather thanfeature-map activation analysis.
Multi-scale networks. Merging and modulating information extracted from multiple scales is vastly popular[15, 21, 47, 48, 46]. In [48] feature maps are resized by afactor to obtain cascades of multiple resolutions. Incremental resolution changes during GAN (Generative Adversarial Network) training are proposed in [20]. Convolutional
weight sharing over multiple scales is proposed in [1, 47].
Similarly [6] performs convolutions over multiple scales incombination with residual connections. In [21] convolutions are performed over a grid of scales, thus combining
information from multiple scales in one response, and [39]combines responses over multiples scales, where filters aredefined using 2D Hermite polynomials with a Gaussian envelope. Spatial pyramid pooling is proposed in [11] for aggregating information at multiple scales. In this work, we
also extract multi-resolution feature maps, in order to start
processing from the lowest image scale and gradually restore high frequency information at deeper layers.
Beneficial effects of blurring. Suppressing high frequency
information by blurring the input can lead to enhanced robustness [43, 53]. Models trained on blurred inputs exhibit increased robustness to distributional shift [19]. The
work in [9] reveals the bias of CNNs towards texture, andanalyzes the effect of blurring distortions on the proposed
Stylized-ImageNet dataset. Anti-aliasing by blurring before
downsampling contributes to preserving shift invariance in
CNNs [51]. By using Gaussian kernels with learnable variance, [37] adapts the receptive field size. Rather than changing the receptive field size, works such as [27, 26, 31] use
spatial smoothing for improved resistance to adversarial attacks. Similarly, we also rely on Gaussian blurring before
downsampling the feature maps to avoid aliasing effects,
and as a consequence we observe improved robustness toadversarial attacks.

Identity

DownscaleUpscaleConv-ReLU-Downscale
Conv-ReLU
Resolution merge
OutputInput

Cat

Bottom-up (BU) network

Cat

Input

Top-down (TD) network

Figure 2. Left: The bottom-up (BU ) baseline network. Feature maps decrease in spatial resolution with network depth. Right: The
proposed top-down (T D) network. The T D model reverses the feature extraction pathway of the baseline network. It employs three inputs
from highest to lowest scale, starts processing from the lowest resolution and progressively adds high resolution information.

3. Top-down networks
Top-down (T D) networks mirror the baseline bottom-up(BU ) networks, and reverse their feature extraction pathway. Information flows in the opposite direction, movingfrom lower to higher resolution feature maps. The initial input of the network corresponds to the minimum spatial resolution occurring in the BU baseline network. Downscaling
operations are replaced by upscaling, leading to the coarseto-fine information flow. By upscaling, the network canmerely “hallucinate” higher resolution features. To restore
the high frequency information, we use resolution merges,
which combine the hallucinated features with higher frequency inputs, after each upscaling operation. Figure 2 depicts the difference between the BU architecture and our
proposed T D architecture.

3.1. Input and feature map resizing
To avoid artifacts hampering the performance of the network [51], we blur the inputs before downsampling. For the
upsampling operation we use interpolation followed by convolution. We have experimented with both nearest neighbour and bilinear interpolation, and have noticed improved
robustness against adversarial attacks for nearest neighborinterpolation. We have also considered the use of transpose
convolutions, however we did not adopt these due to detrimental checkerboard artifacts.

ERF (effective receptive field) size computation. Neurons in each layer i of a typical bottom-up network has asingle ERF size ri determined by the kernel size ki and thecumulative stride mi (given stride si at layer i).
ri = ri−1 + (ki − 1) mi−1
mi = mi−1 · si

(1)Assuming only 3 × 3 convolutions with stride 1, the example BU architecture in Figure 2 will have an ERF size of 3
pixels, and 18 pixels in each direction after the first and final
convolutional layers, respectively. In contrast, for the T D
network, considering a Gaussian blurring window of width
6σ, the lowest resolution branch will already have an ERF
size of 12σ + 2 at the input level and of 12σ + 10 after the
first convolutional layer (comparable to the final layer of a
BU network already with σ = 2/3 pixels). Furthermore,in contrast to BU , output from neurons with varying ERFs
are propagated through the merging points. To get a lower
bound on the T D ERF sizes, we consider resolution merging methods which do not provide RF enlargement (e.g. as
depicted in fig. 3, but without the 3 × 3 convolution at theend). Thus, at the final merging point of the T D architecture, ERF sizes of 3 pixels and 12σ +14 pixels are merged
together. In conclusion, already from the first layer, T D has
the ERF size that the BU only obtains at the last layer.

3.2. Merging low and high resolution
Figure 3 depicts the considered method for merging thehigh resolution input with the low resolution information.
We first upsample the low resolution input via a 1 × 1 convolution and use an element-wise addition with the highresolution branch. This information is then concatenatedwith the original high resolution information on the channel
dimension. We subsequently use a 3 × 3 convolution to expand the receptive field of the filters. The proposed merging
of information slightly increases the number of parameters,
while being effective in practice.

Figure 3. Merging low and high-frequency feature maps: we use a
1×1 convolution followed by an element-wise addition; this information is concatenated with the high-resolution input and followed
by a 3 × 3 convolution that expands the receptive field size.

3.3. Filter arrangement
Feature extraction pathway of the T D network reverses
the BU : information propagates from lower to higher spatial dimensions in a T D network, while the number of filters shrinks with increasing depth. The choice of expanding the number of filters at deeper layers in the BU network is efficiency-oriented. As the feature map resolution
decreases, the number of channels increases, retaining thecomputational complexity roughly fixed per layer. Typically, in standard architectures the filters are doubled every
time dimensions are halved [12, 38].
In our method we consider three options for deciding thenumber of filters per layer: the T D model which is exactly
the opposite of the BU in that the number of channels are
reduced with depth; the uniform model (T Duni ) where thelayers have a uniform number of filters; and the reversed
model (T Drev ) which follows the BU filter arrangement,with channel dimension widened with depth.

4. Experiments
In Exp 1 we evaluate the three different filter arrangement options proposed for the top-down model. We compare these model variations with the bottom-up baseline on
the MNIST, Fashion-MNIST and CIFAR10 classification
tasks. In Exp 2 we evaluate the robustness of our proposed model against various adversarial attacks applied on
the same datasets. Finally, in Exp 3 we illustrate the explainability capabilities of our top-down model when compared to the bottom-up, and demonstrate its benefits for asmall object localization task.
Experimental setup. We compare our T D proposal with
its respective BU baseline on MNIST, Fashion-MNIST and
CIFAR10. For the simpler MNIST tasks we consider asbaselines the “LeNetFC”, a fully-convolutional variant of
LeNet [24] and following [28], a lightweight version of
the NIN (Network-In-Network) architecture, namely “NINlight” with reduced filters. The original architecture wasused for the CIFAR10 task, along with the ResNet32 introduced in [12] incorporating the pre-activation unit of [13].
Batch Normalization [17] is used in all the networks prior
to the non-linearities. The corresponding T D networks aredefined based on their BU baselines. Table 1 depicts thenumber of parameters of different models. For T D we consider three variants: T D – which is mirroring the BU architecture also in terms of filter depth; T Duni using uniform
filter depth; and T Drev where the filter depth of the T D
is reversed, thus following the filter depth of BU . Thereis an increase in the number of parameters for the T D networks, because we need additional convolutional layers for
merging the high and low resolution information.
We abide by the setup found in the initial publications
for the BU models. For the T D networks we performed a#parameters

Model

LeNetFC
NIN-light
ResNet32
NIN

BU

TD

T DuniT Drev

8k
62k
468k
970k

14k
213k
528k
3,368k

23k
215k
320k
3,397k

58k
214k
563k
3,388kTable 1. Number of trainable parameters for the different architectures considered. Different rows correspond to the different baseline architectures and columns indicate the bottom-up model andthe three top-down variants with different filter arrangements (section 3.3). There is an increase in the number of parameters for
the T D networks, because they merge the high and low resolution
information using additional convolutional layers.

linear search for learning rate, batch size, and weight decay.
For all cases we train with a 90/10 train/val split, using SGD
with momentum of 0.9 and a 3-stage learning rate decay
scheme, dividing the learning rate by 10 at 50% and 80%of the total number of epochs. For the CIFAR10 dataset wetest with and without augmentation—employing horizontal translations and flips. We repeat runs four times, withdataset reshuffling and extracting new training and validation splits, and report mean and standard deviation of thetest accuracy.

4.1. Exp. 1: Bottom-up versus top-downFigure 4 shows the test accuracy of the considered models across datasets. The T D networks are on par with, andin some cases surpassing the corresponding baseline BU
performance. When considering the different filter depth
configurations, T Drev performs best due to increased representational power at higher scales, coming though at costof increased complexity. The NIN architecture adopts a
close to uniform filter arrangement, hence the three T D
variants reach roughly the same performance. We adopt
the T D variants henceforth, on account of the small gap
in performance and reduced complexity. This experiment
provides empirical evidence of the applicability of the proposed pipeline to different network architectures.

4.2. Exp. 2: Adversarial robustness
We evaluate the robustness of BU versus T D against
various attacks, where we attack the test set of each dataset
using the Foolbox [32]. For all the attacks, the default parameters were used. To make the attack bound tighter, we
repeat each attack three times and keep the worst case foreach to define the minimum required perturbation for fooling the network.
Figure 6 provides for each attack, plots of loss in test
accuracy versus the L2 distance between the original andthe perturbed input. T D networks are visibly more re-MNIST

Test accuracy (%)

100

Fashion-MNISTCIFAR10CIFAR10_aug

95
90

BU
TD
TD_uni
TD_rev

85
80

LeNetFC

NIN-light

LeNetFC

NIN-light

ResNet32

NIN

ResNet32

NIN

Figure 4. Exp 1: Comparison of MNIST, Fashion-MNIST, CIFAR10, and CIFAR10 aug (with augmentation) mean test accuracies between
BU and the three different configurations of T D proposed in subsection 3.3. T D networks perform on par with, and at times surpassing,
the baseline performance of its respective BU . Regarding filter depth configurations, T Drev displays the highest performance, at thecost of increased parameters. Considering the small gap in performance and the increased cost for T Drev , we henceforth adopt the T D
configuration.

NIN-light_TD

Test accuray (%)

100
80
60
40
20
0

MNIST

NIN-light_TD

Fashion-MNISTNIN_TD

CIFAR10_aug

Original
SinglePixelAttack
Salt&PepperNoiseAttack
PointwiseAttack
AdditiveGaussianNoiseAttack
AdditiveUniformNoiseAttack
BlendedUniformNoiseAttack
GaussianBlurAttack
ContrastReductionAttack
SpatialAttack
ShiftsAttackhighmediumlow

highmediumlow

highmediumlow

Figure 5. Exp 2: Test accuracy when extracted adversarial perturbations are fed to either the highest, medium, or lowest scale input of
the T D network (refer to figure 2), using the NIN-light baseline on MNIST and Fashion-MNIST, and NIN on CIFAR10. The remaining
two inputs are fed the original, unperturbed samples. As the dataset becomes more challenging, the highest vulnerability moves from themedium input to the highest scale input. This is attributed to the absence of information in the high frequency region for the simpler cases:
i.e. MNIST. (See supplementary material for additional results.)

silient against attacks introducing uncorrelated noise, due
to the coarse-to-fine processing adopted, with downscaled
inputs diminishing the noise. For attacks introducing correlated noise such as the “Pointwise” attack [35], the perturbed pixels tend to lie in smooth regions of the image.
Thus each single pixel value of 0 (or 1) in a region of 1s(or 0s) essentially acts as a Dirac delta function. Based on
the convolutional nature of CNNs this type of attack “pollutes” the input with imprints of the learned filters1 , which
gradually span a greater part of the feature map as more
convolutions are applied. Due to the highly correlated nature of the perturbation, the blurred downsampling can notcompletely eradicate the noise, but helps decrease the introduced pollution. On the contrary, for BU networks, thenoise is directly propagated down the network. Additionally, the blurred downsampling wired in the network architecture offers enhanced robustness against blurring attacks,
as the network encounters the input image at multiple scales
1 For imperfect delta function, this yields blurred versions of the filters.

during training, and is, thus, more resilient to resolutionchanges. Since anti-aliasing before downsampling is suggested to better preserve shift-invariance [51], we expected
our networks to also be more robust against the “Spatial”
attack [4]. However, no enhanced robustness is reported
for T D networks; a substantial difference in robustness isobserved for ResNet32, which could be due to the performance gap measured in Exp 1 between the T D and its BU
baseline. We also tested with the T Duni and T Drev variants of the ResNet32 architecture, with respective results
provided in the supplementary material.
To get a better insight on T D robustness, we introduce
the generated attacks to a single resolution branch of the
T D networks using the NIN-light architecture on MNISTand Fashion-MNIST, and NIN on CIFAR10. This is displayed in figure 5. We feed the extracted perturbations toeither the low, medium or high resolution input branch, as
illustrated in the model architecture in figure 2. For the simpler MNIST task, the medium-resolution input of the network is the most vulnerable, which is mainly attributed toTest accuracy loss (\%) - MNIST

100
80
60
40
20
0 0

Fashion-MNIST

100
80
60
40
20
0 0

CIFAR10_aug

100
80
60
40
20
0 0

correlated:
PointwiseAttack

uncorrelated:
AdditiveGaussianNoiseAttack

blurring:
GaussianBlurAttack

spatial:SpatialAttack

LeNetFC
LeNetFC_TD
NIN-light
NIN-light_TD

5

10

15

20

0

5

10

0.0

2.5

5.0

7.5

10.0

0

5

10

15

0

5

10

15

0

10

20
L2 bound30

LeNetFC
LeNetFC_TD
NIN-light
NIN-light_TD

5

10

15

0.0

2.5

5.0

7.5

10.0

0

0

5

10
15
L2 bound

20

0

5

10

ResNet32
ResNet32_TD
NIN
NIN_TD5

10 15
L2 bound

20

5

10
15
L2 bound20

Figure 6. Exp 2: Comparison of adversarial robustness considering different datasets, models and attacks. The x-axis of each figure
corresponds to the L2 distance between the original and the perturbed image and the y-axis is the introduced loss in test accuracy. A lower
curve suggests increased robustness. Green curves corresponding to T D are consistently underneath the respective red curves of the BU
networks, for most attacks. The T D networks are more robust against both correlated and uncorrelated noise attacks due to the coarseto-fine processing, suppressing high frequency information on earlier stages. Additionally, the blurred downsampling offers enhanced
robustness against blurring attacks. For spatial attacks, we see no increased robustness. (See supplementary material for additional results.)

the absence of information in the high frequency region of
the input’s spectrum. Moving to more challenging FashionMNIST and CIFAR10 tasks, the high frequency input becomes the easiest path for fooling the network. Please see
the supplementary material for additional results when perturbing two inputs simultaneously.

4.3. Exp 3: Explainability and localization(a) Grad-CAM heatmap visualizations. Grad-CAM [36]provides class-discriminative localization maps, based on
the feature maps of a convolutional layer, highlighting themost informative features for the classification task. Here,
we use the features of the last convolutional layer. The extracted heatmap is restored to the original image scale, thus
producing a coarser map in the case of the BU whose feature map size at the final layer is smaller. On the contrary,for T D the corresponding scale of the feature maps matches
the scale of the input, hence Grad-CAM outputs a finer map.
The Grad-CAM heatmaps corresponding to a BU and
T D network are provided in figure 7. These are obtained from various layers of a ResNet18 architecture [12]trained on the Imagenette dataset [18]. For further information about the setup please refer to supplementary material. “Layer 1” corresponds to the activation of the input to the first group of residual blocks, and “Layer 2” to“Layer 5” to the activations of the output of each of thesefour groups, each one corresponding to different spatial resolution. The visualizations demonstrate that T D follows an
opposite, coarse-to-fine path starting from a coarser representation and gradually enriching it with higher frequency
information. Hence, T D networks do not only mirror the
BU solely in the architectural design, but also in their learning process.
Additional heatmaps corresponding to correctly classified images, taken from the last convolutional layer of the
networks are visualized in figure 8. The figures depict thecoarse localization in BU versus the fine localization in
T D. We selected intentionally images with multiple objects. The T D networks recognize objects based on finegrained information: such as the spots on the dog, the crosson the church or boundary information of various objects.

BU

TD

Layer 1

Layer 2

Layer 3Layer 4Layer 5Figure 7. Exp 3.(a): Fine-to-coarse versus coarse-to-fine processing. We show Grad-CAM heatmaps for ResNet18 BU versus its respective
T D, trained on the Imagenette dataset [18] for a random validation image. Higher layer index means increased depth in the architecture:“Layer 1” corresponds to the activation of the input to the first group of residual blocks, and “Layer 2” to “Layer 5” to the activations of theoutput of each of these four groups, each one corresponding to different spatial resolution. Top: the BU network, employing fine-to-coarse
processing. Bottom: the respective T D network following the opposite path, starting with a holistic representation and gradually adding
higher frequency information in deeper layers.

Input

BU

TD

Figure 8. Exp 3.(a): Grad-CAM heatmaps corresponding to the last convolutional layer in the network. Top: The original input image,randomly selected from the validation set. Middle: Corresponding Grad-CAM heatmaps for the BU ResNet18. Bottom: Grad-CAM
heatmaps for the T D ResNet18. Contrary to the coarse output of the BU , the T D network outputs high frequency feature maps, based on
which the final classification is performed. T D recognized objects based on their fine-grained attributes: such as the spots on the dogs, orthe cross on the church, or shape information. (See supplementary material for additional results.)

0.8

NIN-light

Precision

0.6

MNIST

NIN-light

Recall

BU
TD

0.4
0.2

Fashion-MNIST

0.0
0.8
0.6
0.4
0.2
0.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Grad-CAM threshold

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Grad-CAM threshold

Figure 9. Exp 3.(b): Precision and recall for the MNIST and
Fashion-MNIST datasets using the NIN-light architecture. The
numbers are reported over four runs and we also plot standard deviations. For each run, models are trained from scratch and theset of TP (true positive), FP (false positive), FN (false negative) is
computed, between the Grad-CAM heatmaps and the segregatedobjects. The T D model has higher precision on both MNIST and
Fashion-MNIST due to more accurate object localization, while
having slightly lower recall than BU on the Fashion-MNIST.

(b) Weakly-supervised object localization. For a quantitative evaluation of the localization abilities of T D, weused the MNIST and Fashion-MNIST datasets and the NINlight model as a backbone architecture. Figure 9 shows
mean precision and recall scores for the T D and BU models over four runs. For each run models were trained fromscratch, then TP (true positive), FP (false positive), FN(false negative) values were computed between the GradCAM heatmaps and the thresholded objects, corresponding
to the test set of the considered task. We used a threshold
empirically set to t = 0.2. Based on the computed values
precision and recall scores were extracted and aggregatedover the four runs. For a fair comparison only the samples
correctly classified from both T D and BU were considered.
The T D models report higher precision for both tasks
considered, suggesting finer object localization. The lower
recall scores for the Fashion-MNIST is attributed to thehigher number of FN compared to the BU model. The
larger object sizes of the Fashion-MNIST task, along withthe coarse output of the BU model, being able to capture agreater extent of them, leads to fewer FN. On the contrary,
the T D models focus on finer aspects of the objects, which
are informative for the classification task. Considering
the fine-grained focus in the Grad-CAM outputs and thepotential for weakly-supervised object localization, ourproposed T D networks comprise a promising direction for
future research.

5. Discussion
The current work aims at providing a fresh perspectiveon the architecture of CNNs, which is currently taken for
granted. The coarse-to-fine pathway is biologically inspiredby how humans perceive visual information: first understanding the context and then filling in the salient details.
One downside of our proposed T D networks is that expanding dimensions at increased network depth leads tomemory and computational bottlenecks. This is due to thefeature map size being larger at higher depths. Moreover,for the same reason, adding fully-connected layers before
the output layer of the T D architectures leads to a vast increase in the number of model parameters. Hence, fully
convolutional networks are preferable. This increase inmemory is also more visible with large-scale datasets suchas ImageNet [2]. A simple workaround requiring no architectural adaptations would be to employ mixed-precision
training, which would decrease the memory requirements,
but would increase the computational complexity. Insteadof increasing the spatial resolution of the feature maps atlater depths, we could use patches of the input of limited
sizes. The selection of these informative patches could
be defined using the Grad-CAM heatmaps by selecting thehigh-activation areas of the heatmap, or considering selfattention mechanisms [45]. In addition to addressing the
aforementioned limitations, we find the weakly-supervised
setting to be a promising area of future research.

6. Conclusion
In the current work, we revisit the architecture of conventional CNNs, aiming at diverging from the manner inwhich resolution is typically processed in deep networks.
We propose novel network architectures which reverse theresolution processing of standard CNNs. The proposed
T D paradigm adopts a coarse-to-fine information processing pathway, starting from the low resolution information,
providing the visual context, and subsequently adding backthe high frequency information. We empirically demonstrate the applicability of our proposed T D architectures
when starting from a range of baseline architectures, andconsidering multiple visual recognition tasks. T D networks
exhibit enhanced robustness against certain types of adversarial attacks. This resistance to adversarial attacks is induced directly by the network design choices. Additionally,
the high spatial dimensions of the feature maps in the lastlayer significantly enhance the explainability of the model,
and demonstrate potential for weakly-supervised object localization tasks.

References[1] Shubhra Aich, Masaki Yamazaki, Yasuhiro Taniguchi, and
Ian Stavness. Multi-scale weight sharing network for image recognition. Pattern Recognition Letters, 131:348–354,2020.

[2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li,
and Li Fei-Fei. Imagenet: A large-scale hierarchical image
database. In Conference on Computer Vision and Pattern
Recognition, 2009.
[3] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map
prediction from a single image using a multi-scale deep network. In Advances in neural information processing systems,
pages 2366–2374, 2014.
[4] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig
Schmidt, and Aleksander Madry. Exploring the landscape of
spatial robustness. CoRR, 2017.
[5] Quanfu Fan, Chun-Fu Richard Chen, Hilde Kuehne, Marco
Pistoia, and David Cox. More is less: Learning efficient
video representations by big-little network and depthwise
temporal aggregation. In Advances in Neural Information
Processing Systems, pages 2261–2270, 2019.
[6] Yuchen Fan, Jiahui Yu, Ding Liu, and Thomas S Huang.
Scale-wise convolution for image restoration. Association
for the Advancement of Artificial Intelligence (AAAI, 2020.
[7] David J Field, Anthony Hayes, and Robert F Hess. Contour
integration by the human visual system: evidence for a local“association field”. Vision research, 33(2):173–193, 1993.
[8] Francois Fleuret and Donald Geman. Coarse-to-fine facedetection. International Journal of Computer Vision, 41(12):85–107, 2001.
[9] Robert Geirhos, Patricia Rubisch, Claudio Michaelis,
Matthias Bethge, Felix A Wichmann, and Wieland Brendel.
Imagenet-trained cnns are biased towards texture; increasing
shape bias improves accuracy and robustness. International
Conference on Learning Representations, 2019.
[10] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Spatial pyramid pooling in deep convolutional networks forvisual recognition. IEEE transactions on Pattern Analysis
and Machine Intelligence, 37(9):1904–1916, 2015.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern
Recognition, pages 770–778, 2016.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Identity mappings in deep residual networks. In European
conference on Computer Vision, pages 630–645, 2016.
[14] Jay Hegdé. Time course of visual perception: coarse-to-fineprocessing and beyond. Progress in neurobiology, 2008.
[15] Sina Honari, Jason Yosinski, Pascal Vincent, and Christopher Pal. Recombinator networks: Learning coarse-to-finefeature aggregation. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 5743–
5752, 2016.
[16] Yinlin Hu, Rui Song, and Yunsong Li. Efficient coarse-tofine patchmatch for large displacement optical flow. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 5704–5712, 2016.
[17] Sergey Ioffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. CoRR, 2015.

[18] FastAI Jeremy Howard. The imagenette dataset. https://github.com/fastai/imagenette.
[19] Jason Jo and Yoshua Bengio. Measuring the tendency of
cnns to learn surface statistical regularities. CoRR, 2017.
[20] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.
Progressive growing of gans for improved quality, stability,and variation. International Conference on Learning Representations, 2018.
[21] Tsung-Wei Ke, Michael Maire, and Stella X Yu. Multigrid
neural architectures. In Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition, pages 6665–
6673, 2017.
[22] I Kovács and B Julesz. A closed curve is much more than anincomplete one: effect of closure in figure-ground segmentation. PNAS, 1993.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 2012.
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick
Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998.
[25] Mengyan Li, Yuechuan Sun, Zhaoyu Zhang, and Jun Yu. A
coarse-to-fine face hallucination method by exploiting facial
prior knowledge. In International Conference on Image Processing (ICIP), pages 61–65, 2018.
[26] Xin Li and Fuxin Li. Adversarial examples detection in deep
networks with convolutional filter statistics. In Proceedingsof the IEEE International Conference on Computer Vision,
pages 5764–5772, 2017.
[27] Bin Liang, Hongcheng Li, Miaoqiang Su, Xirong Li, Wenchang Shi, and Xiaofeng Wang. Detecting adversarial examples in deep networks with adaptive noise reduction. CoRR,
2017.
[28] Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. International Conference on Learning Representations, 2014.
[29] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He,
Bharath Hariharan, and Serge Belongie. Feature pyramid
networks for object detection. In Proceedings of the IEEE
conference on Computer Vision and Pattern Recognition,
pages 2117–2125, 2017.
[30] Bruce D Lucas, Takeo Kanade, et al. An iterative image
registration technique with an application to stereo vision.
1981.
[31] Ravi Raju and Mikko Lipasti. Blurnet: Defense by filtering
the feature maps. CoRR, 2019.
[32] Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. CoRR, 2017.
[33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. Unet: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234–241,
2015.
[34] Hichem Sahbi. Coarse-to-fine deep kernel networks. In Proceedings of the IEEE International Conference on Computer
Vision Workshops, pages 1131–1139, 2017.

[35] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland
Brendel. Towards the first adversarially robust neural network model on mnist. CoRR, 2018.
[36] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das,
Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.
Grad-cam: Visual explanations from deep networks via
gradient-based localization. In Proceedings of the IEEE International conference on Computer Iision, 2017.
[37] Evan Shelhamer, Dequan Wang, and Trevor Darrell. Blurring the line between structure and learning to optimize andadapt receptive fields. CoRR, 2019.
[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
[39] Ivan Sosnovik, Michał Szmaja, and Arnold Smeulders.
Scale-equivariant steerable networks. International Conference on Learning Representations, 2020.
[40] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet,
Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent
Vanhoucke, and Andrew Rabinovich. Going deeper withconvolutions. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages 1–9, 2015.
[41] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan
Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2014.
[42] Johan Wagemans, James H Elder, Michael Kubovy,
Stephen E Palmer, Mary A Peterson, Manish Singh, andRüdiger von der Heydt. A century of gestalt psychology invisual perception: I. perceptual grouping and figure-groundorganization. Psychological bulletin, 2012.
[43] Haohan Wang, Xindi Wu, Pengcheng Yin, and Eric P Xing.
High frequency component helps explain the generalizationof convolutional neural networks. CoRR, 2019.
[44] Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, and Larry S
Davis. Liteeval: A coarse-to-fine framework for resource efficient video recognition. In Advances in Neural Information
Processing Systems, pages 7778–7787, 2019.
[45] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron
Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua
Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on
machine learning, 2015.
[46] Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang,
and Zheng Zhang. Scale-invariant convolutional neural networks. CoRR, 2014.
[47] Taojiannan Yang, Sijie Zhu, Shen Yan, Mi Zhang, Andrew
Willis, and Chen Chen. A closer look at network resolution
for efficient network design. CoRR, 2019.
[48] Chengxi Ye, Chinmaya Devaraj, Michael Maynord, Cornelia
Fermüller, and Yiannis Aloimonos. Evenly cascaded convolutional networks. In 2018 IEEE International Conference
on Big Data (Big Data), pages 4640–4647, 2018.
[49] Jianming Zhang, Zhe Lin, Jonathan Brandt, Xiaohui Shen,and Stan Sclaroff. Top-down neural attention by excitation
backprop. In European Conference on Computer Vision,
2016.

[50] Jie Zhang, Shiguang Shan, Meina Kan, and Xilin Chen.
Coarse-to-fine auto-encoder networks (cfan) for real-time
face alignment. In European conference on computer vision,
pages 1–16, 2014.
[51] Richard Zhang. Making convolutional networks shiftinvariant again. International Conference on Machine
Learning, 2019.
[52] Xilin Zhang, Li Zhaoping, Tiangang Zhou, and Fang Fang.
Neural Activities in V1 Create a Bottom-Up Saliency Map.
Neuron, 2012.
[53] Zhendong Zhang, Cheolkon Jung, and Xiaolong Liang. Adversarial defense by suppressing high-frequency components. CoRR, 2019.
[54] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva,
and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, 2016.