Visualizing the Impact of Feature Attribution Baselines

Path attribution methods are a gradient-based way
 of explaining deep models. These methods require choosing a
 hyperparameter known as the *baseline input*.
 What does this hyperparameter mean, and how important is it? In this article,
 we investigate these questions using image classification networks
 as a case study. We discuss several different ways to choose a baseline
 input and the assumptions that are implicit in each baseline.
 Although we focus here on path attribution methods, our discussion of baselines
 is closely connected with the concept of missingness in the feature space -
 a concept that is critical to interpretability research.

Introduction
------------

 If you are in the business of training neural networks,
 you might have heard of the integrated gradients method, which
 was introduced at
 [ICML](https://en.wikipedia.org/wiki/International_Conference_on_Machine_Learning) two years ago
 .
 The method computes which features are important
 to a neural network when making a prediction on a
 particular data point. This helps users
 understand which features their network relies on.
 Since its introduction,
 integrated gradients has been used to interpret
 networks trained on a variety of data types,
 including retinal fundus images
 and electrocardiogram recordings .

 If you’ve ever used integrated gradients,
 you know that you need to define a baseline input \(x’\) before
 using the method. Although the original paper discusses the need for a baseline
 and even proposes several different baselines for image data - including
 the constant black image and an image of random noise - there is
 little existing research about the impact of this baseline.
 Is integrated gradients sensitive to the
 hyperparameter choice? Why is the constant black image
 a “natural baseline” for image data? Are there any alternative choices?

 In this article, we will delve into how this hyperparameter choice arises,
 and why understanding it is important when you are doing model interpretation.
 As a case-study, we will focus on image classification models in order
 to visualize the effects of the baseline input. We will explore several
 notions of missingness, including both constant baselines and baselines
 defined by distributions. Finally, we will discuss different ways to compare
 baseline choices and talk about why quantitative evaluation
 remains a difficult problem.

Image Classification
--------------------

 We focus on image classification as a task, as it will allow us to visually
 plot integrated gradients attributions, and compare them with our intuition
 about which pixels we think should be important. We use the Inception V4 architecture
 , a convolutional
 neural network designed for the ImageNet dataset ,
 in which the task is to determine which class an image belongs to out of 1000 classes.
 On the ImageNet validation set, Inception V4 has a top-1 accuracy of over 80%.
 We download weights from TensorFlow-Slim ,
 and visualize the predictions of the network on four different images from the
 validation set.

 Right: The predicted logits of the network on the original image. The
 network correctly classifies all images with high confidence.
 Left: Pixel-wise attributions of the Inception V4 network using integrated gradients.
 You might notice that some attributions highlight pixels that do not seem important
 relative to the true class label.

 Although state of the art models perform well on unseen data,
 users may still be left wondering: *how* did the model figure
 out which object was in the image? There are a myriad of methods to
 interpret machine learning models, including methods to
 visualize and understand how the network represents inputs internally ,
 feature attribution methods that assign an importance score to each feature
 for a specific input ,
 and saliency methods that aim to highlight which regions of an image
 the model was looking at when making a decision
 .
 These categories are not mutually exclusive: for example, an attribution method can be
 visualized as a saliency method, and a saliency method can assign importance
 scores to each individual pixel. In this article, we will focus
 on the feature attribution method integrated gradients.

 Formally, given a target input \(x\) and a network function \(f\),
 feature attribution methods assign an importance score \(\phi\_i(f, x)\)
 to the \(i\)th feature value representing how much that feature
 adds or subtracts from the network output. A large positive or negative \(\phi\_i(f, x)\)
 indicates that feature strongly increases or decreases the network output
 \(f(x)\) respectively, while an importance score close to zero indicates that
 the feature in question did not influence \(f(x)\).

 In the same figure above, we visualize which pixels were most important to the network’s correct
 prediction using integrated gradients.
 The pixels in white indicate more important pixels. In order to plot
 attributions, we follow the same design choices as .
 That is, we plot the absolute value of the sum of feature attributions
 across the channel dimension, and cap feature attributions at the 99th percentile to avoid
 high-magnitude attributions dominating the color scheme.

A Better Understanding of Integrated Gradients
----------------------------------------------

 As you look through the attribution maps, you might find some of them
 unintuitive. Why does the attribution for “goldfinch” highlight the green background?
 Why doesn’t the attribution for “killer whale” highlight the black parts of the killer whale?
 To better understand this behavior, we need to explore how
 we generated feature attributions. Formally, integrated gradients
 defines the importance value for the \(i\)th feature value as follows:
 $$\phi\_i^{IG}(f, x, x’) = \overbrace{(x\_i - x’\_i)}^{\text{Difference from baseline}}
 \times \underbrace{\int\_{\alpha = 0}^ 1}\_{\text{From baseline to input…}}
 \overbrace{\frac{\delta f(x’ + \alpha (x - x’))}{\delta x\_i} d \alpha}^{\text{…accumulate local gradients}}
 $$
 where \(x\) is the current input,
 \(f\) is the model function and \(x’\) is some baseline input that is meant to represent
 “absence” of feature input. The subscript \(i\) is used
 to denote indexing into the \(i\)th feature.

 As the formula above states, integrated gradients gets importance scores
 by accumulating gradients on images interpolated between the baseline value and the current input.
 But why would doing this make sense? Recall that the gradient of
 a function represents the direction of maximum increase. The gradient
 is telling us which pixels have the steepest local slope with respect
 to the output. For this reason, the gradient of a network at the input
 was one of the earliest saliency methods.

 Unfortunately, there are many problems with using gradients to interpret
 deep neural networks .
 One specific issue is that neural networks are prone to a problem
 known as saturation: the gradients of input features may have small magnitudes around a
 sample even if the network depends heavily on those features. This can happen
 if the network function flattens after those features reach a certain magnitude.
 Intuitively, shifting the pixels in an image by a small amount typically
 doesn’t change what the network sees in the image. We can illustrate
 saturation by plotting the network output at all
 images between the baseline \(x’\) and the current image. The figure
 below displays that the network
 output for the correct class increases initially, but then quickly flattens.

 A plot of network outputs at \(x’ + \alpha (x - x’)\).
 Notice that the network output saturates the correct class
 at small values of \(\alpha\). By the time \(\alpha = 1\),
 the network output barely changes.

 What we really want to know is how our network got from
 predicting essentially nothing at \(x’\) to being
 completely saturated towards the correct output class at \(x\).
 Which pixels, when scaled along this path, most
 increased the network output for the correct class? This is
 exactly what the formula for integrated gradients gives us.

 By integrating over a path,
 integrated gradients avoids problems with local gradients being
 saturated. We can break the original equation
 down and visualize it in three separate parts: the interpolated image between
 the baseline image and the target image, the gradients at the interpolated
 image, and accumulating many such gradients over \(\alpha\).

 $$
 \int\_{\alpha’ = 0}^{\alpha} \underbrace{(x\_i - x’\_i) \times
 \frac{\delta f(\text{ }\overbrace{x’ + \alpha’ (x - x’)}^{\text{(1): Interpolated Image}}\text{ })}
 {\delta x\_i} d \alpha’}\_{\text{(2): Gradients at Interpolation}}
 = \overbrace{\phi\_i^{IG}(f, x, x’; \alpha)}^{\text{(3): Cumulative Gradients up to }\alpha}
 $$

 We visualize these three pieces of the formula below.Note that in practice, we use a discrete sum
 approximation of the integral with 500 linearly-spaced points between 0 and 1.

 Integrated gradients, visualized. In the line chart, the red line refers to
 equation (4) and the blue line refers to \(f(x) - f(x’)\). Notice how high magnitude gradients
 accumulate at small values of \(\alpha\).

 We have casually omitted one part of the formula: the fact
 that we multiply by a difference from a baseline. Although
 we won’t go into detail here, this term falls out because we
 care about the derivative of the network
 function \(f\) with respect to the path we are integrating over.
 That is, if we integrate over the
 straight-line between \(x’\) and \(x\), which
 we can represent as \(\gamma(\alpha) =
 x’ + \alpha(x - x’)\), then:
 $$
 \frac{\delta f(\gamma(\alpha))}{\delta \alpha} =
 \frac{\delta f(\gamma(\alpha))}{\delta \gamma(\alpha)} \times
 \frac{\delta \gamma(\alpha)}{\delta \alpha} =
 \frac{\delta f(x’ + \alpha’ (x - x’))}{\delta x\_i} \times (x\_i - x’\_i)
 $$
 The difference from baseline term is the derivative of the
 path function \(\gamma\) with respect to \(\alpha\).
  The theory behind integrated gradients is discussed
 in more detail in the original paper. In particular, the authors
 show that integrated gradients satisfies several desirable
 properties, including the completeness axiom:
 $$
 \textrm{Axiom 1: Completeness}\\
 \sum\_i \phi\_i^{IG}(f, x, x’) = f(x) - f(x’)
 $$
 Note that this theorem holds for any baseline \(x’\).
 Completeness is a desirable property because it states that the
 importance scores for each feature break down the output of the network:
 each importance score represents that feature’s individual contribution to
 the network output, and added when together, we recover the output value itself.
 Although it’s not essential to our discussion here, we can prove
 that integrated gradients satisfies this axiom using the
 [fundamental
 theorem of calculus for path integrals](https://en.wikipedia.org/wiki/Gradient_theorem). We leave a
 full discussion of all of the properties that integrated
 gradients satisfies to the original paper, since they hold
 independent of the choice of baseline.  The completeness
 axiom also provides a way to measure convergence.

 In practice, we can’t compute the exact value of the integral. Instead,
 we use a discrete sum approximation with \(k\) linearly-spaced points between
 0 and 1 for some value of \(k\). If we only chose 1 point to
 approximate the integral, that feels like too few. Is 10 enough? 100?
 Intuitively 1,000 may seem like enough, but can we be certain?
 As proposed in the original paper, we can use the completeness axiom
 as a sanity check on convergence: run integrated gradients with \(k\)
 points, measure \(|\sum\_i \phi\_i^{IG}(f, x, x’) - (f(x) - f(x’))|\),
 and if the difference is large, re-run with a larger \(k\)
 Of course, this brings up a new question: what is “large” in this context?
 One heuristic is to compare the difference with the magnitude of the
 output itself.
 .

 The line chart above plots the following equation in red:
 $$
 \underbrace{\sum\_i \phi\_i^{IG}(f, x, x’; \alpha)}\_{\text{(4): Sum of Cumulative Gradients up to }\alpha}
 $$
 That is, it sums all of the pixel attributions in the saliency map.
 This lets us compare to the blue line, which plots \(f(x) - f(x’)\).
 We can see that with 500 samples, we seem (at least intuitively) to
 have converged. But this article isn’t about how
 to get good convergence - it’s about baselines! In order
 to advance our understanding of the baseline, we will need a brief excursion
 into the world of game theory.

Game Theory and Missingness
---------------------------

 Integrated gradients is inspired by work
 from cooperative game theory, specifically the Aumann-Shapley value
 . In cooperative game theory,
 a non-atomic game is a construction used to model large-scale economic systems
 where there are enough participants that it is desirable to model them continuously.
 Aumann-Shapley values provide a theoretically grounded way to
 determine how much different groups of participants contribute to the system.

 In game theory, a notion of missingness is well-defined. Games are defined
 on coalitions - sets of participants - and for any specific coalition,
 a participant of the system can be in or out of that coalition. The fact
 that games can be evaluated on coalitions is the foundation of
 the Aumann-Shapley value. Intuitively, it computes how
 much value a group of participants adds to the game
 by computing how much the value of the game would increase
 if we added more of that group to any given coalition.

 Unfortunately, missingness is a more difficult notion when
 we are speaking about machine learning models. In order
 to evaluate how important the \(i\)th feature is, we
 want to be able to compute how much the output of
 the network would increase if we successively increased
 the “presence” of the \(i\)th feature. But what does this mean, exactly?
 In order to increase the presence of a feature, we would need to start
 with the feature being “missing” and have a way of interpolating
 between that missingness and its current, known value.

 Hopefully, this is sounding awfully familiar. Integrated gradients
 has a baseline input \(x’\) for exactly this reason: to model a
 feature being absent. But how should you choose
 \(x’\) in order to best represent this? It seems to be common practice
 to choose a baseline input \(x’\) to be the vector of
 all zeros. But consider the following scenario: you’ve learned a model
 on a healthcare dataset, and one of the features is blood sugar level.
 The model has correctly learned that excessively low levels of blood sugar,
 which correspond to hypoglycemia, is dangerous. Does
 a blood sugar level of \(0\) seem like a good choice to represent missingness?

 The point here is that fixed feature values may have unintended meaning.
 The problem compounds further when you consider the difference from
 baseline term \(x\_i - x’\_i\).
 For the sake of a thought experiment, suppose a patient had a blood sugar level of \(0\).
 To understand why our machine learning model thinks this patient
 is at high risk, you run integrated gradients on this data point with a
 baseline of the all-zeros vector. The blood sugar level of the patient would have \(0\) feature importance,
 because \(x\_i - x’\_i = 0\). This is despite the fact that
 a blood sugar level of \(0\) would be fatal!

 We find similar problems when we move to the image domain.
 If you use a constant black image as a baseline, integrated gradients will
 not highlight black pixels as important even if black pixels make up
 the object of interest. More generally, the method is blind to the color you use as a baseline, which
 we illustrate with the figure below. Note that this was acknowledged by the original
 authors in , and is in fact
 central to the definition of a baseline: we wouldn’t want integrated gradients
 to highlight missing features as important! But then how do we avoid
 giving zero importance to the baseline color?

 Mouse over the segmented image to choose a different color
 as a baseline input \(x’\). Notice that pixels
 of the baseline color are not highlighted as important,
 even if they make up part of the main object in the image.

Alternative Baseline Choices
----------------------------

 It’s clear that any constant color baseline will have this problem.
 Are there any alternatives? In this section, we
 compare four alternative choices for a baseline in the image domain.
 Before proceeding, it’s important to note that this article isn’t
 the first article to point out the difficulty of choosing a baselines.
 Several articles, including the original paper, discuss and compare
 several notions of “missingness”, both in the
 context of integrated gradients and more generally
 .
 Nonetheless, choosing the right baseline remains a challenge. Here we will
 present several choices for baselines: some based on existing literature,
 others inspired by the problems discussed above. The figure at the end
 of the section visualizes the four baselines presented here.

### The Maximum Distance Baseline

 If we are worried about constant baselines that are blind to the baseline
 color, can we explicitly construct a baseline that doesn’t suffer from this
 problem? One obvious way to construct such a baseline is to take the
 farthest image in L1 distance from the current image such that the
 baseline is still in the valid pixel range. This baseline, which
 we will refer to as the maximum distance baseline (denoted
 *max dist.* in the figure below),
 avoids the difference from baseline issue directly.

### The Blurred Baseline

 The issue with the maximum distance baseline is that it doesn’t
 really represent *missingness*. It actually contains a lot of
 information about the original image, which means we are no longer
 explaining our prediction relative to a lack of information. To better
 preserve the notion of missingness, we take inspiration from
 . In their paper,
 Fong and Vedaldi use a blurred version of the image as a
 domain-specific way to represent missing information. This baseline
 is attractive because it captures the notion of missingness in images
 in a very human intuitive way. In the figure below, this baseline is
 denoted *blur*. The figure lets you play with the smoothing constant
 used to define the baseline.

### The Uniform Baseline

 One potential drawback with the blurred baseline is that it is biased
 to highlight high-frequency information. Pixels that are very similar
 to their neighbors may get less importance than pixels that are very
 different than their neighbors, because the baseline is defined as a weighted
 average of a pixel and its neighbors. To overcome this, we can again take inspiration
 from both  and the original integrated
 gradients paper. Another way to define missingness is to simply sample a random
 uniform image in the valid pixel range and call that the baseline.
 We refer to this baseline as the *uniform* baseline in the figure below.

### The Gaussian Baseline

 Of course, the uniform distribution is not the only distribution we can
 draw random noise from. In their paper discussing the SmoothGrad (which we will
 touch on in the next section), Smilkov et al.
 make frequent use of a gaussian distribution centered on the current image with
 variance \(\sigma\). We can use the same distribution as a baseline for
 integrated gradients! In the figure below, this baseline is called the *gaussian*
 baseline. You can vary the standard deviation of the distribution \(\sigma\) using the slider.
 One thing to note here is that we truncate the gaussian baseline in the valid pixel
 range, which means that as \(\sigma\) approaches \(\infty\), the gaussian
 baseline approaches the uniform baseline.

 Comparing alternative baseline choices. For the blur and gaussian
 baselines, you can vary the parameter \(\sigma\), which refers
 to the width of the smoothing kernel and the standard deviation of
 noise respectively.

Averaging Over Multiple Baselines
---------------------------------

 You may have nagging doubts about those last two baselines, and you
 would be right to have them. A randomly generated baseline
 can suffer from the same blindness problem that a constant image can. If
 we draw a uniform random image as a baseline, there is a small chance
 that a baseline pixel will be very close to its corresponding input pixel
 in value. Those pixels will not be highlighted as important. The resulting
 saliency map may have artifacts due to the randomly drawn baseline. Is there
 any way we can fix this problem?

 Perhaps the most natural way to do so is to average over multiple
 different baselines, as discussed in
 .
 Although doing so may not be particularly natural for constant color images
 (which colors do you choose to average over and why?), it is a
 very natural notion for baselines drawn from distributions. Simply
 draw more samples from the same distribution and average the
 importance scores from each sample.

### Assuming a Distribution

 At this point, it’s worth connecting the idea of averaging over multiple
 baselines back to the original definition of integrated gradients. When
 we average over multiple baselines from the same distribution \(D\),
 we are attempting to use the distribution itself as our baseline.
 We use the distribution to define the notion of missingness:
 if we don’t know a pixel value, we don’t assume its value to be 0 - instead
 we assume that it has some underlying distribution \(D\). Formally, given
 a baseline distribution \(D\), we integrate over all possible baselines
 \(x’ \in D\) weighted by the density function \(p\_D\):
 $$ \phi\_i(f, x) = \underbrace{\int\_{x’}}\_{\text{Integrate over baselines…}} \bigg( \overbrace{\phi\_i^{IG}(f, x, x’
 )}^{\text{integrated gradients
 with baseline } x’
 } \times \underbrace{p\_D(x’) dx’}\_{\text{…and weight by the density}} \bigg)
 $$

 In terms of missingness, assuming a distribution might intuitively feel
 like a more reasonable assumption to make than assuming a constant value.
 But this doesn’t quite solve the issue: instead of having to choose a baseline
 \(x’\), now we have to choose a baseline distribution \(D\). Have we simply
 postponed the problem? We will discuss one theoretically motivated
 way to choose \(D\) in an upcoming section, but before we do, we’ll take
 a brief aside to talk about how we compute the formula above in practice,
 and a connection to an existing method that arises as a result.

### Expectations, and Connections to SmoothGrad

 Now that we’ve introduced a second integral into our formula,
 we need to do a second discrete sum to approximate it, which
 requires an additional hyperparameter: the number of baselines to sample.
 In , Erion et al. make the
 observation that both integrals can be thought of as expectations:
 the first integral as an expectation over \(D\), and the second integral
 as an expectation over the path between \(x’\) and \(x\). This formulation,
 called *expected gradients*, is defined formally as:
 $$ \phi\_i^{EG}(f, x; D) = \underbrace{\mathop{\mathbb{E}}\_{x’ \sim D, \alpha \sim U(0, 1)}}\_
 {\text{Expectation over \(D\) and the path…}}
 \bigg[ \overbrace{(x\_i - x’\_i) \times
 \frac{\delta f(x’ + \alpha (x - x’))}{\delta x\_i}}^{\text{…of the
 importance of the } i\text{th pixel}} \bigg]
 $$

 Expected gradients and integrated gradients belong to a family of methods
 known as “path attribution methods” because they integrate gradients
 over one or more paths between two valid inputs.
 Both expected gradients and integrated gradients use straight-line paths,
 but one can integrate over paths that are not straight as well. This is discussed
 in more detail in the original paper.  To compute expected gradients in
 practice, we use the following formula:
 $$
 \hat{\phi}\_i^{EG}(f, x; D) = \frac{1}{k} \sum\_{j=1}^k (x\_i - x’^j\_i) \times
 \frac{\delta f(x’^j + \alpha^{j} (x - x’^j))}{\delta x\_i}
 $$
 where \(x’^j\) is the \(j\)th sample from \(D\) and
 \(\alpha^j\) is the \(j\)th sample from the uniform distribution between
 0 and 1. Now suppose that we use the gaussian baseline with variance
 \(\sigma^2\). Then we can re-write the formula for expected gradients as follows:

 $$
 \hat{\phi}\_i^{EG}(f, x; N(x, \sigma^2 I))
 = \frac{1}{k} \sum\_{j=1}^k
 \epsilon\_{\sigma}^{j} \times
 \frac{\delta f(x + (1 - \alpha^j)\epsilon\_{\sigma}^{j})}{\delta x\_i}
 $$

 where \(\epsilon\_{\sigma}\ \sim N(\bar{0}, \sigma^2 I)\)

 To see how we arrived
 at the above formula, first observe that
 $$
 \begin{aligned}
 x’ \sim N(x, \sigma^2 I) &= x + \epsilon\_{\sigma} \\
 x’- x &= \epsilon\_{\sigma} \\
 \end{aligned}
 $$
 by definition of the gaussian baseline. Now we have:
 $$
 \begin{aligned}
 x’ + \alpha(x - x’) &= \\
 x + \epsilon\_{\sigma} + \alpha(x - (x + \epsilon\_{\sigma})) &= \\
 x + (1 - \alpha)\epsilon\_{\sigma}
 \end{aligned}
 $$
 The above formula simply substitues the last line
 of each equation block back into the formula.
 .

 This looks awfully familiar to an existing method called SmoothGrad
 . If we use the (gradients \(\times\) input image)
 variant of SmoothGrad  SmoothGrad is
 was a method designed to sharpen saliency maps and was meant to be run
 on top of an existing saliency method. The idea is simple:
 instead of running a saliency method once on an image, first
 add some gaussian noise to an image, then run the saliency method.
 Do this several times with different draws of gaussian noise, then
 average the results. Multipying the gradients by the input and using that as a saliency map
 is discussed in more detail in the original SmoothGrad paper.,
 then we have the following formula:
 $$
 \phi\_i^{SG}(f, x; N(\bar{0}, \sigma^2 I))
 = \frac{1}{k} \sum\_{j=1}^k
 (x + \epsilon\_{\sigma}^{j}) \times
 \frac{\delta f(x + \epsilon\_{\sigma}^{j})}{\delta x\_i}
 $$

 We can see that SmoothGrad and expected gradients with a
 gaussian baseline are quite similar, with two key differences:
 SmoothGrad multiplies the gradient by \(x + \epsilon\_{\sigma}\) while expected
 gradients multiplies by just \(\epsilon\_{\sigma}\), and while expected
 gradients samples uniformly along the path, SmoothGrad always
 samples the endpoint \(\alpha = 0\).

 Can this connection help us understand why SmoothGrad creates
 smooth-looking saliency maps? When we assume the above gaussian distribution as our baseline, we are
 assuming that each of our pixel values is drawn from a
 gaussian *independently* of the other pixel values. But we know
 this is far from true: in images, there is a rich correlation structure
 between nearby pixels. Once your network knows the value of a pixel,
 it doesn’t really need to use its immediate neighbors because
 it’s likely that those immediate neighbors have very similar intensities.

 Assuming each pixel is drawn from an independent gaussian
 breaks this correlation structure. It means that expected gradients
 tabulates the importance of each pixel independently of
 the other pixel values. The generated saliency maps
 will be less noisy and better highlight the object of interest
 because we are no longer allowing the network to rely
 on only pixel in a group of correlated pixels. This may be
 why SmoothGrad is smooth: because it is implicitly assuming
 independence among pixels. In the figure below, you can compare
 integrated gradients with a single randomly drawn baseline
 to expected gradients sampled over a distribution. For
 the gaussian baseline, you can also toggle the SmoothGrad
 option to use the SmoothGrad formula above. For all figures,
 \(k=500\).

 The difference between a single baseline and multiple
 baselines from the same distribution. Use the
 “Multi-Reference” button to toggle between the two. For the gaussian
 baseline, you can also toggle the “Smooth Grad” button
 to toggle between expected gradients and SmoothGrad
 with gradients \* inputs.

### Using the Training Distribution

 Is it really reasonable to assume independence among
 pixels while generating saliency maps? In supervised learning,
 we make the assumption that the data is drawn
 from some distribution \(D\_{\text{data}}\). This assumption that the training and testing data
 share a common, underlying distribution is what allows us to
 do supervised learning and make claims about generalizability. Given
 this assumption, we don’t need to
 model missingness using a gaussian or a uniform distribution:
 we can use \(D\_{\text{data}}\) to model missingness directly.

 The only problem is that we do not have access to the underlying distribution.
 But because this is a supervised learning task, we do have access to many
 independent draws from the underlying distribution: the training data!
 We can simply use samples from the training data as random draws
 from \(D\_{\text{data}}\). This brings us to the variant
 of expected gradients used in ,
 which we again visualize in three parts:
 $$
 \frac{1}{k} \sum\_{j=1}^k
 \underbrace{(x\_i - x’^j\_i) \times
 \frac{\delta f(\text{ }
 \overbrace{x’^j + \alpha^{j} (x - x’^j)}^{\text{(1): Interpolated Image}}
 \text{ })}{\delta x\_i}}\_{\text{ (2): Gradients at Interpolation}}
 = \overbrace{\hat{\phi\_i}^{EG}(f, x, k; D\_{\text{data}})}
 ^{\text{(3): Cumulative Gradients up to }\alpha}
 $$

 A visual representation of expected gradients. Instead of taking contributions
 from a single path, expected gradients averages contributions from
 all paths defined by the underlying data distribution. Note that
 this figure only displays every 10th sample to avoid loading many images.

 In (4) we again plot the sum of the importance scores over pixels. As mentioned
 in the original integrated gradients paper, all path methods, including expected
 gradients, satisfy the completeness axiom. We can definitely see that
 completeness is harder to satisfy when we integrate over both a path
 and a distribution: that is, with the same number
 of samples, expected gradients doesn’t converge as quickly as
 integrated gradients does. Whether or not this is an acceptable price to
 pay to avoid color-blindness in attributions seems subjective.

Comparing Saliency Methods
--------------------------

 So we now have many different choices for a baseline. How do we choose
 which one we should use? The different choices of distributions and constant
 baselines have different theoretical motivations and practical concerns.
 Do we have any way of comparing the different baselines? In this section,
 we will touch on several different ideas about how to compare
 interpretability methods. This section is not meant to be a comprehensive overview
 of all of the existing evaluation metrics, but is instead meant to
 emphasize that evaluating interpretability methods remains a difficult problem.

### The Dangers of Qualitative Assessment

 One naive way to evaluate our baselines is to look at the saliency maps
 they produce and see which ones best highlight the object in the image.
 From our earlier figures, it does seem like using \(D\_{\text{data}}\) produces
 reasonable results, as does using a gaussian baseline or the blurred baseline.
 But is visual inspection really a good way judge our baselines? For one thing,
 we’ve only presented four images from the test set here. We would need to
 conduct user studies on a much larger scale with more images from the test
 set to be confident in our results. But even with large-scale user studies,
 qualitative assessment of saliency maps has other drawbacks.

 When we rely on qualitative assessment, we are assuming that humans
 know what an “accurate” saliency map is. When we look at saliency maps
 on data like ImageNet, we often check whether or not the saliency map
 highlights the object that we see as representing the true class in the image.
 We make an assumption between the data and the label, and then further assume
 that a good saliency map should reflect that assumption. But doing so
 has no real justification. Consider the figure below, which compares
 two saliency methods on a network that gets above 99% accuracy
 on (an altered version of) MNIST.
 The first saliency method is just an edge detector plus gaussian smoothing,
 while the second saliency method is expected gradients using the training
 data as a distribution. Edge detection better reflects what we humans
 think is the relationship between the image and the label.

 Qualitative assessment can be dangerous because we rely
 on our human knowledge of the relationship between
 the data and the labels, and then we assume
 that an accurate model has learned that very relationship.

 Unfortunately, the edge detection method here does not highlight
 what the network has learned. This dataset is a variant of
 decoy MNIST, in which the top left corner of the image has
 been altered to directly encode the image’s class
 . That is, the intensity
 of the top left corner of each image has been altered to
 be \(255 \times \frac{y}{9} \) where \(y\) is the class
 the image belongs to. We can verify by removing this
 patch in the test set that the network heavily relies on it to make
 predictions, which is what the expected gradients saliency maps show.

 This is obviously a contrived example. Nonetheless, the fact that
 visual assessment is not necessarily a useful way to evaluate
 saliency maps and attribution methods has been extensively
 discussed in recent literature, with many proposed qualitative
 tests as replacements
 .
 At the heart of the issue is that we don’t have ground truth explanations:
 we are trying to evaluate which methods best explain our network without
 actually knowing what our networks are doing.

### Top K Ablation Tests

 One simple way to evaluate the importance scores that
 expected/integrated gradients produces is to see whether
 ablating the top k features as ranked by their importance
 decreases the predicted output logit. In the figure below, we
 ablate either by mean-imputation or by replacing each pixel
 by its gaussian-blurred counter-part (*Mean Top K* and *Blur Top K* in the plot). We generate pixel importances
 for 1000 different correctly classified test-set images using each
 of the baselines proposed above
 For the blur baseline and the blur
 ablation test, we use \(\sigma = 20\).
 For the gaussian baseline, we use \(\sigma = 1\). These choices
 are somewhat arbitrary - a more comprehensive evaluation
 would compare across many values of \(\sigma\).
 . As a
 control, we also include ranking features randomly
 (*Random Noise* in the plot).

 We plot, as a fraction of the original logit, the output logit
 of the network at the true class. That is, suppose the original
 image is a goldfinch and the network predicts the goldfinch class correctly
 with 95% confidence. If the confidence of class goldfinch drops
 to 60% after ablating the top 10% of pixels as ranked by
 feature importance, then we plot a curve that goes through
 the points \((0.0, 0.95)\) and \((0.1, 0.6)\). The baseline choice
 that best highlights which pixels the network
 should exhibit the fastest drop in logit magnitude, because
 it highlights the pixels that most increase the confidence of the network.
 That is, the lower the curve, the better the baseline.

### Mass Center Ablation Tests

 One problem with ablating the top k features in an image
 is related to an issue we already brought up: feature correlation.
 No matter how we ablate a pixel, that pixel’s neighbors
 provide a lot of information about the pixel’s original value.
 With this in mind, one could argue that progressively ablating
 pixels one by one is a rather meaningless thing to do. Can
 we instead perform ablations with feature correlation in mind?

 One straightforward way to do this is simply compute the
 center of mass
 of the saliency map, and ablate a boxed region centered on
 the center of mass. This tests whether or not the saliency map
 is generally highlighting an important region in the image. We plot
 replacing the boxed region around the saliency map using mean-imputation
 and blurring below as well (*Mean Center* and *Blur Center*, respectively).
 As a control, we compare against a saliency map generated from random gaussian
 noise (*Random Noise* in the plot).

 A variety of ablation tests on a variety of baselines.
 Using the training distribution and using the uniform distribution
 outperform most other methods on the top k ablation tests. The
 blur baseline inspired by
 does equally well on the blur top-k test. All methods
 perform similarly on the mass center ablation tests. Mouse
 over the legend to highlight a single curve.

 The ablation tests seem to indicate some interesting trends.
 All methods do similarly on the mass center ablation tests, and
 only slightly better than random noise. This may be because the
 object of interest generally lies in the center of the image - it
 isn’t hard for random noise to be centered at the image. In contrast,
 using the training data or a uniform distribution seems to do quite well
 on the top-k ablation tests. Interestingly, the blur baseline
 inspired by  also
 does quite well on the top k baseline tests, especially when
 we ablate pixels by blurring them! Would the uniform
 baseline do better if you ablate the image with uniform random noise?
 Perhaps the training distribution baseline would do even better if you ablate an image
 by progressively replacing it with a different image. We leave
 these experiments as future work, as there is a more pressing question
 we need to discuss.

### The Pitfalls of Ablation Tests

 Can we really trust the ablations tests presented above? We ran each method with 500 samples.
 Constant baselines tend to not need as many samples
 to converge as baselines over distributions. How do we fairly compare between baselines that have
 different computational costs? Valuable but computationally-intensive future work would be
 comparing not only across baselines but also across number of samples drawn,
 and for the blur and gaussian baselines, the parameter \(\sigma\).
 As mentioned above, we have defined many notions of missingness other than
 mean-imputation or blurring: more extensive comparisons would also compare
 all of our baselines across all of the corresponding notions of missing data.

 But even with all of these added comparisons, do ablation
 tests really provide a well-founded metric to judge attribution methods?
 The authors of  argue
 against ablation tests. They point out that once we artificially ablate
 pixels an image, we have created inputs that do not come from
 the original data distribution. Our trained model has never seen such
 inputs. Why should we expect to extract any reasonable information
 from evaluating our model on them?

 On the other hand, integrated gradients and expected gradients
 rely on presenting interpolated images to your model, and unless
 you make some strange convexity assumption, those interpolated images
 don’t belong to the original training distribution either.
 In general, whether or not users should present
 their models with inputs that don’t belong to the original training distribution
 is a subject of ongoing debate
 . Nonetheless,
 the point raised in  is still an
 important one: “it is unclear whether the degradation in model
 performance comes from the distribution shift or because the
 features that were removed are truly informative.”

### Alternative Evaluation Metrics

 So what about other evaluation metrics proposed in recent literature? In
 , Hooker et al. propose a variant of
 an ablation test where we first ablate pixels in the training and
 test sets. Then, we re-train a model on the ablated data and measure
 by how much the test-set performance degrades. This approach has the advantage
 of better capturing whether or not the saliency method
 highlights the pixels that are most important for predicting the output class.
 Unfortunately, it has the drawback of needing to re-train the model several
 times. This metric may also get confused by feature correlation.

 Consider the following scenario: our dataset has two features
 that are highly correlated. We train a model which learns to only
 use the first feature, and completely ignore the second feature.
 A feature attribution method might accurately reveal what the model is doing:
 it’s only using the first feature. We could ablate that feature in the dataset,
 re-train the model and get similar performance because similar information
 is stored in the second feature. We might conclude that our feature
 attribution method is lousy - is it? This problem fits into a larger discussion
 about whether or not your attribution method
 should be “true to the model” or “true to the data”
 which has been discussed in several recent articles
 .

 In , the authors propose several
 sanity checks that saliency methods should pass. One is the “Model Parameter
 Randomization Test”. Essentially, it states that a feature attribution
 method should produce different attributions when evaluated on a trained
 model (assumedly a trained model that performs well) and a randomly initialized
 model. This metric is intuitive: if a feature attribution method produces
 similar attributions for random and trained models, is the feature
 attribution really using information from the model? It might just
 be relying entirely on information from the input image.

 But consider the following figure, which is another (modified) version
 of MNIST. We’ve generated expected gradients attributions using the training
 distribution as a baseline for two different networks. One of the networks
 is a trained model that gets over 99% accuracy on the test set. The other
 network is a randomly initialized model that doesn’t do better than random guessing.
 Should we now conclude that expected gradients is an unreliable method?

 A comparison of two network’s saliency maps using expected gradients. One
 network has randomly initialized weights, the other gets >99% accuracy
 on the test set.

 Of course, we modified MNIST in this example specifically so that expected gradients
 attributions of an accurate model would look exactly like those of a randomly initialized model.
 The way we did this is similar to the decoy MNIST dataset, except instead of the top left
 corner encoding the class label, we randomly scattered noise througout each training and
 test image where the intensity of the noise encodes the true class label. Generally,
 you would run these kinds of saliency method sanity checks on un-modified data.

 But the truth is, even for natural images, we don’t actually
 know what an accurate model’s saliency maps should look like.
 Different architectures trained on ImageNet can all get good performance
 and have very different saliency maps. Can we really say that
 trained models should have saliency maps that don’t look like
 saliency maps generated on randomly initialized models? That isn’t
 to say that the model randomization test doesn’t have merit: it
 does reveal interesting things about what saliency methods are are doing.
 It just doesn’t tell the whole story.

 As we mentioned above, there’s a variety of metrics that have been proposed to evaluate
 interpretability methods. There are many metrics we do not explicitly discuss here
 .
 Each proposed metric comes with their various pros and cons.
 In general, evaluating supervised models is somewhat straightforward: we set aside a
 test-set and use it to evaluate how well our model performs on unseen data. Evaluating explanations is hard:
 we don’t know what our model is doing and have no ground truth to compare
 against.

Conclusion
----------

 So what should be done? We have many baselines and
 no conclusion about which one is the “best.” Although
 we don’t provide extensive quantitative results
 comparing each baseline, we do provide a foundation
 for understanding them further. At the heart of
 each baseline is an assumption about missingness
 in our model and the distribution of our data. In this article,
 we shed light on some of those assumptions, and their impact
 on the corresponding path attribution. We lay
 groundwork for future discussion about baselines in the
 context of path attributions, and more generally about
 the relationship between representations of missingness
 and how we explain machine learning models.

 A side-by-side comparison of integrated gradients
 using a black baseline
 and expected gradients using the training data
 as a baseline.

Related Methods
---------------

 This work focuses on a specific interpretability method: integrated gradients
 and its extension, expected gradients. We refer to these
 methods as path attribution methods because they integrate
 importances over a path. However, path attribution methods
 represent only a tiny fraction of existing interpretability methods. We focus
 on them here both because they are amenable to interesting visualizations,
 and because they provide a springboard for talking about missingness.
 We briefly cited several other methods at the beginning of this article.
 Many of those methods use some notion of baseline and have contributed to
 the discussion surrounding baseline choices.

 In , Fong and Vedaldi propose
 a model-agnostic method to explain neural networks that is based
 on learning the minimal deletion to an image that changes the model
 prediction. In section 4, their work contains an extended discussion on
 how to represent deletions: that is, how to represent missing pixels. They
 argue that one natural way to delete pixels in an image is to blur them.
 This discussion inspired the blurred baseline that we presented in our article.
 They also discuss how noise can be used to represent missingness, which
 was part of the inspiration for our uniform and gaussian noise baselines.

 In , Shrikumar et al.
 propose a feature attribution method called deepLIFT. It assigns
 importance scores to features by propagating scores from the output
 of the model back to the input. Similar to integrated gradients,
 deepLIFT also defines importance scores relative to a baseline, which
 they call the “reference”. Their paper has an extended discussion on
 why explaining relative to a baseline is meaningful. They also discuss
 a few different baselines, including “using a blurred version of the original
 image”.

 The list of other related methods that we didn’t discuss
 in this article goes on: SHAP and DeepSHAP
 ,
 layer-wise relevance propagation ,
 LIME ,
 RISE  and
 Grad-CAM
 among others. Many methods for explaining machine learning models
 define some notion of baseline or missingness,
 because missingness and explanations are closely related. When we explain
 a model, we often want to know which features, when missing, would most
 change model output. But in order to do so, we need to define
 what missing means because most machine learning models cannot
 handle arbitrary patterns of missing inputs. This article
 does not discuss all of the nuances presented alongside
 each existing method, but it is important to note that these methods were
 points of inspiration for a larger discussion about missingness.

Pascal Sturmfels
Scott Lundberg
Su-In Lee

Exploring the baseline input hyperparameter, and how it impacts interpretations of neural network behavior.

Axiomatic attribution for deep networks

Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy

Ensembling convolutional and long short-term memory networks for electrocardiogram arrhythmia detection

Inception-v4, inception-resnet and the impact of residual connections on learning

Imagenet: A large-scale hierarchical image database

Tensorflow-slim image classification model library

The Building Blocks of Interpretability

Feature Visualization

A unified approach to interpreting model predictions

Layer-wise relevance propagation for neural networks with local renormalization layers

Learning important features through propagating activation differences

Smoothgrad: removing noise by adding noise

Understanding the difficulty of training deep feedforward neural networks

Gradients of counterfactuals

Values of non-atomic games

Learning Explainable Models Using Attribution Priors


Computing Receptive Fields of Convolutional Neural Networks

While deep neural networks have overwhelmingly established state-of-the-art
 results in many artificial intelligence problems, they can still be
 difficult to develop and debug.
 Recent research on deep learning understanding has focused on
 feature visualization ,
 theoretical guarantees ,
 model interpretability ,
 and generalization .

 In this work, we analyze deep neural networks from a complementary
 perspective, focusing on convolutional models.
 We are interested in understanding the extent to
 which input signals may affect output features, and mapping
 features at any part of the network to the region in the input that
 produces them. The key parameter to associate an output feature to an input
 region is the *receptive field* of the convolutional network, which is
 defined as the size of the region in the input that produces the feature.

 As our first contribution, we
 present a mathematical derivation and an efficient algorithm to compute
 receptive fields of modern convolutional neural networks.
 Previous work
 discussed receptive field
 computation for simple convolutional
 networks where there is a single path from the input to the output,
 providing recurrence equations that apply to this case.
 In this work, we revisit these derivations to obtain a closed-form
 expression for receptive field computation in the single-path case.
 Furthermore, we extend receptive field computation to modern convolutional
 networks where there may be multiple paths from the input to the output.
 To the best of our knowledge, this is the first exposition of receptive
 field computation for such recent convolutional architectures.

 Today, receptive field computations are needed in a variety of applications. For example,
 for the computer vision task of object detection, it is important
 to represent objects at multiple scales in order to recognize small and large instances;
 understanding a convolutional feature’s span is often required for that goal
 (e.g., if the receptive field of the network is small, it may not be able to recognize large objects).
 However, these computations are often done by hand, which is both tedious and error-prone.
 This is because there are no libraries to compute these parameters automatically.
 As our second contribution, we fill the void by introducing an
 [open-source library](https://github.com/google-research/receptive_field)
 which handily performs the computations described here. The library is integrated into the Tensorflow codebase and
 can be easily employed to analyze a variety of models,
 as presented in this article.

 We expect these derivations and open-source code to improve the understanding of complex deep learning models,
 leading to more productive machine learning research.

Overview of the article
-----------------------

 We consider fully-convolutional neural networks, and derive their receptive
 field size and receptive field locations for output features with respect to the
 input signal.
 While the derivations presented here are general enough for any type of signal used at the input of convolutional
 neural networks, we use images as a running example, referring to modern computer vision architectures when
 appropriate.

 First, we derive closed-form expressions when the network has a
 single path from input to output (as in
 AlexNet
 or
 VGG ). Then, we discuss the
 more general case of arbitrary computation graphs with multiple paths from the
 input to the output (as in
 ResNet
 or
 Inception ). We consider
 potential alignment issues that arise in this context, and explain
 an algorithm to compute the receptive field size and locations.

 Finally, we analyze the receptive fields of modern convolutional neural networks, showcasing results obtained
 using our open-source library.

Problem setup
-------------

 Consider a fully-convolutional network (FCN) with \(L\) layers, \(l = 1,2,\ldots
  ,L\). Define feature map \(f\_l \in R^{h\_l\times w\_l\times d\_l}\) to denote the
 output of the \(l\)-th layer, with height \(h\_l\), width \(w\_l\) and depth
 \(d\_l\). We denote the input image by \(f\_0\). The final output feature map
 corresponds to \(f\_{L}\).

 To simplify the presentation, the derivations presented in this document consider
 \(1\)-dimensional input signals and feature maps. For higher-dimensional signals
 (e.g., \(2\)D images), the
 derivations can be applied to each dimension independently. Similarly, the figures
 depict \(1\)-dimensional depth, since this does not affect the receptive field computation.

Each layer \(l\)’s spatial configuration is parameterized by 4 variables, as illustrated in the following figure:

* \(k\_l\): kernel size (positive integer)
* \(s\_l\): stride (positive integer)
* \(p\_l\): padding applied to the left side of the input feature map
 (non-negative integer)

 A more general definition of padding may also be considered: negative
 padding, interpreted as cropping, can be used in our derivations
 without any changes. In order to make the article more concise, our
 presentation focuses solely on non-negative padding.
* \(q\_l\): padding applied to the right side of the input feature map
 (non-negative integer)

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 2
Left Padding (pl): 1
Right Padding (ql): 1
Stride (sl): 3

fl
fl-1
kl
kl
kl
sl
pl
ql

 We consider layers whose output features depend locally on input features:
 e.g., convolution, pooling, or elementwise operations such as non-linearities,
 addition and filter concatenation. These are commonly used in state-of-the-art
 networks. We define elementwise operations to
 have a “kernel size” of \(1\), since each output feature depends on a single
 location of the input feature maps.

 Our notation is further illustrated with the simple
 network below. In this case, \(L=4\) and the model consists of a
 convolution, followed by ReLU, a second convolution and max-pooling.

 We adopt the convention where the first output feature for each layer is
 computed by placing the kernel on the left-most position of the input,
 including padding. This convention is adopted by all major deep learning
 libraries.

![](https://distill.pub/2018/feature-wise-transformations/images/pointer.svg)

 Sorry, your browser does not support inline SVG.

f0
f1
f2
f3
f4

**Convolution**

 Kernel Size (k1): 1

 Padding (p1, q1): 0

 Stride (s1): 1

Invalid setup! Decrease kernel
size or increase padding.

**ReLU**

 Kernel Size (k2): 1

 Padding (p2, q2): 0

 Stride (s2): 1

**Convolution**

 Kernel Size (k3): 1

 Padding (p3, q3): 0

 Stride (s3): 1

Invalid setup! Decrease kernel
size or increase padding.

**Max Pooling**

 Kernel Size (k4): 1

 Padding (p4, q4): 0

 Stride (s4): 1

Invalid setup! Decrease kernel
size or increase padding.

Single-path networks
--------------------

 In this section, we compute recurrence and closed-form expressions for
 fully-convolutional networks with a single path from input to output
 (e.g.,
 AlexNet
 or
 VGG ).

### Computing receptive field size

 Define \(r\_l\) as the receptive field size of
 the final output feature map \(f\_{L}\), with respect to feature map \(f\_l\). In
 other words, \(r\_l\) corresponds to the number of features in feature map
 \(f\_l\) which contribute to generate one feature in \(f\_{L}\). Note
 that \(r\_{L}=1\).

 As a simple example, consider layer \(L\), which takes features \(f\_{L-1}\) as
 input, and generates \(f\_{L}\) as output. Here is an illustration:

 Sorry, your browser does not support inline SVG.

Kernel Size (kL): 2
Padding (pL, qL): 0
Stride (sL): 3

kL
kL
fL-1
fL

 It is easy to see that \(k\_{L}\)
 features from \(f\_{L-1}\) can influence one feature from \(f\_{L}\), since each
 feature from \(f\_{L}\) is directly connected to \(k\_{L}\) features from
 \(f\_{L-1}\). So, \(r\_{L-1} = k\_{L}\).

 Now, consider the more general case where we know \(r\_{l}\) and want to compute
 \(r\_{l-1}\). Each feature \(f\_{l}\) is connected to \(k\_{l}\) features from
 \(f\_{l-1}\).

 First, consider the situation where \(k\_l=1\): in this case, the \(r\_{l}\)
 features in \(f\_{l}\) will cover \(r\_{l-1}=s\_l\cdot r\_{l} - (s\_l - 1)\) features
 in in \(f\_{l-1}\). This is illustrated in the figure below, where \(r\_{l}=2\)
 (highlighted in red). The first term \(s\_l \cdot r\_{l}\) (green) covers the
 entire region where the
 features come from, but it will cover \(s\_l - 1\) too many features (purple),
 which is why it needs to be deducted.

 As in the illustration below, note that, in some cases, the receptive
 field region may contain “holes”, i.e., some of the input features may be
 unused for a given layer.

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 1
Padding (pl, ql): 0
Stride (sl): 3

sl⋅rl
sl-1
fl-1
fl
rl=2

 For the case where \(k\_l > 1\), we just need to add \(k\_l-1\) features, which
 will cover those from the left and the right of the region. For example, if we
 use a kernel size of \(5\) (\(k\_l=5\)), there would be \(2\) extra features used
 on each side, adding \(4\) in total. If \(k\_l\) is even, this works as well,
 since the left and right padding will add to \(k\_l-1\).

 Due to border effects, note that the size of the region in the original
 image which is used to compute each output feature may be different. This
 happens if padding is used, in which case the receptive field for
 border features includes the padded region. Later in the article, we
 discuss how to compute the receptive field region for each feature,
 which can be used to determine exactly which image pixels are used for
 each output feature.

 Sorry, your browser does not support inline SVG.

Kernel Size (kl): 5
Padding (pl, ql): 0
Stride (sl): 3

sl⋅rl - (sl-1)
(kl-1)/2
(kl-1)/2
fl-1
fl
rl=2

 So, we obtain the general recurrence equation (which is
 [first-order,
 non-homogeneous, with variable
 coefficients](https://en.wikipedia.org/wiki/Recurrence_relation#Solving_first-order_non-homogeneous_recurrence_relations_with_variable_coefficients)
 ):

 \(\begin{align}
 r\_{l-1} = s\_l \cdot r\_{l} + (k\_l - s\_l)
 \label{eq:rf\_recurrence}\
 \end{align}\)

 This equation can be used in a recursive algorithm to compute the receptive
 field size of the network, \(r\_0\). However, we can do even better: we can [solve
 the recurrence equation](#solving-receptive-field-size) and obtain a solution in terms of the \(k\_l\)’s and
 \(s\_l\)’s:

 \begin{equation}
 r\_0 = \sum\_{l=1}^{L} \left((k\_l-1)\prod\_{i=1}^{l-1}
 s\_i\right) + 1 \label{eq:rf\_recurrence\_final} \end{equation}

 This expression makes intuitive sense, which can be seen by considering some
 special cases. For example, if all kernels are of size 1, naturally the
 receptive field is also of size 1. If all strides are 1, then the receptive
 field will simply be the sum of \((k\_l-1)\) over all layers, plus 1, which is
 simple to see. If the stride is greater than 1 for a particular layer, the region
 increases proportionally for all layers below that one. Finally, note that
 padding does not need to be taken into account for this derivation.

### Computing receptive field region in input image

 While it is important to know the size of the region which generates one feature
 in the output feature map, in many cases it is also critical to precisely
 localize the region which generated a feature. For example, given feature
 \(f\_{L}(i, j)\), what is the region in the input image which generated it? This
 is addressed in this section.

 Let’s denote \(u\_l\) and \(v\_l\) the left-most
 and right-most coordinates (in \(f\_l\)) of the region which is used to compute the
 desired feature in \(f\_{L}\). In these derivations, the coordinates are zero-indexed (i.e., the first feature in
 each map is at coordinate \(0\)).

 Note that \(u\_{L} = v\_{L}\) corresponds to the
 location of the desired feature in \(f\_{L}\). The figure below illustrates a
 simple 2-layer network, where we highlight the region in \(f\_0\) which is used
 to compute the first feature from \(f\_2\). Note that in this case the region
 includes some padding. In this example, \(u\_2=v\_2=0\), \(u\_1=0,v\_1=1\), and
 \(u\_0=-1, v\_0=4\).

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 3
Padding (p1, q1): 1
Stride (s1): 3

Kernel Size (k2): 2
Padding (p2, q2): 0
Stride (s2): 1

u0 = -1
v0 = 4
f0
f1
f2

 We’ll start by asking the following question: given \(u\_{l}, v\_{l}\), can we
 compute \(u\_{l-1},v\_{l-1}\)?

 Start with a simple case: let’s say \(u\_{l}=0\) (this corresponds to the first
 position in \(f\_{l}\)). In this case, the left-most feature \(u\_{l-1}\) will
 clearly be located at \(-p\_l\), since the first feature will be generated by
 placing the left end of the kernel over that position. If \(u\_{l}=1\), we’re
 interested in the second feature, whose left-most position \(u\_{l-1}\) is \(-p\_l
 + s\_l\); for \(u\_{l}=2\), \(u\_{l-1}=-p\_l + 2\cdot s\_l\); and so on. In general:

 \(\begin{align}
 u\_{l-1}&= -p\_l + u\_{l}\cdot s\_l \label{eq:rf\_loc\_recurrence\_u} \\
 v\_{l-1}&= -p\_l + v\_{l}\cdot s\_l + k\_l -1
 \label{eq:rf\_loc\_recurrence\_v}
 \end{align}\)

 where the computation of \(v\_l\) differs only by adding \(k\_l-1\), which is
 needed since in this case we want to find the right-most position.

 Note that these expressions are very similar to the recursion derived for the
 receptive field size \eqref{eq:rf\_recurrence}. Again, we could implement a
 recursion over the network to obtain \(u\_l,v\_l\) for each layer; but we can also
 [solve for \(u\_0,v\_0\)](#solving-receptive-field-region) and obtain closed-form expressions in terms of the
 network parameters:

 \(\begin{align}
 u\_0&= u\_{L}\prod\_{i=1}^{L}s\_i - \sum\_{l=1}^{L}
 p\_l\prod\_{i=1}^{l-1} s\_i
 \label{eq:rf\_loc\_recurrence\_final\_left}
 \end{align}\)

 This gives us the left-most feature position in the input image as a function of
 the padding (\(p\_l\)) and stride (\(s\_l\)) applied in each layer of the network,
 and of the feature location in the output feature map (\(u\_{L}\)).

 And for the right-most feature location \(v\_0\):

 \(\begin{align}
 v\_0&= v\_{L}\prod\_{i=1}^{L}s\_i -\sum\_{l=1}^{L}(1 + p\_l -
 k\_l)\prod\_{i=1}^{l-1} s\_i
 \label{eq:rf\_loc\_recurrence\_final\_right}
 \end{align}\)

 Note that, different from \eqref{eq:rf\_loc\_recurrence\_final\_left}, this
 expression also depends on the kernel sizes (\(k\_l\)) of each layer.

**Relation between receptive field size and region.**
 You may be wondering that
 the receptive field size \(r\_0\) must be directly related to \(u\_0\) and
 \(v\_0\). Indeed, this is the case; it is easy to show that \(r\_0 = v\_0 - u\_0 +
 1\), which we leave as a follow-up exercise for the curious reader. To
 emphasize, this means that we can rewrite
 \eqref{eq:rf\_loc\_recurrence\_final\_right} as:

 \(\begin{align}
 v\_0&= u\_0 + r\_0 - 1
 \label{eq:rf\_loc\_recurrence\_final\_right\_rewrite}
 \end{align}\)

**Effective stride and effective padding.**
 To compute \(u\_0\) and \(v\_0\) in practice, it
 is convenient to define two other variables, which depend only on the paddings
 and strides of the different layers:

* *effective stride*
 \(S\_l = \prod\_{i=l+1}^{L}s\_i\): the stride between a
 given feature map \(f\_l\) and the output feature map \(f\_{L}\)
* *effective padding*
 \(P\_l = \sum\_{m=l+1}^{L}p\_m\prod\_{i=l+1}^{m-1} s\_i\):
 the padding between a given feature map \(f\_l\) and the output feature map
 \(f\_{L}\)

 With these definitions, we can rewrite \eqref{eq:rf\_loc\_recurrence\_final\_left}
 as:

 \(\begin{align}
 u\_0&= -P\_0 + u\_{L}\cdot S\_0
 \label{eq:rf\_loc\_recurrence\_final\_left\_effective}
 \end{align}\)

 Note the resemblance between \eqref{eq:rf\_loc\_recurrence\_final\_left\_effective}
 and \eqref{eq:rf\_loc\_recurrence\_u}. By using \(S\_l\) and \(P\_l\), one can
 compute the locations \(u\_l,v\_l\) for feature map \(f\_l\) given the location at
 the output feature map \(u\_{L}\). When one is interested in computing feature
 locations for a given network, it is handy to pre-compute three variables:
 \(P\_0,S\_0,r\_0\). Using these three, one can obtain \(u\_0\) using
 \eqref{eq:rf\_loc\_recurrence\_final\_left\_effective} and \(v\_0\) using
 \eqref{eq:rf\_loc\_recurrence\_final\_right\_rewrite}. This allows us to obtain the
 mapping from any output feature location to the input region which influences
 it.

 It is also possible to derive recurrence equations for the effective stride and
 effective padding. It is straightforward to show that:

 \(\begin{align}
 S\_{l-1}&= s\_l \cdot S\_l \label{eq:effective\_stride\_recurrence} \\
 P\_{l-1}&= s\_l \cdot P\_l + p\_l \label{eq:effective\_padding\_recurrence}
 \end{align}\)

 These expressions will be handy when deriving an algorithm to solve the case
 for arbitrary computation graphs, presented in the next section.

**Center of receptive field region.**
 It is also interesting to derive an
 expression for the center of the receptive field region which influences a
 particular output feature. This can be used as the location of the feature in
 the input image (as done for recent
 deep learning-based local features , for
 example).

 We define the center of the receptive field region for each layer \(l\) as
 \(c\_l = \frac{u\_l + v\_l}{2}\). Given the above expressions for \(u\_0,v\_0,r\_0\),
 it is straightforward to derive \(c\_0\) (remember that \(u\_{L}=v\_{L}\)):

 \(\begin{align}
 c\_0&= u\_{L}\prod\_{i=1}^{L}s\_i
 - \sum\_{l=1}^{L}
 \left(p\_l - \frac{k\_l - 1}{2}\right)\prod\_{i=1}^{l-1} s\_i \nonumber \\&= u\_{L}\cdot S\_0
 - \sum\_{l=1}^{L}
 \left(p\_l - \frac{k\_l - 1}{2}\right)\prod\_{i=1}^{l-1} s\_i
 \nonumber \\&= -P\_0 + u\_{L}\cdot S\_0 + \left(\frac{r\_0 - 1}{2}\right)
 \label{eq:rf\_loc\_recurrence\_final\_center\_effective}
 \end{align}\)

 This expression can be compared to
 \eqref{eq:rf\_loc\_recurrence\_final\_left\_effective} to observe that the center is
 shifted from the left-most pixel by \(\frac{r\_0 - 1}{2}\), which makes sense.
 Note that the receptive field centers for the different output features are
 spaced by the effective stride \(S\_0\), as expected. Also, it is interesting to
 note that if \(p\_l = \frac{k\_l - 1}{2}\) for all \(l\), the centers of the
 receptive field regions for the output features will be aligned to the first
 image pixel and located at \({0, S\_0, 2S\_0, 3S\_0, \ldots}\) (note that in this
 case all \(k\_l\)’s must be odd).

**Other network operations.**
 The derivations provided in this section cover most basic operations at the
 core of convolutional neural networks. A curious reader may be wondering
 about other commonly-used operations, such as dilation, upsampling, etc. You
 can find a discussion on these [in the appendix](#other-network-operations).

Arbitrary computation graphs
----------------------------

 Most state-of-the-art convolutional neural networks today (e.g.,
 ResNet  or
 Inception ) rely on models
 where each layer may have more than one input, which
 means that there might be several different paths from the input image to the
 final output feature map. These architectures are usually represented using
 directed acyclic computation graphs, where the set of nodes \(\mathcal{L}\)
 represents the layers and the set of edges \(\mathcal{E}\) encodes the
 connections between them (the feature maps flow through the edges).

 The computation presented in the previous section can be used for each of the
 possible paths from input to output independently. The situation becomes
 trickier when one wants to take into account all different paths to find the
 receptive field size of the network and the receptive field regions which
 correspond to each of the output features.

**Alignment issues.**
 The first potential issue is that one output feature may
 be computed using misaligned regions of the input image, depending on the
 path from input to output. Also, the relative position between the image regions
 used for the computation of each output feature may vary. As a consequence,
 **the receptive field size may not be shift-invariant**
  . This is illustrated in the
 figure below with a toy example, in which case the centers of the regions used
 in the input image are different for the two paths from input to output.

![](https://distill.pub/2018/feature-wise-transformations/images/pointer.svg)

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 5
Left Pad (p1): 2
Right Pad (q1): 1
Stride (s1): 2

Kernel Size (k2): 3
Left Pad (p2): 0
Right Pad (q2): 0
Stride (s2): 1

Kernel Size (k3): 3
Left Pad (p3): 0
Right Pad (q3): 0
Stride (s3): 1

Add

 In this example, padding is used only for the left branch. The first three layers
 are convolutional, while the last layer performs a simple addition.
 The relative position between the receptive field regions of the left and
 right paths is inconsistent for different output features, which leads to a
 lack of alignment (this can be seen by hovering over the different output features).
 Also, note that the receptive field size for each output
 feature may be different. For the second feature from the left, \(6\) input
 samples are used, while only \(5\) are used for the third feature. This means
 that the receptive field size may not be shift-invariant when the network is not
 aligned.

 For many computer vision tasks, it is highly desirable that output features be aligned:
 “image-to-image translation” tasks (e.g., semantic segmentation, edge detection,
 surface normal estimation, colorization, etc), local feature matching and
 retrieval, among others.

 When the network is aligned, all different paths lead to output features being
 centered consistently in the same locations. All different paths must have the
 same effective stride. It is easy to see that the receptive field size will be
 the largest receptive field among all possible paths. Also, the effective
 padding of the network corresponds to the effective padding for the path with
 largest receptive field size, such that one can apply
 \eqref{eq:rf\_loc\_recurrence\_final\_left\_effective},
 \eqref{eq:rf\_loc\_recurrence\_final\_center\_effective} to localize the region which
 generated an output feature.

 The figure below gives one simple example of an aligned network. In this case,
 the two different paths lead to the features being centered at the same
 locations. The receptive field size is \(3\), the effective stride is \(4\) and
 the effective padding is \(1\).

![](https://distill.pub/2018/feature-wise-transformations/images/pointer.svg)

 Sorry, your browser does not support inline SVG.

Kernel Size (k1): 1
Left Pad (p1): 0
Right Pad (q1): 0
Stride (s1): 4

Kernel Size (k2): 3
Left Pad (p2): 1
Right Pad (q2): 0
Stride (s2): 2

Kernel Size (k3): 1
Left Pad (p3): 0
Right Pad (q3): 0
Stride (s3): 2

Add

**Alignment criteria**
  . More precisely, for a network to be aligned at every
 layer, we need every possible pair of paths \(i\) and \(j\) to have
 \(c\_l^{(i)} = c\_l^{(j)}\) for any layer \(l\) and output feature \(u\_{L}\). For
 this to happen, we can see from
 \eqref{eq:rf\_loc\_recurrence\_final\_center\_effective} that two conditions must be
 satisfied:

 \(\begin{align}
 S\_l^{(i)}&= S\_l^{(j)} \label{eq:align\_crit\_1} \\
 -P\_l^{(i)} + \left(\frac{r\_l^{(i)} - 1}{2}\right)&= -P\_l^{(j)} + \left(\frac{r\_l^{(j)} - 1}{2}\right)
 \label{eq:align\_crit\_2}
 \end{align}\)

for all \(i,j,l\).

**Algorithm for computing receptive field parameters: sketch.**
 It is straightforward to develop an efficient algorithm that computes the receptive
 field size and associated parameters for such computation graphs.
 Naturally, a brute-force approach is to use the expressions presented above to
 compute the receptive field parameters for each route from the input to output independently,
 coupled with some bookkeeping in order to compute the parameters for the entire network.
 This method has a worst-case complexity of
 \(\mathcal{O}\left(\left|\mathcal{E}\right| \times \left|\mathcal{L}\right|\right)\).

 But we can do better. Start by topologically sorting the computation graph.
 The sorted representation arranges the layers in order of dependence: each
 layer’s output only depends on layers that appear before it.
 By visiting layers in reverse topological order, we ensure that all paths
 from a given layer \(l\) to the output layer \(L\) have been taken into account
 when \(l\) is visited. Once the input layer \(l=0\) is reached, all paths
 have been considered and the receptive field parameters of the entire model
 are obtained. The complexity of this algorithm is
 \(\mathcal{O}\left(\left|\mathcal{E}\right| + \left|\mathcal{L}\right|\right)\),
 which is much better than the brute-force alternative.

 As each layer is visited, some bookkeeping must be done in order to keep
 track of the network’s receptive field parameters. In particular, note that
 there might be several different paths from layer \(l\) to the output layer
 \(L\). In order to handle this situation, we keep track of the parameters
 for \(l\) and update them if a new path with larger receptive field is found,
 using expressions \eqref{eq:rf\_recurrence}, \eqref{eq:effective\_stride\_recurrence}
 and \eqref{eq:effective\_padding\_recurrence}.
 Similarly, as the graph is traversed, it is important to check that the network is aligned.
 This can be done by making sure that the receptive field parameters of different paths satisfy
 \eqref{eq:align\_crit\_1} and \eqref{eq:align\_crit\_2}.

Discussion: receptive fields of modern networks
-----------------------------------------------

 In this section, we present the receptive field parameters of modern
 convolutional networks

 The models used for receptive field computations, as well as the accuracy reported on ImageNet experiments,
 are drawn from the [TF-Slim
 image classification model library](https://github.com/tensorflow/models/tree/master/research/slim).

  , which were computed using the new open-source
 library (script
 [here](https://github.com/google-research/receptive_field/blob/master/receptive_field/python/util/examples/rf_benchmark.py)).
 The pre-computed parameters for
 AlexNet ,
 VGG ,
 ResNet ,
 Inception
 and
 MobileNet
 are presented in the table below.
 For a more
 comprehensive list, including intermediate network end-points, see
 [this
 table](https://github.com/google-research/receptive_field/blob/master/receptive_field/RECEPTIVE_FIELD_TABLE.md).

| ConvNet Model | Receptive Field (r) | Effective Stride (S) | Effective Padding (P) | Model Year |
| --- | --- | --- | --- | --- |
| alexnet\_v2 | 195 | 32 | 64 | [2014](https://arxiv.org/abs/1404.5997v2) |
| vgg\_16 | 212 | 32 | 90 | [2014](https://arxiv.org/abs/1409.1556) |
| mobilenet\_v1 | 315 | 32 | 126 | [2017](https://arxiv.org/abs/1704.04861) |
| mobilenet\_v1\_075 | 315 | 32 | 126 | [2017](https://arxiv.org/abs/1704.04861) |
| resnet\_v1\_50 | 483 | 32 | 239 | [2015](https://arxiv.org/abs/1512.03385) |
| inception\_v2 | 699 | 32 | 318 | [2015](https://arxiv.org/abs/1502.03167) |
| resnet\_v1\_101 | 1027 | 32 | 511 | [2015](https://arxiv.org/abs/1512.03385) |
| inception\_v3 | 1311 | 32 | 618 | [2015](https://arxiv.org/abs/1512.00567) |
| resnet\_v1\_152 | 1507 | 32 | 751 | [2015](https://arxiv.org/abs/1512.03385) |
| resnet\_v1\_200 | 1763 | 32 | 879 | [2015](https://arxiv.org/abs/1512.03385) |
| inception\_v4 | 2071 | 32 | 998 | [2016](https://arxiv.org/abs/1602.07261) |
| inception\_resnet\_v2 | 3039 | 32 | 1482 | [2016](https://arxiv.org/abs/1602.07261) |

 As models evolved, from
 AlexNet, to VGG, to ResNet and Inception, the receptive fields increased
 (which is a natural consequence of the increased number of layers).
 In the most recent networks, the receptive field usually covers the entire input image:
 this means that the context used by each feature in the final output feature map
 includes all of the input pixels.

 We can also relate the growth in receptive fields to increased
 classification accuracy. The figure below plots ImageNet
 top-1 accuracy as a function of the network’s receptive field size, for
 the same networks listed above. The circle size for each data point is
 proportional to the number of floating-point operations (FLOPs) for each
 architecture.

 google.charts.load('current', { 'packages': ['corechart'] });
 google.charts.setOnLoadCallback(drawSeriesChart);

 function drawSeriesChart() {

 var data = google.visualization.arrayToDataTable([
 ['ID', 'Receptive field size (pixels)', 'ImageNet top-1 accuracy', 'Family', 'FLOPS (Billion)'],
 ['alexnet\_v2', 195, 0.5720, 'alexnet', 1.38],
 ['vgg\_16', 212, 0.7150, 'vgg\_16', 30.71],
 ['inception\_v2', 699, 0.7390, 'inception', 3.88],
 ['inception\_v3', 1311, 0.7800, 'inception', 5.69],
 ['inception\_v4', 2071, 0.8020, 'inception', 12.27],
 ['inception\_resnet\_v2', 3039, 0.8040, 'inception\_resnet', 12.96],
 ['resnet\_v1\_50', 483, 0.7520, 'resnet', 6.97],
 ['resnet\_v1\_101', 1027, 0.7640, 'resnet', 14.40],
 ['resnet\_v1\_152', 1507, 0.7680, 'resnet', 21.82],
 ['mobilenet\_v1', 315, 0.7090, 'mobilenet', 1.14]
 ]);

 var options = {
 title: '',
 hAxis: { title: 'Receptive field size (pixels)', gridlines: { count: 10 } },
 vAxis: { title: 'ImageNet top-1 accuracy', format: 'percent' },
 sizeAxis: { maxSize: 10 },
 legend: { position: 'bottom', textStyle: { fontSize: 12 } },
 bubble: { textStyle: { fontSize: 11 } }
 };

 var chart = new google.visualization.BubbleChart(document.getElementById('series\_chart\_div'));
 chart.draw(data, options);
 }

 We observe a logarithmic relationship between
 classification accuracy and receptive field size, which suggests
 that large receptive fields are necessary for high-level
 recognition tasks, but with diminishing rewards.
 For example, note how MobileNets achieve high recognition performance even
 if using a very compact architecture: with depth-wise convolutions,
 the receptive field is increased with a small compute footprint.
 In comparison, VGG-16 requires 27X more FLOPs than MobileNets, but produces
 a smaller receptive field size; even if much more complex, VGG’s accuracy
 is only slightly better than MobileNet’s.
 This suggests that networks which can efficiently generate large receptive
 fields may enjoy enhanced recognition performance.

 Let us emphasize, though, that the receptive field size is not the only factor contributing
 to the improved performance mentioned above. Other factors play a very important
 role: network depth (i.e., number of layers) and width (i.e., number of filters per layer),
 residual connections, batch normalization, to name only a few.
 In other words, while we conjecture that a large receptive field is necessary,
 by no means it is sufficient.

 Additional experimentation is needed to confirm this hypothesis: for
 example, researchers may experimentally investigate how classification
 accuracy changes as kernel sizes and strides vary for different
 architectures. This may indicate if, at least for those architectures, a
 large receptive field is necessary.

 Finally, note that a given feature is not equally impacted by all input pixels within
 its receptive field region: the input pixels near the center of the receptive field have more “paths” to influence
 the feature, and consequently carry more weight.
 The relative importance of each input pixel defines the
 *effective receptive field* of the feature.
 Recent work
 provides a mathematical formulation and a procedure to measure effective
 receptive fields, experimentally observing a Gaussian shape,
 with the peak at the receptive field center. Better understanding the
 relative importance of input pixels in convolutional neural networks is
 an active research topic.

Detailed derivations and open-source code to analyze the receptive fields of convnets.

Visualizing higher-layer features of a deep network

Visualizing and Understanding Convolutional Networks

Global Optimality in Neural Network Training

On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport

Object Detectors Emerge in Deep Scene CNNs

Interpreting Deep Visual Representations via Network Dissection

In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning

Understanding Deep Learning Requires Rethinking Generalization

A Guide to Receptive Field Arithmetic for Convolutional Neural Networks

What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?

ImageNet Classification with Deep Convolutional Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Deep Residual Learning for Image Recognition

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

Large-Scale Image Retrieval with Attentive Deep Local Features

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Understanding the Effective Receptive Field in Deep Convolutional Neural Networks