Granger-causal Attentive Mixtures of Experts
Patrick Schwab 1 Walter Karlen 1

arXiv:1802.02195v1 [cs.LG] 6 Feb 2018

Abstract
Several methods have recently been proposed to
detect salient input features for outputs of neural networks. Those methods offer a qualitative
glimpse at feature importance, but they fall short
of providing quantifiable attributions that can be
compared across decisions and measures of the
expected quality of their explanations. To address these shortcomings, we present an attentive
mixture of experts (AME) that couples attentive
gating with a Granger-causal objective to jointly
produce accurate predictions as well as measures
of feature importance. We demonstrate the utility
of AMEs by determining factors driving demand
for medical prescriptions, comparing predictive
features for Parkinson’s disease and pinpointing
discriminatory genes across cancer types.

1. Introduction
Neural networks are often criticised for being black-box
models. Researchers have attempted to address this criticism by developing tools that provide visualisations and
qualitative explanations of the inner workings of neural networks (Baehrens et al., 2010; Simonyan et al., 2014; Zeiler
& Fergus, 2014; Xu et al., 2015; Shrikumar et al., 2017;
Zhang et al., 2017; Montavon et al., 2017; Koh & Liang,
2017). However, comparatively little progress has been
made towards providing quantifiable measures of relative
feature importance and their expected accuracy.
Quantifiable measures are desirable for the many machinelearning use cases in which both predictive performance
and interpretability are of paramount importance (Kindermans et al., 2017; Smilkov et al., 2017; Doshi-Velez &
Kim, 2017). Such measures facilitate communicating, adjusting and arguing for the decisions of machine-learning
models (Riemer et al., 2016). They enable us to tell when
1

Institute of Robotics and Intelligent Systems, MHSL, DHEST, ETH Zurich, Switzerland. Correspondence to: Patrick
Schwab <patrick.schwab@hest.ethz.ch>.
arXiv preprint.
February 8, 2018.
Copyright 2018 by the author(s).

+
h1

y1

a1

Expert E1

G1
h2

y

y2

Granger-causally grounded
a2

Expert E2

G2
h3

y3

a3

Expert E3

Attentive Gating
Network G3
hc = (h1,y1,h2,y2,h3,y3)

Figure 1. An overview of attentive mixtures of experts (AMEs).
The attentive gating networks Gi (red) attend to the combined
hidden state hc (blue) of the AME. Each expert’s Gi assigns an
attentive factor ai to opportunistically control it’s contribution yi
to the AMEs final prediction y.

algorithmic decisions might be biased or discriminating
(Hardt et al., 2016) and uncover the basis of decisions when
there are legal or ethical circumstances that call for explanations (Goodman & Flaxman, 2016). Moreover, information about the source of a decision is especially important
when a wrong decision could potentially have severe consequences (Choi et al., 2016), when a model’s decisions are
merely hints that suggest more thorough follow-up assessments (Schwab et al., 2017), and when a model discovers
patterns that could advance the scientific understanding of
the underlying phenomena (Shrikumar et al., 2017).
Estimating the relative contribution of input features towards individual outputs of a deep neural network is hard
because the input features undergo multiple hierarchical,
interdependent and non-linear transformations as they pass
through the network (Montavon et al., 2017). We propose
a new approach to importance attribution that optimises
jointly for predictive performance and accurate attribution
of feature importance in an end-to-end trained neural network. Our approach builds on the intuitive idea of distributing the feature groups of interest among experts in a mixture of experts model that uses attentive gating to assign
weights to individual experts. By optimising the attentive
gating networks with a Granger-causal objective, we ensure that the weights given to individual experts correlate
strongly and measurably with their ability to contribute to
the decision at hand. We demonstrate that our approach is
able to combine the strength of neural networks as scalable
feature extractors with the ability to quantitatively measure
feature importance - enabling new use cases for neural networks in knowledge discovery from observational data and
allowing measurably accurate attribution in existing ones.

Granger-causal Attentive Mixtures of Experts

Contributions. We subdivide this work along the following distinct contributions:
(i) We delineate an end-to-end trained attentive mixture
of experts (AME) model that uses attentive gates to
orchestrate the outputs of multiple heterogenous expert networks.
(ii) We introduce a Granger-causal objective that ensures
that the attention weights assigned by the attentive
gates in AMEs correlate with the respective expert’s
ability to contribute to an individual decision.
(iii) We demonstrate the utility of AMEs in a variety of
settings by determining factors driving demand for
medical prescriptions, comparing predictive features
for Parkinson’s disease and pinpointing discriminatory genes across multiple types of cancer.

2. Related Work
There are two main categories of approaches towards attribution in neural networks:
Perturbation-based Approaches. In perturbation-based
approaches, the sensitivity of individual inputs and neurons in a neural network are explained by modelling the
impact of local perturbations (Ribeiro et al., 2016; Adler
et al., 2016).
Gradient-based Sensitivity Analysis. Gradient-based approaches are based on the idea of following the gradient
from output nodes to the input nodes to obtain the features
that the output was most sensitive to (Baehrens et al., 2010;
Simonyan et al., 2014). Several improvements to this technique have since been proposed (Zeiler & Fergus, 2014;
Selvaraju et al., 2016; Smilkov et al., 2017; Sundararajan
et al., 2017).
Both perturbation-based and gradient-based approaches
share the same limitations: They neither provide measures
of relative feature importance for all input features whose
magnitude can be meaningfully compared across decisions,
nor an expected accuracy for their explanations - limiting
their use to qualitative analyses on singular decisions.
Orthogonal to the above-mentioned categories are approaches that provide attribution by pointing out the influential training data for a given decision (Koh & Liang,
2017). Similarly, AMEs are outside any of the abovementioned categories, as they optimise jointly for predictive performance and accurate attribution in a single endto-end trained model for each individual decision.
Mixtures of Experts. Conceptually, AMEs can be compared to hierarchical mixtures of experts (HME) with gating networks (Jacobs et al., 1991; Jordan & Jacobs, 1994).
In HMEs, (Jacobs et al., 1991) trained a gating network
that, based on the respective input sample, stochastically

activates exactly one out of many specialised expert networks. The goal of their approach is to achieve decoupled,
competing experts that decompose a given learning problem into subtasks that are ideally easier to solve. More
recently, (Shazeer et al., 2017) proposed an end-to-end optimised mixture of experts layer that lowers computational
cost in machine translation through conditional computation. The aforementioned mixture models use gating to increase either computational or predictive performance and
require that each expert receives input of the same structure, thus making them unsuitable for heterogenous inputs.
Gating itself has been used in neural networks for a long
time. For example, gating is used to enable learning longterm dependencies in Long Short-Term Memory layers
(Hochreiter & Schmidhuber, 1997) and to facilitate training of deep neural architectures (Srivastava et al., 2015).
Attentive Models. Attentive models have been used in various domains to improve both interpretability (Xu et al.,
2015; Schwab et al., 2017; Zhang et al., 2017; Riemer
et al., 2016) and performance (Bahdanau et al., 2015; Yang
et al., 2016). In computer vision, related works used attention in convolutional neural networks that selectively focus
on input data (Ba et al., 2014) and internal convolutional
filters (Stollenga et al., 2014). However, fundamentally,
naı̈ve soft attention mechanisms do not provide any incentive for a neural network to yield attention factors that correlate with feature importance. Soft attention mechanisms
may therefore interact with other components of neural networks in opaque ways (Sundararajan et al., 2017). Furthermore, it is not possible to measure the expected quality of
explanations provided by naı̈ve soft attention mechanisms.
We therefore follow a new approach to attribution in deep
neural networks that models and optimises explicitly for
predictive performance as well as attribution accuracy. This
approach has two key advantages over existing ones: It provides (i) a measure of the expected quality of the network’s
attributions and has (ii) the ability to compare the relative
magnitude of attributions across individual decisions - enabling the quantitative analysis of model errors and biases,
and the identification of novel patterns over multiple independent decisions in observational data.

3. Attentive Mixtures of Experts
AMEs consist of n experts Ei and their corresponding attentive gating networks Gi with i ∈ [1, n]. At prediction
time, the attentive gating networks output an attention factor ai for each expert to control its respective contribution
yi to the AMEs final prediction y. All of the experts and
the attentive gating network are neural networks with their
own parameters and architectures. AMEs do not impose
any restrictions on the expert networks other than that they
need to expose their topmost feature representation hi and

Granger-causal Attentive Mixtures of Experts

We denote Ei (xi ) as the output yi of the ith expert for the
given subset of the input data xi . Gi (hc ) represents the
output ai of the ith attentive gating network Gi with respect
to the combined hidden state hc of the AME. The output y
of an AME is then given by:
y=

n
X

Gi (hc )Ei (xi )

(1)

i=1

AMEs use the attentive gating networks to control the contributions of the expert outputs Ei (xi ) (Figure 2). The output Gi (hc ) of the attentive gating network is the attention
factor ai for the ith expert. The attention factors dynamically modulate the contribution of each expert to the final
prediction based on the current combined hidden state hc
of the AME. We calculate the attention factors ai using:
exp(uTi uEi )
ai = Pn
T
j=1 exp(uj uEj )

(2)

ui = activation(Wi hc + bi )

(3)

where

corresponds to a single-hidden-layer multi-layer perceptron (MLP) with an activation function, a weight matrix Wi
and bias bi that yields ui as a hidden representation of hc
(Yang et al., 2016). In equation (2), we calculate the attention factors ai for each expert by computing a softmax over
the dot-product similarity of ui to the ith expert’s context
vector uEi . uEi corresponds to a hidden representation of
the most informative combined hidden state hc for expert
Ei (Yang et al., 2016). We jointly optimise Wi , bi and uEi
with the other parameters during training. By comparing
the current hidden representation ui of hc to the most informative hidden representation uEi for the ith expert, the
attentive gating network learns to activate experts for those
hidden states hc in which they can most contribute to the
final prediction y of the AME. Due to the use of an individual attention mechanism for each expert, the soft attention
mechanism described by equations (3) and (2) differs significantly to that used in related works. Related works typically train a shared most informative state over a common

hc

Attentive Gating Network Gi

uEi

}

their local prediction yi for a given input subset xi . As input to the gating networks, the hidden states hi and local
predictions yi of each expert are concatenated to form the
combined hidden state hc of the whole AME. An important property to note is that AMEs do not require that every
expert makes its predictions based on the same subset of
the input data x. Each expert may therefore operate on its
own distinct subset xi of the input data. In our experiments,
we make use of this property to be able to quantify the impact of different feature groups xi of the input data on the
AME’s decisions. Figure 1 shows a conceptual overview
of an AME’s components and their interactions.

h0 y1 h2 y2 ...

Expert Ei

MLPi

<>
ui

ai
yi

Figure 2. A graphical illustration of the attentive gating network
Gi (red, dashed) placed on top of each expert Ei within an AME
model. Each gating network has an associated MLPi (grey)
with its own weight Wi and bias bi parameters. The experts’
MLPi transforms the the combined hidden state hc (blue) of the
AME into a hidden representation ui . We measure the similarity
(”<>”) of the hidden representation ui to the most informative
hidden context representation uEi (orange) to obtain the attention
factor ai for the local prediction yi of expert Ei .

hidden representation, i.e. (Yang et al., 2016), rather than
an individual most informative state for each branch.
Conceptually, the formulation of equation (1) is reminiscent to that of gradient boosting (Friedman, 2002), where
new estimators are iteratively added to reduce the residual error of the previous estimators, and to that of generalised additive models (Hastie & Tibshirani, 1990). Aside
from lending themselves to end-to-end optimisation, we
note that key characteristics of AMEs are the rich introspective ability, given both to the gating network as well as
to human observers through the attention mechanism, and
the ability of the attentive gating network to dynamically
activate experts depending on the combined hidden state of
the whole AME rather than just their inputs.

4. A Granger-causal Objective for Attentive
Mixtures of Experts
In line with reports of related works that rely on gating
networks (Bengio et al., 2015; Shazeer et al., 2017), we
find that early on during training with mini-batch stochastic gradient descent our proposed attentive gating mechanism commonly collapses to either consistently assign an
attention weight ai = 1 to a single expert and 0 to all others, or to assign the same amount of attention ai = n1
to all experts. Once the AME is stuck one of these local minima, it is unlikely to recover. The fundamental
problem is that a naı̈vely-trained AME has no incentive
to learn feature representations that yield accurate attributions. To combat this undesired tendency towards attributions that do not correspond to the experts’ respective ability to contribute, we introduce a secondary Granger-causal
objective function LGranger . Granger-causality follows the
Humean definition of causality that declares a causal relationship X → Y between random variables X and Y
if we are better able to predict Y using all available information than if the information apart from X had been

Granger-causal Attentive Mixtures of Experts

used (Granger, 1969). In spite of its name, in empirical
settings, Granger-causality is more akin to correlation than
causation and we therefore strongly caution the reader to
not mistake Granger-causality for the standard definition
of causality (Pearl et al., 2009). Nonetheless, the theory of
Granger-causality allows us to naturally define a simple and
efficient loss function that ensures that the relative magnitudes of attention factors ai correlate with their respective
expert’s ability to contribute to the AME’s final output y.
Given a set of heterogenous input data X, we denote εX,i
as the AME’s prediction error without including any information from the ith expert and εX as the AME’s prediction
error when considering all available information (Figure 3).
We then define the degree ∆εi to which the ith expert is
able to contribute to the final output y as the decrease in error associated with adding that expert’s information to the
set of available information sources:
∆εX,i = εX\{i} − εX

(4)

We note that using ∆εX,i to define the degree of attribution
intuitively resolves even cases where combinations of features enable improvements in the AME’s prediction error both experts would be attributed equally for the decrease.
Using ∆εi , we define the desired attribution ωi corresponding to the ith experts’ attention weights ai for a given input
X as:
∆εX,i
ωi (X) = Pn
j=1 ∆εX,j

(5)

Where equation (5) normalises the attributions across all
experts to ensure that they are on the same scale within and
across decisions. We then arrive at the Granger-causal objective function LGranger by computing the mean KullbackLeibler divergence KL over n samples between the target
distribution Ω, with Ω(i) = ωi , and the actual distribution
A, with A(i) = ai , of attention values:
1X
KL(Ω, A)
(6)
LGranger =
n
X

Because the Granger-causal loss measures the probabilistic
distance of the actual attributions to the desired Grangercausal attributions, it is valid to use it as a proxy for the
expected quality of explanations. A Granger-causal loss of
0 indicates a perfect match with the Granger-causal attributions and satisfies both axioms outlined by (Sundararajan
et al., 2017). We can therefore apply the familiar framework of cross-validation and held-out test data to estimate
the expected accuracy of the attributions provided by the
attention factors ai . Finally, the total loss L of the AME is
the sum of the main loss, which may be arbitrarily chosen
depending on the dataset at hand, and the Granger-causal
loss:
L = Lmain + LGranger

(7)

ε

εX

faux,2 X/{2}

faux
E1 E2

E3 E4

E1 E2

E3 E4

Figure 3. We quantify the degree to which an expert is able to contribute to individual decisions by comparing the prediction error
when using all available information (left) and the prediction error
when using the information apart from that expert (right).

If desired, it is possible to express an a-priori preference
for more accurate explanations over accurate predictions
(or vice-versa) by weighting the components of L accordingly. In terms of implementation, we estimate both εX
and εX\{i} using the ground truth labels ytrue , the main loss
Lmain , and single-layer auxiliary models faux and faux,i that
receive as input the combined hidden state hc of the AME
and the combined hidden state minus the hidden state and
local prediction of the ith expert hc \ {hi , yi }, respectively.
Because faux can be reused across experts, the error estimation requires only one additional auxiliary output per
expert. We additionally clip ∆εX,i to be positive to not
consider cases where εX > εX\{i} , which might happen
due to different degrees of intermittent convergence in the
auxiliary outputs. Furthermore, we note that it is important
to prevent gradient flow towards ∆εX,i , e.g. using TensorFlow’s tf.stop gradient, to prevent the network
from optimising the error to match its desired attention values rather than the other way around.
Theoretically, training to provide attributions and predictions at the same time should have no adverse impact on
the predictive performance of a neural network, as both objectives share a common feature space, i.e. being able to
accurately attribute is helpful for producing predictions and
vice-versa. In essence, AMEs make the attribution that is
implicitly performed by a neural network explicit.

5. Experiments
In order to demonstrate the utility of AMEs, we performed
experiments on a variety of tasks with distinctly different characteristics. Our experiments include settings with
high numbers of samples, high numbers of features, high
correlations between features and heterogenous input data.
Specifically, we use AMEs to (i) determine the drivers of
medical prescription demand at scale, (ii) compare predictive features for monitoring Parkinson’s disease, and (iii)
pinpoint discriminatory genes across multiple types of cancers. The aim of our experiments is to showcase the breadth
of tasks that neural networks as scalable, non-linear feature
extractors can be used for when equipped with the ability to
quantitatively measure feature importance across decisions.
In addition, we wish to evaluate to what degree attentive
gating decisions correlate with an expert’s ability to contribute and whether providing attributions has an adverse

Granger-causal Attentive Mixtures of Experts

effect on predictive performance. The source code for our
experiments is available at https://github.com/
d909b/attentive_mixtures_of_experts.
5.1. Drivers of Medical Prescription Demand
Accurate demand forecasts for medical prescriptions play
an important role in preventing stock outages and keeping
the cost of supply operations in check. However, demand
forecasting for prescriptions is an extremely challenging
task due to the large set of prescribed items with poorly
understood characteristics and influence factors. To gain a
deeper understanding of what factors drive demand for individual prescription items, we trained an AME model to
predict the next month’s demand (in terms of revenue) for
individual prescription items at the level of practices. The
inspection of the assigned attention factors in the AME enables us to analyse which factors are most predictive for
specific prescription items. This regression setting is challenging due to the highly heterogenous input data on different time scales and the need to scale to tens of millions
of available training samples.
5.1.1. DATASET
We collected multiple heterogenous data streams covering
the whole country of England, United Kingdom (UK) during the time frame from January 2011 to December 2012
(24 months). We used data streams corresponding to six
feature groups1 . Each data stream listed below is represented as an expert in our AME model (Figure 4).
Demand History. As the primary data stream, we used
the monthly data on all revenues generated by reimbursed
prescription items in 9,455 general practices (GPs) in England, UK as released by the British National Health Service
(NHS)2 . To ensure a full history of the past demand for pre1

The supplementary material contains the exhaustive list of
input features per expert.
2
https://data.gov.uk/dataset/prescribing-by-gp-practicepresentation-level; accessed 20th January 2018
y

+

y
NN
x1 x2 ... xn

G1

G2

...

Gn

E1

E2

...

En

x1

x2

xn

Figure 4. An illustration of the process of converting an existing
neural network (NN) of any architecture into an AME with experts Ei operating on predefined feature subsets xi . Because each
expert is limited to make their local prediction based only on their
subset of the data, the post-hoc analysis of the attention factors
output by the attentive gating networks Gi (red) enables us to
quantify the contribution of individual experts on the output y.

dictions, we removed those practices from the dataset that
remained closed at any point during the prediction time
frame. To remove the noise introduced by different prescription formulations and packagings, we aggregated all
prescription items into their prescription item class as defined by their pharmaceutical code in the British National
Formulary (BNF) reported by the NHS.
For context, we added the monthly total revenue, the region the current practice belongs to and the practice’s distance to that region’s centre as additional input features.
We used the territory level 3 (TL3) regions as defined by
(OECD, 2017) to geographically subdivide England into
145 regions.
Online Search Interest. We used Google Trends3 to retrieve the relative monthly online search interest for our
time frame and regional code (GB-EN). In total, we queried
online search interest for 109,779 medical terms from the
comprehensive medical subject headings (MeSH) ontology
(Lipscomb, 2000). As many of the medical terms in the
MeSH ontology are not common in vernacular, ultimately
only 9,225 (8.40%) out of the 109,779 medical terms had a
search activity that was significant enough to return a result
on Google Trends. We applied a principal component analysis (PCA) transform that explains more than 98% of the
variance in the search interest data and reduces the feature
vector to just 23 dimensions.
Regional Weather. Weather impacts consumption across a
wide variety of industries (Lazo et al., 2011). We therefore
added monthly weather data, including rainfall, days of air
frost, hours of sunshine and average temperature data, from
23 UK Met Office4 stations to our dataset.
Regional Demographic, Economic and Labor Data. Regional demographic, economic and labor data in proximity
to a practice could affect future prescription demand, as
groups with different population-level profiles potentially
require different types of care in different amounts. To
further analyse the predictive potential of population-level
factors, we added 45 yearly indicators on the demographic
profile, 27 yearly indicators on the economic profile and
15 yearly indicators of the labor profile of each TL3 region
from (OECD, 2017) to our dataset. We represented each of
the three population-level feature groups of indicators as a
distinct expert in our AME model. Analogous to the online
search interest data, we applied PCA transforms that explain over 99% of the variance in the three feature groups.
3

https://trends.google.com/; accessed 20th January 2018
http://www.metoffice.gov.uk/public/weather/climatehistoric/; accessed 20th January 2018
4

Granger-causal Attentive Mixtures of Experts

5.1.2. E VALUATION S ETUP
As baselines, we trained autoregressive integrated moving average (ARIMA) models, recurrent neural networks
(RNN) and feedforward neural networks (FNN). The architectures of the RNN and FNN models were derived from
the trained AME model (Figure 4). The main difference
between the evaluated FNN and the RNN is that the former
receives the input data collapsed over time rather than sequentially. Univariate ARIMA models are commonly used
for forecasting in practice (Riemer et al., 2016) and, in this
comparison, served as a baseline that does not make use of
any information apart from the revenue history.
Preprocessing. Prior to fitting the models, we standardised
the prescription revenue history data for each time series to
the range [0, 1]. We preserved the minimum and maximum
time series value as separate input features. We additionally normalised all numeric features, except the prescription revenue data, to have a mean value of 0 and a standard
deviation of 1 using the data in the training set. All compared models benefited from the same preprocessing.
Hyperparameters. To ensure a fair comparison, we took
a systematic approach towards hyperparameter search. For
all models based on neural networks, we performed a hyperparameter search with hyperparameters chosen at random from predefined ranges over 35 runs. For our performance comparison, we selected those models from the hyperparameter search that achieved the best performance on
the validation data. We used batch normalisation (Ioffe &
Szegedy, 2015) and dropout in all the neural network models. Methodologically, we optimised the neural networks’
mean squared error with early stopping on the validation
set. For ARIMA, we used the iterative parameter selection
algorithm from (Hyndman et al., 2007).
Dataset Split. We applied a random split by general practice to separate the available data into a training (60%),
validation (20%) and test set (20%). We report the main
properties of the different folds in Table 1.
Metrics. We compared the predictive accuracy of the different models by computing their symmetric mean absolute percentage error (SMAPE) (Flores, 1986) and rate of
anomalous events (in %) on our held-out test set of 1,891
Table 1. The properties of the dataset folds used in the demand
forecasting experiment. We report the number of time series, the
number of general practices (GPs), the number of distinct prescription item classes (Items), the average monthly prescription
item revenue in pound sterling per practice per month (£ / mo.)
and the corresponding percentage (%) of the whole dataset.
Training
Validation
Test
Total

Time series
24.30 million
9.07 million
9.07 million
42.44 million

GPs
5,673
1,891
1,891
9,455

Items
1,463
1,367
1,368
1,495

£ / mo.
64,976
63,776
64,327
64,606

%
60
20
20
100

Table 2. Comparison of the symmetric mean absolute percentage
error (SMAPE) and the rate of anomalous events (in %) on the
held out test set of 1,891 practices (= 9.07 million time series).
We also report the average± standard deviation of CPU hours required for training and evaluation across the 35 runs.
Method
ARIMA
RNN
FNN
AME

SMAPE (%)
34.98
32.79
32.87
32.84

Anomaly (%)
40.38
34.59
34.83
34.83

CPU (hr)
527.96
2.92 ± 1.08
0.25 ± 0.07
2.28 ± 0.71

practices. We define an anomalous event as a model’s prediction being either two times smaller or two times bigger
than the ground truth demand (Riemer et al., 2016).
5.1.3. R ESULTS AND D ISCUSSION
Performance. Our results showed that the extra information collected from various sources indeed improved the
prediction performance over the univariate ARIMA baseline - especially for anomalous events (Table 2). This finding suggests that healthcare providers and suppliers should
consider incorporating more open data sources into their
forecasting models in order to reduce supply costs and to
better manage the risk of out-of-stock events. In addition,
we found that (i) providing attributions did not have an adverse impact on performance when comparing the AME
and the corresponding FNN model and that (ii) the time to
train AME models was comparable to that of RNNs.
Attribution. To map out the factors driving demand for
prescription items, we plotted the mean contribution peryi |
) and the average
centage M CPi = mean( Pn|ai |a
j=1 j yj |
amount of attention given to experts for specific prescription items. Our results showed that there are significant differences in sensitivity profiles across the different prescription items (Figure 5). To determine the degree to which
unmodulated yi and modulated expert predictions ai yi correlate with the ground truth demand ytrue , we performed
hypothesis tests using Kendall’s τ (Kendall, 1945). We
tested for correlation for each of the prescription items on
the held-out test set. We applied the Benjamini-HochbergYekutieli procedure (Benjamini & Yekutieli, 2001) at α =
0.05 to control the false discovery rate due to multiple testing. The large differences - with the exception of the deTable 3. The ratio of prescription items (in %) for which local
predictions yi or contributions ai yi were considered significantly
correlated to the ground truth prescription demand ytrue , and the
average absolute effect size (|τ |) of the contributions’ correlations
over all prescription items in the held-out test set.
Expert
(a) Demand History
(b) Online Search interest
(c) Weather
(d) Demographic Data
(e) Economic Data
(f) Labor Data

yi (%)
82.82
41.59
40.72
30.77
31.21
33.55

ai yi (%)
81.80
71.34
70.03
58.77
64.98
62.94

|τ |
0.21
0.13
0.12
0.09
0.11
0.11

Granger-causal Attentive Mixtures of Experts
contributes more often
Aspirin

Insulin Glargine

Rosuvastatin

larger contributions

Hydrocortisone

0.05

0.05

(a) Demand History

(b) Online Search Interest

(c) Regional Weather

(d) Regional Demographic Data

(e) Regional Economic Data

(f) Regional Labor Data

Figure 5. The influence of experts specialised on demand history (a, red), online search interest (b, orange), regional weather (c, blue),
regional demographic data (d, purple), regional economic data (e, brown) and regional labor data (f, grey) on the predictions y made by
the AME for each of the 1, 368 prescription item classes (dots) in our held-out test set. For each item, we report the average attention
(x-axis, in %) and the mean contribution percentage (MCP, y-axis) attributed to each expert. The dashed lines indicate the mean average
attention and MCP over all prescription item classes. Symbols (star, square, circle, triangle) point out positions of a selected set of
well-known prescription items that have significant correlations (Kendall’s τ ) to the ground truth demand ytrue in the respective experts.

mand history expert - between modulated and unmodulated
expert outputs indicate that the attentive gating decisions
were indeed highly effective at dynamically activating experts according to their ability to contribute (Table 3). We
reason that the rate of significant correlations did not increase for the demand expert because it was already highly
correlated to the ground truth before modulation.
5.2. Predictive Features for Parkinson’s Disease
Parkinson’s disease (PD) is a slowly progressing neurodegenerative disease for which there currently exists no cure.
However, once diagnosed, the symptoms of PD can be
managed effectively using surgical or pharmacological interventions. An early diagnosis can therefore significantly
improve the quality of life of patients (Little et al., 2009).
Because PD manifests itself through speech impairments,
microphones could be used for this task (Little et al., 2009).
We train an AME to classify a set of audio features as to
whether they were produced by a person with or without
PD. Analysing the attention weights assigned by the AME
enables us to compare the suitability of various features for
this task. In this setting, our goal was to evaluate the behavior of AMEs in the presence of highly correlated inputs.
Dataset. We used the Oxford Parkinson’s tele-monitoring
dataset (Little et al., 2009) consisting of 195 sustained
vowel phonations from 31 study participants (23 diagnosed
with PD). (Little et al., 2009) reported 22 highly correlated features and determined several subsets of features
that achieve similar performances on this dataset. For comparison, we use the results of (Little et al., 2009) as a baseline. We applied a stratified random split to separate the

available data into 117 samples for training (60%), 39 samples for validation (20%) and 39 for testing (20%).
Hyperparameters. For each of the 22 features, we used a
MLP with a single hidden layer with batch normalisation,
20% dropout rate and 16 neurons as expert networks in the
AME. We trained the AME with a learning rate of 0.0003
and 25 epochs of early stopping patience on the validation
set. We chose the hyperparameters based on a grid search
against the validation set.
5.2.1. R ESULTS AND D ISCUSSION
Performance. Our AME achieves a classification error of
7.69% on the test set, which is well within the best classification error interval of 9.4 ± 4.1% reported by (Little et al.,
2009) using bootstrap resampling.
Attribution. We found an almost uniform distribution of
attention factors of 3.39 ± 0, 22% across all 22 features on
the held-out test set (LGranger = 0.43). The uniform distribution of attention factors indicates that many of the 22
features are redundant. Indeed, (Little et al., 2009) reported
that a subset of just 4 of these features suffices to reach the
performance of all available features. The AME therefore
correctly identified that those features are largely equivalent in importance. However, because the differences between feature importances were within the range of the estimation error of attributions, it was not possible to determine the best subset of features from the attention factors.
We hypothesise that a larger training set might raise the
attribution accuracy to a level that enables distinguishing
between highly correlated features.

Granger-causal Attentive Mixtures of Experts

5.3. Discriminatory Genes Across Cancer Types
With the widespread adoption of next generation sequencing, there now exists a wealth of genomic data from which
we could infer genes that are predictive of specific phenotypes. Knowledge of the genomic factors related to phenotypes can help guide the development of treatments, as well
as increase the understanding of the biological processes
involved in complex diseases. To pinpoint the genes that
differentiate between different types of cancer, we trained
an AME model to classify gene expression data as being either breast invasive carcinoma (BRCA), kidney renal clear cell carcinoma (KIRC), colon adenocarcinoma
(COAD), lung adenocarcinoma (LUAD) and prostate adenocarcinoma (PRAD). This setting is characterised by a
large number of input genes out of which few are relevant.
Dataset. We used a subset of the data from The Cancer
Genomic Atlas (TCGA) pan-cancer RNAseq dataset (Weinstein et al., 2013), which includes gene expression data
from multiple cancer types in 800 individuals. To keep visualisations succinct, we used a subset of 100 genes as input data. We applied a stratified random split to separate
the available data into 480 samples for training (60%), 160
samples for validation (20%) and 161 for testing (20%).
Hyperparameters. For each of the 100 genes, we used a
MLP with a single hidden layer with batch normalisation
and a single neuron as expert networks. We trained the
AME with a learning rate of 0.001 and 12 epochs of early
stopping patience on the validation set. We chose the hyperparameters based on a grid search against the validation
set. We weighted the main and the Granger-causal loss of
the AME with a 4:1 ratio. We additionally trained a FNN
All
BRCA
KIRC
COAD
LUAD
0.1

ai
0

PRAD
HM

G

1L
1

AA

AA AB AB
DA
SD C C
CL
H A17 A2
P
4

AB AB AB AB AB
CC CC CC CC CC
10 2 3 5 8

Figure 6. The importance of specific genes (coloured bars) for distinguishing between multiple cancer types as measured by average assignment of attention factors ai . We report the average attention factors over all samples (All) and over the per-cancer subsets of the test set. The grey bars spanning through the subsets
highlight the 10 most discriminatory genes by average attention
over All. We bolded the names of those genes whose associations
are substantiated by literature evidence.

baseline with matching hyperparameters (100 neurons) and
architecture. We trained each model on 35 random initialisations with the same hyperparameters.
5.3.1. R ESULTS AND D ISCUSSION
Performance. Across the 35 runs, the best AME and FNN
models both achieved classification errors of 0.62% on the
test set - reinforcing our hypothesis that providing explicit
attributions in neural networks has no adverse impact on
performance.
Attribution. When inspecting the AMEs assigned attributions, we found that the AME based its decisions primarily
on a small number of highly predictive genes for the different types of cancer (Figure 6). For 7 out of the 10 genes
with the highest average attributions, we found prior works
that report the same links between respective cancer type
and gene locus as identified by the AME (see supplementary material).

6. Limitations
Expert Convergence. It is possible that an expert does not
get weighted correspondingly to its feature group’s ability
to contribute because it is either over- or underfit. Similarly,
the expert network may simply not have enough capacity,
not the best architecture, or right choice of hyperparameters to accurately represent the value of its underlying feature group. However, one can monitor the performance and
convergence of experts through their local predictions yi .
Confounding. A Granger-causal association does not imply causation because confounders could have influenced
the attribution. Verifying causality from observational data
is impossible for this very reason (Pearl et al., 2009).
Nonetheless, as demonstrated by our experiments, AME’s
ability to quantify predictive associations between inputs
and outputs across decisions can provide valuable explanations and help generate insights from observational data.

7. Conclusion
We present a novel approach towards attribution in neural
networks that is built on the intuitive idea of distributing
the feature groups of interest among experts in a mixture of
experts model that uses attentive gating to assign weights to
individual experts. We show that the assigned attributions
contain valuable information and that providing such attributions does not adversely impact predictive performance.
Our approach enables new and exciting use cases for neural
networks in knowledge discovery from observational data
by providing measurably accurate attributions that can be
compared across decisions.

Granger-causal Attentive Mixtures of Experts

Acknowledgments
This work was partially funded by the Swiss National
Science Foundation (SNSF) project No. 167302 within
the National Research Program (NRP) 75 “Big Data” and
SNSF project No. 150640. Contains public sector information licensed under the Open Government Licence (OGL).

References
Adler, Philip, Falk, Casey, Friedler, Sorelle A, Rybeck,
Gabriel, Scheidegger, Carlos, Smith, Brandon, and
Venkatasubramanian, Suresh. Auditing black-box models for indirect influence. In Data Mining (ICDM), 2016
IEEE 16th International Conference on, pp. 1–10. IEEE,
2016.

Goodman, Bryce and Flaxman, Seth. European union regulations on algorithmic decision-making and a” right to
explanation”. arXiv preprint arXiv:1606.08813, 2016.
Granger, Clive WJ. Investigating causal relations by econometric models and cross-spectral methods. Econometrica: Journal of the Econometric Society, pp. 424–438,
1969.
Hardt, Moritz, Price, Eric, Srebro, Nati, et al. Equality
of opportunity in supervised learning. In Advances in
Neural Information Processing Systems, pp. 3315–3323,
2016.
Hastie, Trevor and Tibshirani, Robert. Generalized additive
models. Wiley Online Library, 1990.

Ba, Jimmy, Mnih, Volodymyr, and Kavukcuoglu, Koray.
Multiple object recognition with visual attention. arXiv
preprint arXiv:1412.7755, 2014.

Hochreiter, Sepp and Schmidhuber, Jürgen. Long shortterm memory. Neural computation, 9(8):1735–1780,
1997.

Baehrens, David, Schroeter, Timon, Harmeling, Stefan,
Kawanabe, Motoaki, Hansen, Katja, and Müller, KlausRobert. How to explain individual classification decisions. Journal of Machine Learning Research, 11(Jun):
1803–1831, 2010.

Hyndman, Rob J, Khandakar, Yeasmin, et al. Automatic
time series for forecasting: the forecast package for R.
Monash University, Department of Econometrics and
Business Statistics, 2007.

Bahdanau, Dzmitry, Cho, Kyunghyun, and Bengio,
Yoshua. Neural machine translation by jointly learning
to align and translate. In International Conference on
Learning Representations, 2015.

Ioffe, Sergey and Szegedy, Christian. Batch normalization:
Accelerating deep network training by reducing internal
covariate shift. In International Conference on Machine
Learning, pp. 448–456, 2015.

Bengio, Emmanuel, Bacon, Pierre-Luc, Pineau, Joelle,
and Precup, Doina.
Conditional computation in
neural networks for faster models. arXiv preprint
arXiv:1511.06297, 2015.

Jacobs, Robert A, Jordan, Michael I, Nowlan, Steven J, and
Hinton, Geoffrey E. Adaptive mixtures of local experts.
Neural computation, 3(1):79–87, 1991.

Benjamini, Yoav and Yekutieli, Daniel. The control of
the false discovery rate in multiple testing under dependency. Annals of statistics, pp. 1165–1188, 2001.
Choi, Edward, Bahadori, Mohammad Taha, Sun, Jimeng,
Kulas, Joshua, Schuetz, Andy, and Stewart, Walter. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in
Neural Information Processing Systems, pp. 3504–3512,
2016.
Doshi-Velez, Finale and Kim, Been. Towards a rigorous
science of interpretable machine learning. arXiv preprint
arXiv:1702.08608, 2017.
Flores, Benito E. A pragmatic view of accuracy measurement in forecasting. Omega, 14(2):93–98, 1986.
Friedman, Jerome H. Stochastic gradient boosting. Computational Statistics & Data Analysis, 38(4):367–378,
2002.

Jordan, Michael I and Jacobs, Robert A. Hierarchical mixtures of experts and the EM algorithm. Neural computation, 6(2):181–214, 1994.
Kendall, Maurice G. The treatment of ties in ranking problems. Biometrika, pp. 239–251, 1945.
Kindermans, Pieter-Jan, Hooker, Sara, Adebayo, Julius,
Alber, Maximilian, Schütt, Kristof T, Dähne, Sven, Erhan, Dumitru, and Kim, Been. The (un) reliability of
saliency methods. arXiv preprint arXiv:1711.00867,
2017.
Koh, Pang Wei and Liang, Percy. Understanding blackbox predictions via influence functions. International
Conference of Machine Learning, 2017.
Lazo, Jeffrey K, Lawson, Megan, Larsen, Peter H, and
Waldman, Donald M. Us economic sensitivity to
weather variability. Bulletin of the American Meteorological Society, 92(6):709–720, 2011.

Granger-causal Attentive Mixtures of Experts

Lipscomb, Carolyn E. Medical subject headings (MeSH).
Bulletin of the Medical Library Association, 88(3):265,
2000.
Little, Max A, McSharry, Patrick E, Hunter, Eric J, Spielman, Jennifer, Ramig, Lorraine O, et al. Suitability of
dysphonia measurements for telemonitoring of parkinson’s disease. IEEE Transactions on Biomedical Engineering, 56(4):1015–1022, 2009.
Montavon, Grégoire, Lapuschkin, Sebastian, Binder,
Alexander, Samek, Wojciech, and Müller, Klaus-Robert.
Explaining nonlinear classification decisions with deep
taylor decomposition. Pattern Recognition, 65:211–222,
2017.
OECD.
OECD Regional Statistics (database).
http://stats.oecd.org/ (Accessed 10th Oct 2017),
2017.
Pearl, Judea et al. Causal inference in statistics: An
overview. Statistics surveys, 3:96–146, 2009.
Ribeiro, Marco Tulio, Singh, Sameer, and Guestrin, Carlos.
Why should i trust you?: Explaining the predictions of
any classifier. In Proceedings of the 22nd ACM SIGKDD
International Conference on Knowledge Discovery and
Data Mining, pp. 1135–1144. ACM, 2016.
Riemer, Matthew, Vempaty, Aditya, Calmon, Flavio,
Heath, Fenno, Hull, Richard, and Khabiri, Elham. Correcting forecasts with multifactor neural attention. In International Conference on Machine Learning, pp. 3010–
3019, 2016.
Schwab, Patrick, Scebba, Gaetano C., Zhang, Jia, Delai,
Marco, and Karlen, Walter. Beat by Beat: Classifying
Cardiac Arrhythmias with Recurrent Neural Networks.
In Computing in Cardiology, 2017.
Selvaraju, Ramprasaath R, Das, Abhishek, Vedantam, Ramakrishna, Cogswell, Michael, Parikh, Devi, and Batra,
Dhruv. Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization. arXiv preprint arXiv:1610.02391, 2016.
Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof,
Davis, Andy, Le, Quoc, Hinton, Geoffrey, and Dean,
Jeff.
Outrageously large neural networks: The
sparsely-gated mixture-of-experts layer. arXiv preprint
arXiv:1701.06538, 2017.
Shrikumar, Avanti, Greenside, Peyton, Shcherbina, Anna,
and Kundaje, Anshul. Learning important features
through propagating activation differences. International Conference of Machine Learning, 2017.

Simonyan, Karen, Vedaldi, Andrea, and Zisserman, Andrew. Deep inside convolutional networks: Visualising
image classification models and saliency maps. International Conference on Learning Representations, 2014.
Smilkov, Daniel, Thorat, Nikhil, Kim, Been, Viégas, Fernanda, and Wattenberg, Martin. Smoothgrad: removing
noise by adding noise. arXiv preprint arXiv:1706.03825,
2017.
Srivastava, Rupesh K, Greff, Klaus, and Schmidhuber,
Jürgen. Training very deep networks. In Advances in
Neural Information Processing Systems, pp. 2377–2385,
2015.
Stollenga, Marijn F, Masci, Jonathan, Gomez, Faustino,
and Schmidhuber, Jürgen. Deep networks with internal
selective attention through feedback connections. In Advances in Neural Information Processing Systems, pp.
3545–3553, 2014.
Sundararajan, Mukund, Taly, Ankur, and Yan, Qiqi. Axiomatic attribution for deep networks. International
Conference on Machine Learning, 2017.
Weinstein, John N, Collisson, Eric A, Mills, Gordon B,
Shaw, Kenna R Mills, Ozenberger, Brad A, Ellrott, Kyle,
Shmulevich, Ilya, Sander, Chris, Stuart, Joshua M, Network, Cancer Genome Atlas Research, et al. The cancer
genome atlas pan-cancer analysis project. Nature genetics, 45(10):1113–1120, 2013.
Xu, Kelvin, Ba, Jimmy, Kiros, Ryan, Cho, Kyunghyun,
Courville, Aaron, Salakhudinov, Ruslan, Zemel, Rich,
and Bengio, Yoshua. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pp. 2048–2057,
2015.
Yang, Zichao, Yang, Diyi, Dyer, Chris, He, Xiaodong,
Smola, Alexander J, and Hovy, Eduard H. Hierarchical
Attention Networks for Document Classification. In Proceedings of the 2016 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489,
2016.
Zeiler, Matthew D and Fergus, Rob. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818–833. Springer, 2014.
Zhang, Zizhao, Xie, Yuanpu, Xing, Fuyong, McGough,
Mason, and Yang, Lin. MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network.
In International Conference on Computer Vision and
Pattern Recognition, 2017.

Supplementary Material for:
”Granger-causal Attentive Mixtures of Experts”
Patrick Schwab 1 Walter Karlen 1

1. Source Code and Full List of Outputs
The source code for all experiments is available online at
https://github.com/d909b/attentive_mixtures_of_experts. A full list of expert mean contribution
percentages (MCPs), average attention factors ai and their respective effect sizes (Kendall’s τ ) and significances (p-values)
for the medical prescription demand forecasting experiment are also available at the same address.

2. Appendix: Medical Prescription Demand Experiment

Choice of Time Frame. We chose the evaluated time frame because overlapping and continuous historical data was
available from all listed data sources.
Closed Practices. For our purposes, we define closed practices as those practices that have had no revenue for more than
3 continuous months throughout the 24 month timeframe that we analysed.
MeSH Term Selection. We extracted all MeSH terms from the categories ’Diseases’ (C), ’Drugs and Chemicals’ (D) and
’Mental Disorders’ (F03).

3. Appendix: Gene Expression Experiment
For 7 out of the 10 most predictive genes (by average attention assigned over all samples), we found prior reports that
suggest a link between those genes and the specific types of cancers they were most predictive for (by average attention
over the cancer type subsets):
• ABCC8 and breast carcinoma (BRCA) (Lehman et al., 2008; Soucek et al., 2015; Kim et al., 2017)
• ABCC5 and clear cell carcinoma (KIRC) (Eggen et al., 2012; Ween et al., 2015)
• HMG1L1 (also known as CTCF) and colon adenocarcinoma (COAD) (Giannakis et al., 2016)
• ABCA2 and COAD (Mack et al., 2007; 2008)
• ABCC10 and lung adenocarcinoma (LUAD) (Wangari-Talbot & Hopper-Borge, 2013)
• ABCC2 and ABCC3 for prostate adenocarcinoma (PRAD) (Karatas et al., 2016)

1

Institute of Robotics and Intelligent Systems, MHSL, DHEST, ETH Zurich, Switzerland. Correspondence to: Patrick
Schwab <patrick.schwab@hest.ethz.ch>.
arXiv preprint.
February 6, 2018.
Copyright 2018 by the author(s).

Supplementary Material: Granger-causal Attentive Mixtures of Experts
Table S1. Full list of input features used in the multivariate models in the medical prescription demand forecasting experiment.
Input Feature
(a) Demand History
Monthly Prescription item revenue history
Min. and max. revenue
Monthly total revenues at practice
Month of forecast (one-hot)
TL3 Region (one-hot)
Prescription item type (embedding)
(b) Online Search Interest
Monthly online search interest in 9,225 search terms (PCA)
Prescription item type (embedding)
(c) Regional Weather
Monthly weather data
Distance to weather station
Prescription item type (embedding)
(d) Regional Demographic Data
Distance to TL3 region
Prescription Item type (embedding)
Note: All of the following features were aggregated into a single feature vector using a PCA transform.
Population, Number of people
Age-adjusted mortality rate, Deaths per 1,000 population
Crude youth death rate in age band 0 - 14 years, Deaths per 100,000 population
Crude death rate, Deaths per 100,000 population
Population in age band 0 - 4 years
Population in age band 5 - 9 years
Population in age band 10 - 14 years
Population in age band 15 - 19 years
Population in age band 20 - 24 years
Population in age band 25 - 29 years
Population in age band 30 - 34 years
Population in age band 35 - 39 years
Population in age band 40 - 44 years
Population in age band 45 - 49 years
Population in age band 50 - 54 years
Population in age band 55 - 59 years
Population in age band 60 - 64 years
Population in age band 65 - 69 years
Population in age band 70 - 74 years
Population in age band 75 - 79 years
Population in age band 80 - max. years
Population in age band 0 - 14 years
Population in age band 15 - 64 years
Population in age band 65 - max. years
Deaths, age 0 - 4 years
Deaths, age 5 - 9 years
Deaths, age 10 - 14 years
Deaths, age 15 - 19 years
Deaths, age 20 - 24 years
Deaths, age 25 - 29 years
Deaths, age 30 - 34 years
Deaths, age 35 - 39 years
Deaths, age 40 - 44 years
Deaths, age 45 - 49 years
Deaths, age 50 - 54 years
Deaths, age 55 - 59 years
Deaths, age 60 - 64 years
Deaths, age 65 - 69 years
Deaths, age 70 - 74 years
Deaths, age 75 - 79 years
Deaths, age 80 - max. years
Deaths, age 0 - 14 years
Deaths, age 15 - 64 years
Deaths, age 65 - max. years
Deaths, all ages

Dimensionality
12×1
2
12×1
12
145
45
12×23
45
12×5
1
45
1
45
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1

Supplementary Material: Granger-causal Attentive Mixtures of Experts

Table S2. Full list of input features used in the multivariate models in the medical prescription demand forecasting experiment. (cont.)
Input Feature
Dimensionality
(e) Regional Economic Data
Distance to TL3 region
1
Prescription item type (embedding)
45
Note: All of the following features were aggregated into a single feature vector using a PCA transform.
Regional Gross Domestic Product (PDF), Pound Sterling
1
Disposable Household Income, Pound Sterling
1
Primary income of private Households, Pound Sterling
1
Regional employment, Number of people
1
Employment in information and communication, Number of people
1
Employment in financial and insurance, Number of people
1
Employment in real estate, Number of people
1
Employment in manufacturing, Number of people
1
Employment in construction, Number of people
1
Employment in agriculture, forestry and fishing, Number of people
1
Employment in distributive trade, repairs, transport, accommodation, food service, Number of people
1
Employment in professional, scientific, technical, administrative, support service, Number of people
1
Employment in public administration, compulsory secondary school, education, human health, Number of people
1
Employment in industry, including energy, Number of people
1
Employment in other services, Number of people
1
Regional Gross Value Added (GVA), Pound Sterling
1
GVA in information and communication, Pound Sterling
1
GVA in financial and insurance, Pound Sterling
1
GVA in real estate, Pound Sterling
1
GVA in manufacturing, Pound Sterling
1
GVA in construction, Pound Sterling
1
GVA in agriculture, Pound Sterling
1
GVA in distributive trade, repairs, transport, accommodation, food service, Pound Sterling
1
GVA in professional, scientific, technical, administrative, support service, Pound Sterling
1
GVA in public administration, compulsory secondary school, education, human health, Pound Sterling
1
GVA in industry, including energy, Pound Sterling
1
GVA in other services, Pound Sterling
1
(f) Regional Labor Data
Distance to TL3 region
1
Prescription item type (embedding)
45
Note: All of the following features were aggregated into a single feature vector using a PCA transform.
Unemployed, Number of people
1
Unemployment rate, % unemployed over labour force age 15+
1
Unemployment rate, age 15+, Growth index
1
Unemployment rate, age 15+, Growth index (2007=100)
1
Working age population, age 15-64, Number of people
1
Working age population, age 15+, Number of people
1
Labor force, age 15+, Number of people
1
Participation rate, age 15+, % labour force age 15+ over population age 15+
1
Participation rate, age 15+, % labour force age 15+ over population age 15+, Growth rate (2001=100)
1
Participation rate, age 15+, % labour force age 15+ over population age 15+, Growth rate (2007=100)
1
Employment in age band 15 - max. years, Number of people
1
Employment in age band 15 - 64 years, Number of people
1
Employment rate, % employment age 15 - 64 over working age population 15 - 64
1
Employment rate, % employment age 15 - max. over working age population 15 - max.
1
Employment rate growth, 15 years old and over, Index (2001=100)
1
Employment rate growth, 15 years old and over, Index (2007=100)
1

Supplementary Material: Granger-causal Attentive Mixtures of Experts

⨉

M experts

Expert Ei
Input xi

...

(Di)

Fully Connected (Ni)
BN

⨉

ReLu

L

...

(Ni)

...
...

(Ni)

Auxiliaryi
Fully Connected (1)
(1)
yi

ai

(1)
(1)

(Ni+1)

(Ni+M)

Concatenate
(∑i (Ni+1))
Attentive Gating
Network Gi

(1)

Figure S1. An illustration of the architecture of the AME presented in the medical prescription demand forecasting experiment. The
AME consists of M = 6 (architecturally equivalent) experts Ei that each operate on their own distinct input feature group xi of
dimensionality Di after collapsing the temporal dimension (see S1 for the exhaustive list of inputs for each expert). The ”...” layer
indicates the input-dependant data transformation that collapses the temporal dimension for those input features that have a temporal
dimension. Each expert has its own respective number of units per hidden layer Ni with the hidden blocks repeated L times. Note that
the evaluated feedforward neural networks (FNNs) use the same architecture as a single expert Ei of the AME that receives all inputs
xi at once and does not have its output modulated by an attentive gating network. For the evaluated recurrent neural networks (RNNs),
we use the same architecture as for the FNNs but do not collapse the input features x in time and replace the first fully connected hidden
block with a bidirectional long short-term memory (BLSTM) layer. The numbers in parentheses indicate the dimensionality of the
connections between components of the AME.

Supplementary Material: Granger-causal Attentive Mixtures of Experts

References
Eggen, Tormod, Sager, Georg, Berg, Thomas, Nergaard,
Birgitte, Moe, Bjørn T.G., and Ørbo, Anne. Increased
gene expression of the ABCC5 transporter without distinct changes in the expression of PDE5 in human cervical cancer cells during growth. Anticancer research, 32
(8):3055–3061, 2012.
Giannakis, Marios, Mu, Xinmeng Jasmine, Shukla, Sachet A, Qian, Zhi Rong, Cohen, Ofir, Nishihara, Reiko,
Bahl, Samira, Cao, Yin, Amin-Mansour, Ali, Yamauchi,
Mai, et al. Genomic correlates of immune-cell infiltrates
in colorectal carcinoma. Cell reports, 15(4):857–865,
2016.
Karatas, Omer F, Guzel, Esra, Duz, Mehmet B, Ittmann,
Michael, and Ozen, Mustafa. The role of ATP-binding
cassette transporter genes in the progression of prostate
cancer. The Prostate, 76(5):434–444, 2016.
Kim, Ga Ram, Ku, You Jin, Cho, Soon Gu, Kim, Sei Joong,
and Min, Byung Soh. Associations between gene expression profiles of invasive breast cancer and breast imaging
reporting and data system MRI lexicon. Annals of surgical treatment and research, 93(1):18–26, 2017.
Lehman, Teresa, Mallireddy, Vineela, Seddon, Michael,
Modali, Rama, and Ratnasinghe, Luke.
Association between an ABCC8/SUR1 polymorphism and
breast cancer risk. Cancer Research, 68(9 Supplement):1934–1934, 2008. ISSN 0008-5472. URL
http://cancerres.aacrjournals.org/
content/68/9_Supplement/1934.
Mack, Jody T, Townsend, Danyelle M, Beljanski, Vladimir,
and Tew, Kenneth D. The ABCA2 transporter: intracellular roles in trafficking and metabolism of ldl-derived
cholesterol and sterol-related compounds. Current drug
metabolism, 8(1):47–57, 2007.
Mack, Jody T, Brown, Carol B, and Tew, Kenneth D.
ABCA2 as a therapeutic target in cancer and nervous
system disorders. Expert opinion on therapeutic targets,
12(4):491–504, 2008.
Soucek, P, Hlavac, V, Elsnerova, K, Vaclavikova, R,
Kozevnikovova, R, and Raus, K. Whole exome sequencing analysis of ABCC8 and ABCD2 genes associating
with clinical course of breast carcinoma. Physiological
research, 64:S549, 2015.
Wangari-Talbot, Janet and Hopper-Borge, Elizabeth. Drug
resistance mechanisms in non-small cell lung carcinoma.
Journal of cancer research updates, 2(4):265, 2013.

Ween, MP, Armstrong, MA, Oehler, MK, and Ricciardelli,
C. The role of ABC transporters in ovarian cancer progression and chemoresistance. Critical reviews in oncology/hematology, 96(2):220–256, 2015.

