Causal Probabilistic Spatio-temporal Fusion Transformers in Two-sided Ride-Hailing Markets
Authors: Shixiang Wan
Shikai Luo
Hongtu Zhu
Venue:  (2021)
Keywords: Spatio-temporal Prediction
Causal Inference
Efficient Transformers
Two-sided Markets

Abstract: Achieving accurate spatio-temporal predictions in large-scale systems is extremely valuable in many real-world applications, such as weather forecasts, retail forecasting, and urban traffic forecasting. So far, most existing methods for multi-horizon, multi-task and multi-target predictions select important predicting variables via their correlations with responses, and thus it is highly possible that many forecasting models generated from those methods are not causal, leading to poor interpretability. The aim of this paper is to develop a collaborative causal spatio-temporal fusion transformer, named CausalTrans, to establish the collaborative causal effects of predictors on multiple forecasting targets, such as supply and demand in ride-sharing platforms. Specifically, we integrate the causal attention with the Conditional Average Treatment Effect (CATE) estimation method for causal inference. Moreover, we propose a novel and fast multi-head attention evolved from Taylor expansion instead of softmax, reducing time complexity from $O(\mathcal{V}^2)$ to $O(\mathcal{V})$, where $\mathcal{V}$ is the number of nodes in a graph. We further design a spatial graph fusion mechanism to significantly reduce the parameters' scale. We conduct a wide range of experiments to demonstrate the interpretability of causal attention, the effectiveness of various model components, and the time efficiency of our CausalTrans. As shown in these experiments, our CausalTrans framework can achieve up to 15$\%$ error reduction compared with various baseline methods. 

Under review conference paper at ICLR 2021
CAUSAL PROBABILISTIC SPATIO-TEMPORAL FUSION TRANSFORMERS IN TWO-SIDED RIDE-HAILING MARKETS
Anonymous authors Paper under double-blind review
ABSTRACT
Achieving accurate apatio-temport predictions in large-scale systems is extremely valuable in many real-world applications, such as weather forecasts, retail forecasting, and urban traffic forecasting. So far, most existing methods for multi-horizon, multi-task and multi- target predictions select important predicting variables via their correlations with responses of interest, and thus it is highly possible that many forecasting models generated from those methods are not causal, leading to poor interpretability. The aim of this paper is to develop a collaborative causal spatio-temporal fusion transformer, named Causal Trans, to establish the collaborative causal effects of predictors on multiple forecasting targets, such as supply and demand in ride-sharing platforms. Specifically, we integrate the causal attention with the Conditional Average Treatment Effect (CATE) estimation method in causal inference. Moreover, we propose novel and fast multi-head attention evolved from Taylor's expansion instead of softmax, reducing time complexity from O(V2) to O(V), where the number of nodes in a graph. We further design a spatial graph fusion mechanism to significantly reduce the parameters' scale. We conduct wide range of experiments to demonstrate the interpretability of causal attention, the effectiveness of various model components, and the time efficiency of our Causal Trans. As shown in these experiments, our Cause 1 Trans framework can achieve up to 15% error reduction compared with various baseline methods.
INTRODUCTION
This paper is motivated by solving collaborative probabilistic forecasting problem of both supply and demand in two-sided ride-hailing platforms, such as Uber and DiDi. Collaborative supply and demand relationships are common in various two-sided markets, such Amazon, Airbnb, and eBay. We consider two-sided ride-hailing platforms as an example. In this case, we denote supply and demand as online driver number and call orders, respectively, on the platform specific time city. Some major factors for demand include rush hours, weekdays, weather conditions, transportation network, points of interest, and holidays. For instance, rains during peak hours weekdays, demand will dramatically increase and last for a certain time period. In contrast, some major factors for supply include weather, holidays, traffic condition, weekdays, and platform's 's dispatching and repositioning policies. Moreover, supply tends to gradually cover the area with many unsatisfied orders, that is, the distribution of supply tends to match with that of demand.
We are interested in establishing collaborative causal forecasting models for demand and supply by using various predictors (or covariates). Although many learning methods have been developed to address various collaborative prediction tasks, such spatio-temporal traffic flow prediction (Zhu & Laptev, 2017: Du et al., 2018; Zhang et al., 2019b; Ermagun & Levinson, 2018; Luo et al., 2019), multivariate prediction (Bahadori et al., 2014; Liang al., 2018), multi-task prediction (Tang al., 2018; Chen et al., 2018; Chandra et al.,

---

Under review conference paper at ICLR 2021
2017), multi-view prediction (Yao al., 2018), and multi-horizoon prediction (Lim et al., 2019; Yu et al., 2020), these existing methods primarily select important predictors via their correlations with responses, leading to many forecasting models with poor interpretability.. In contrast, we propose Causal Trans: a Collaborative Spatio-temporal Fusion Transformer, that generates causal probabilistic multi-horizon forecasts. To the best of our knowledge, this is the first work that captures collaborative causal effects of external covariates on multiple forecasting targets. Building such models is not only essential to enhancing forecasting performance, but also helps the platform to utilize various platform policies to match the distribution of supply with that of demand in two-sided markets.
In the Causal Trans framework, our major contributions are summarized as follows:
We design the causal attention based on double machine learning (Chernozhukoo et al., 2018) with two layers fully connected neural networks, and successful apply it to various large-scale time series forecasting problems. We conduct a wide range of experiments on real world datasets with multiple covariates and demonstrate that Causa rans with causal atlention outperforms many baseline models in various Ride-hailing scenarios.
We propose spatial fusion mechanism based on graph attention networks (GAT) (Veličković et al., 2017) to gather local regions and enhance robustness adjacent regions always share similar supply and demand patterns.
We propose an approximate time-efficient Taylor expansion attention to replace softmar in multi- head attention of Transformers (Vaswani et al., 2017) such that time complexity reduces from O(V²) to O(V). We carry out two groups of experiments with three multi-heads and five multi-heads to verify such efficiency improvement.
2 RELATED WORK
There is large body of literature on vehicle flow forecasting (Zhu & Laptev, 2017; Bahadori et al., 2014; Tang et al., 2018; Lim et al., 2019; Yao et al., 2018). We selectively review several major methods as follows. In Zhu & Laptev (2017), the time series forecasting task as a two-step procedure includes offline pre-training and online forecasting. The offline pre-training step is encoder-decoder framework for compressing sequential features and extracting principal components, whereas the second step gives explainable prediction changes under external variables. Bahadori et al. (2014) proposed a unified low-rank tensor learning framework for multivariate spatio-tempon analysis by combining various attributes of spatio-temporal data including spatial clustering and shared variables structure. For multi-step traffic flow prediction, Tang al. (2018) proposed a spatio-temporal multi-task collaborative learning model to extract and learn shared information among multiple prediction tasks collaboratively. For example, such model combines spatial features collected from offline observation stations and inherent information between blended time granularities. Lim et al. (2019) proposed temporal fusion transformer (TFT) to capture temporal correlations at each position, which was similar to self-attention mechanism and expected to capture long-term and short-term dependencies. Yao et al. (2018) proposed deep multi-view spatio-temporal network (DMVST-Net) including speed viewpoint (modeling the correlation between historical and future demand by LSTM (Gers & Schmidhuber, 2001)), a spatial viewpoint (modeling local spatial correlation by CNN), and a contextual viewpoint (modeling regional correlations in local temporal patterns). Overall, all above methods improve time series fitting by learning and predicting correlations across multiple
perspectives, targets, and tasks.
However, those methods lack convincing interpretability of "how and to what extent external variables affect supply and demand".. Achieving good demand forecasting involves not only historical demand targets, but also various current external variables (e.g., weather conditions, traffic conditions, holidays, and driver reposition). Those historical demand observations were affected by historical external factors, so the demand forecasting only based on correlation between variables is hardly convincing. Furthermore, supply forecasting
2

---

Under review conference paper at ICLR 2021
is empirically affected by the distribution of demand besides current external variables. Establishing causal relationship between (supply, demand) and multiple external variables is critically important for accurate supply and demand forecasting.
3 METHODOLOGY
We introduce the CausalTrana framework to efficiently establish the collaborative causal effects of multiple predictors on spatio-temporal supply and demand below.
3.1 COLLABORATIVE SUPPLY AND DEMAND FORECASTING
We consider all related observations including supply, demand, and external variables collected in a city. Each day is divided into 24 hour segments and city is divided into non-overlapping hexagonal regions (side length ranges from 600 to 1000 meters). The complete data consists of demand z (t) E R, supply (e E R, and dynamic covariates (t) E R² where & is a specific hour segment and v E V is specific hexagon of the set of hexagonal regions, denoted as V. Dynamic covariates includes weather, holidays, social events, POI (Point Of Interests), and government policies. Weather features consist of temperature (*C), rainfall (mma), wind level and PM2.5 <mg/m²) Holiday features are represented by one-hot boolean vectors, including seasons, weekdays, s. and national and popular holidays, such as Christmas Day. POI features are represented by the number of various positions, including traffic stations, business districts, communities, hospitals and schools. More detail cases about collaborative supply and demand are provided in Appendix A.
The problem of interest is to use all available observations in [xr(( t), Vr(: t), t)(, v V} to predict (t + 1: 8 Tmaa)s (t + 1: +- Tmaa)), v E V}, where Tmaz is a pre-specified time length, x (41 : (2) and yo (4, : (22) are the demand and supply vectors starting from time point ( to time point 12, and x, (2) and (o 12) are the demand and supply vectors starting from the earliest time point to time point t. The demand Ty may depend on historical supply Yo that happens several weeks (or even longer) ago. But in the latest several weeks (training period), based on our understanding of ride-sharing business, demand x, may be primarily influenced by its own recent historical patterns. Based the above description, formulate the learning problem of collaborative demand and supply forecasting as follows:
((x,(t + 1: + Tmaz) (: t), 2e tt++ Tmaz));
(1)
Ply. (t + 1: ++ Taaz)) (: 6), (: + Tmar), Zo(: (: 1 + Tmmax))
where P(-I-) is a conditional distribution. In (I), is assumed that (t + + Tmaz) is primarily affected by historical demands in [:( t) and external covariates in ( + Tmaa)) Furthermore, in (2), it is assumed that future supplies in (t (f+1 1 it Tmaa) are primarily affected by historical supplies in yo (: t), demand patterns in (. & + Tmaz)) and external covariates in (; ++ Tman). Comparing (1) with (2), we assume that the distribution supply during [4 + 1, l + Tmax] driven by the historical and current distributions of demand besides the historical information in yo (: t) and external covariates in 2. t Tmaz).
3.2 PROBABILISTIC FORECASTING
Most time series forecasting methods produce deterministic values, whereas forecasting results might have large variation and were hardly robust due to the variation of covariates and training process. To enhance forecasting reliability, we adapt the quantile loss function with the Poisson distribution as our final optimization function Empirically, following (Salinas et al., 2020; Wen et al., 2017; Li et al., 2019; Lim et al., 2019),
we choose three quantile points q E Q {10% 50%, 90%), in which the gap between forecasting values
Ride-hailing supply and demand variables approximately follow with the Poisson distribution.
3

---

Under review conference paper at ICLR 2021
90% and 10% percentiles can be regarded the confidence interval. Take demand zy forecasting at time point i as example, the final quantile loss function is given by
LQ= - ΣE
M Tmaa
BIE qEQ
where (Z #f) I I (x) 2 ). St is the training dataset, Tiaa is the maximum prediction step, and (() is an indicator function. For a fair comparison, given the test dataset Ω we employ q-risk (Salinas et al., 2020; Lim al., 2019; Li al., 2019), denoted as Rg. to evaluate the risk level of each quantile point as follows:
R₂ 20, Emmae (1r,
(4)
Ex.e Timas [T
There are at least two advantages of using quantile loss function. First, the quantile loss function is more robust and stable than the mean square error the hinge loss, especially when forecasting targets have large variation. Second, we can modify external covariates to change the confidence interval causal attention and analyze real-world cases.
3.3 CAUSAL TRANSFORMER FRAMEWORK
Output (quantle los) Decodar
Encoder
T.A.
T.A.
Self-ckstaring
Input/Targeta
GCN
Clustered Outpus
Fast S.F.
Fast S.F.
O
+ MYPI
Memand/suppy
(demand/uuppy)
(N) Drassl framework
S.F. feett spatial graph fuaion
Decodar
T.A.
Imputs
Solf Adzentiiinnn
Inputs
Sell
By. Syr and
Foee) Forward
Ere--Dec Atantion
C.A. Arantics
Connetette
Attrttin
Clustored
Food
Triull
Atarrion
Oulpur
Forward
Output Sequinces
Forward
Output
Oupul
(4) C.A. crusal attenfon unts
(d) T.A.: attention units
The overview of CausalTrans framework. Demand and supply are trained separately in sequence. (a). The framework consists of three essential components: Fast S.F. (fast graph spatial fusion), C.A. (causal attention), and T.A. (temporal attention). Moreover, we employ the average quantile loss distributed from (10%, 50%, 90%) to optimize forecasting probabilistic distributions. (b). The Fast S.F. consists of self-clustering with GAT and fast attention. (c). The C.A. applies offline trained causal weights 0 to online treatments evaluations. (d). The T.A. aims to keep ordering self self-attentions.
Figure 1:
4

---

Under review conference paper at ICLR 2021
Our CausalTrans is a novel combination of causal estimators and the encoder-decoder architecture. Figure 1 shows the overview of the CausalTrans framework. The three key novel contributions of Causal Trans include fast spatial graph fusion, causal attention, and temporal attention units. First, from the spatial perspective, Causa Trans gathers set of graph attention kernels (GAT) by using assignment scores extracted from temporal patterns. Moreover, we adapt the first-order Taylor's expansion on multi-head attention from transformer to reduce time complexity from square complexity to linear complexity. Second, from the temporal perspective, causal attention based on sufficient historical observations is trained offline to evaluate the causal weights on peek time slots, those weather conditions, and those holidays. which are denoted as 0r. Ow, and fzz, respectively. under diverse spatio-temporal conditions. Furthermore, we simplify three seasonal perspectives (week, month, and holidays) to represent multi-view position encoding (MVPE). Third, temporal attention is used to fill the gap between encoder and decoder, in which we add a sequence mask to ensure that the historical observations of time point only uses observations smaller than L. We set mask out to be -oo and illegal connection weights to be zero. In the following subsections, we introduce the main components of CausalTrans: fast spatial graph fusion and causal attention and show how they works together as a causal spatio-temporal predictor. Moreover, for notational simplicity, we focus on describing those components for forecasting demand Ty in the following subsections, while avoid repeating the same components for supply Ve-
3.4 FAST SPATIAL GRAPH FUSION ATTENTION
In this subsection, we describe the fast graph fusion attention unit based on region clustering and fast multi- head attention. See Figure (b) for the architecture of Fast S.F.. Since GAT has achieved impressive results in traffic forecasts (Park et al. (2019); Kosaraju et al. (2019); Zhang et al. (2019a)), we use GAT to extract contextual features in huge graphs. However, directly applying GAT to large-scale forecasting problems a challenging task, so we design spatial fusion subgraphs that share local supply and demand information. Moreover, we build our framework based on transformers (Vaswani et al., 2017). Transformers have been state-of-the-art structure in various natural language processing (NLP) tasks (Wolf et al., 2019; Wang et al., 2019) and time series forecasting due to its prominent powers of long-term feature extraction and parallel computing. However, the multi-head attention in transformers becomes key bottleneck for time efficiency. We design an approximate Taylor's 's expansion attention instead of using softmax function to accelerate matrix products. More detail results of fast attention can be found in Appendices C.2 and C.3.
We briefly describe the fast spatio- temporl fusion graph attention procedure below. First, let X, be the ppatio-temporal demand feature matrix of all grids V before time t, the temporal patterns of V are represented as assignment scores given by
(5) where Faaeh is the mean operator on the batch mode, k belongs to a K-ddimeniinall cluster vector, v E V, W. and We are, respectively, spatial and temporal weight matrices corresponding X,, and (.) and (-) are sigmoid and relu activation functions, respectively.
(Cz, kk) (o,
Batck
Second, we use the k-th spatial learner Gv (zx) to extract spatial features of sequential data x, in grid v, and the summed outputs of K. clusters are given as follows:
h2
Σ ((x)))co..k REK
(6)
The softmax function is used to get attention weights among regions as follows:
EVEN, explo, (a" x xollWW =z)))) ae
a
Σ
avpy' Fo'
EveN, exp(o,(a?|V) xx - xxrl))
(7)
EEN
5

---

Under review conference paper at ICLR 2021
where Ce is the correlation weight between v and v', a and W are network parameters, the superscript T denotes the transpose of vector or matrix, No = forle' E V, e' & 2) is the neighboring region set of region U, and [-I1-] is the concatenation operation.
In (7), the time complexity of computing exp(o,-(a" (a [ W z.r()) D) is O(V2). Specifically, the exponent operation in exp(a* W) X of the softmax function limits the efficiency of attention. Moreover, cluster number KC < V, and the time complexity of a W X is O(K² V) R O(V). Many recent studies find that linear attention is feasible for tasks, whose primary focus on short-term dependence. More details are discussed in Appendix D, Our novel linear attention is easy to implement and interpret. It follows from Taylor expansion that exp(a a" W) R 1 1+ a² W under the condition of small a W. Analogous to the self-attention in original Transformer (Vaswani al., 2017), the approximate mean and variance of QK" are O and 1, vag
respectively, so a here is limited to small values. We introduce L.2 normalization to ensure small a"W W and + a W 2 such that
T
W
exp(a' a W) ~ T(a"W) = 1 [|al|2
I|
where T is an approximate Taylor expansion. Equation (8) is close to inner dot products, which have advantages on parallel implementation and linear time complexity. Finally, a. can be transformed into
i EVEN. T(o,(a W x.Il xx -..0
EvEN. (a ( x.llW 1wl)))
3.5 CAUSAL ATTENTION MECHANISM
Many external covariates causally change the distribution of demand and supply shown in Figures 2 and 3 of the supplementary document. Meanwhile, many existing works focus on finding the correlation between external covariates and forecasting targets. For example, Li al. (2019) designed causal convolution enhance the locality of attention, whereas Lim et al. (2019) added the variables selection networks and gate mechanism to train attention weights. These two studies (Lim et al., 2019; Li al., 2019) intend calculate correlations among variables, but not causal effects under counterfactual conditions. Statistically, such issue can be regarded as heterogeneous treatment effect (HTE) problem. See Figure (c) for the architecture of C.A.. To the best of our knowledge, causal attention methods for HTE have not been proposed in large-scale spatio-temporal forecasting problems.
First, we briefly describe the conditional average treatment effect (CATE) (Abrevaya al., 2015). We still take demand vectors x. (1 (2) (abbreviated as æ in the following) of grid o starting from time point l time point to (rainfall, temperature and wind level), peek time slots and holidays. Let (ss) be the target variable under treatment 8, and is vector of other covariates. The HTE for comparing two treatment levels so and 81 is defined as
example. The X represents set of z. The treatments we consider include weather
T(50, s1;z)
(s) X (so) iz].
(10)
If treatment s is continuous, then the treatment effect is defined to be CIV.X(s))2) where V. =0/Os.
To unbiasedly estimate treatment effects, we propose causal attention module based on double machine learning (DML) (Chernozhukov et al., 2017) based on two layers non-parametric fully connected neural networks. Specifically, we assume
X(S) = (z) S + (z( + and S = 91(z) + 17.
(11)
where and 17 are independent random variables such that B[e|2] = B[]]z] = 0. go(-) and 91(+) are two non-parametrie neural networks, and ((z) is the constant marginal CATE. Let X X E(X|z) and
6

---

Under review conference paper at ICLR 2021
$ ()))), we can get
= x x = 0(z) {S E(S|))) te 0(2) S e.
(12)
Therefore, we can compute (z) by solving
(()) = arg min En [X - 0(z). ((z))
(13)
8CB
where E, denotes the empirical expectation.
Large historical data source contains all kinds of experimental environment and treatments. According to Algorithm I, given time series z.(( () at grid (o is dropped for readability) and treatment 81 E 8, loop and search two treatment levels so and 81 along with the historical timeline to construct the AB groups {a(to) [sp] and (z(ft)(ss) Then, construct the AA groups (x(to
t1)} by look-back window with the same length before to and f1, and make sure that both are both stationary processes with equal mean (PRpss > 0.05 in KPSS test (Shin & Schmidt, 1992) and P:-rest > 0.05 in T-Test both AA groups' first-order differences). Based on the selected AA/AB groups, we employ DML to estimate causal attention. In our method, trained causal attention will be inserted to transformer, and clustered regions share global each other.
to]} } and {x((1
Algorithm 1 Causal Attention Algorithm with DML
Input:
Given demand matrix æf: t) at grid v before time t. three kinds of treatments includes weekday and hour slots T(: t) = (W(: t), (:: t)}, weather vectors W(: t), and holidays one-hot vectors H(: t)
Output: causal effect coefficients Or for (: l). Ow for (:: t), and Ofr for [ L)
1: Take Or as an example, and suppose that a AA group and AB group on T(: () is TAA TAB = 0
2: for all (to) (. (f1)) (Mon, Tue,... Su.S,n), (t)) (t1) E [1....24 do
(t,),, Pr Test (tot)), (t1)) < 0.05 then 4: for all i E (o) and & E (:4} do
3: if Tw (to) = T (t1), Th (to)
5: Calculate 1st-order differences T(% fo) and T(t
6:
PR (i(t)) to)), Press(((t) t1)) and Pr-rest (T(t) to), 3(1 (1)) > 0.05 then
7:
T append(([([[(( to), 2(1 t1)))
8:
T (=)))
9:
end if
10: end for
11: end if
12: end for
Do DML on TAA and TAB datasets and estimate treatment coefficients Or
14: Repeat from Step 2 and estimate Ow and 0µ by different DML.
15: return By, fw. and H
4 EXPERIMENTS
4.1 DATASETS
We consider four datasets (Electricity, Traffic, Retail? and Ride-hailing) in experiments follows.
Electricity. Electricity contains hourly univariate electricity consumption of 370 customers. According to (Salinas al., 2020), weekly oberservations before are inputs predict the next 24 hours' series.
7

---

Under review conference paper at ICLR 2021
Table Roo/Roo losses on the electricity and traffic datasets the univariate group, where o denotes the results obtained from Li et al. (2019).
TRMP
DeepAR
DeepState ConvTrans Seq2Seq
MORNN
Cassa Trans
Electricity 0.154/0.102 0.101/0.077 0.084/
0.067/0.036" 0.161/0.099 0.167/0.115 133 0.122/0.081 0,10//0.075 0.1 17700822 0.095/1.070' 0.095/0.065
D.075/0.040
0.0830.056" 0.059/0.034"
0.077/0.036
0.055/0.027 0.056/0.029
Traffic 0.22M0.137' 0,236/0 148"
0.18
Table 2: Rao losses on the retail and ride-hailing datasets. Percentages in brackets are loss reductions between CausalTrans and the second best result. denotes results from Li al. (2019).
ConvTrans
SequSeq
DeepAR
DMVST
ST-MGCH
TFT
Retait
0.429
0,411
0.379
0.395
0.354
0.352(-0.6%)
Ride-hailing (Id, city A. Demand)
0.573
0,550
0.495
0.499
0.524
0,482
0.450
0.4341-3.7%
Ride-hailing (1d, city A, Supply)
0.482
0.453
0.428
0.422
0.445
0,421
0.415
0.393(-5.3%)
Ride-hailing (Id, city R. Demand)
0.470
0.455
0.405
0.400
0.422
0.404
0.370
0.361(-2.5%)
Ride-hailing (1d. city B, Supply)
0.426
0.404
0.388
0.384
0,388
0,378
0.357
0.341(-4.5%)
Ride-bailing (7d. city Demand)
0.613(-6.2%)
Ride hailing (7d. city A, Supply)
0.612
0.569
0,516
0.519
0,536
0.575
0,583
(.46%()9.3%)
Ride-haite hailing (7d, city R, Demand)
0.588
0.539..5.6%)
Ride Role-hailing (7d, city B, Supply)
0.568
0,519
0,499
0.501
0.503
0.525
0.528
0.454(-9.0%
Traffic. Traffic contains hourly univariate occupancy rate of 963 San Francisco bay area freeways, where the look-back rolling window and prediction step are the same as Electricity.
Retail. Retail is the Favorita Grocery Sales Dataset from Kaggle competition (Lim et al., 2019), including daily metadata with diverse products, stores and external variables. To compared with some state-of-the-an of-t methods (Lim et al., 2019; Salinas et al., 2020), historical observations across 90 days are trained to forecast product sales in the next 30 days.
Ride-hailing, The Ride-hailing dataset contains real supply, demand. and various of metadata at the hourly and hexagonal grid scale between June 2018 8 and June 2020 in two big cites (city and city B) obtained from a ride-hailing company. The first 70%, the next 10% and the remaining 20% is used for training, validation and testing, respectively.
We group the first two datasets into the univariate group and the last two datasets into the multivariate group.
4.2 BENCHMARKS
In this section, two different forecasting methods, including iterative methods and multi-horizon methods, are compared in wide range of comparison experiments. For our method Causal Trans, a pre-defined search space is used to determine optimal hyperparameters. Experimental details are included in Appendix B.
Iterative methods. Iterative methods generate multi-step prediction results by step-by-step rolling windows, where results in previous steps used to as inputs in the next step. Typically, iterative methods include DeepARt, Deep State Space Models (DeepState ete (Rangapuram et al., 2018), ARIMA (Zhang, 2003), ETS (Jain & Mallick, 2017) and TRME (Yu et al., 2016).
Muuti-horizonn methods. Multi-horizon methods considered here include ConvTrans (Li et al., 2019), MORNNT (Wen et al., 2017), Seq2Seq! (Sutskever et al., 2014), DMVST (Sutskever al., 2014), ST-MGCN (Geng et al., 2019), and TFT (Lim et al., 2019). The methods are trained by using the GluonTS (Alexandrov al., 2019) package. DMVST and ST-MCCC are spatial baselines.
8

---

Under review conference paper at ICLR 2021
Table 3: Roo losses on the retail and ride-hailing datasets. Percentages in brackets are loss reductions between CausalTrans and the second best result. denotes results from Li al. (2019).
ConvTrans
Seq2Seq
neepAR
TMVST
ST-MGCN
TFT
Cauaa Trans
Retail
0.192
0.157
0.152
0.155
0.147
0.143(-2.8%)
Ride hade-hiitng (Id, city A, Demand)
0.238
0.208
0.205
0.205
0.208
0.195
0.192
-.1644(-1.66%)
Ride-hailing (Id, city A, Supply)
0.212
0.177
0.164
0.162
0.173
0.165
0.160
0.142(-11.3%)
Ride-bailing (1d, city B. Demand)
0.208
0,176
0.159
0.158
0.170
0.157
0.155
0.145(-6.5%)
Kide-hailing (1d, city B, Supply)
0.205
0.197
0.157
0.188
0.169
0,151
0.149
0.139(-6.7%)
Ride hailing (7d, city A, Demand)
0.324
0.306
0.276
0.289
0.286
0.280
0.297
0.244
Ride hailing (7d. city A. Supply)
0.259
0,233
0.207
0.204
0.237
0,248
0.237
0.173(-15.2%)
hailing (7d, city B, Demand)
0.255
0.216()
Ride-hune hailing (7d, city B, Supply)
0.214
0.184
0.177
0.179
0.168
0.197
0.204
0.153(-8.9)
4.3 RESULTS AND DISCUSSION
We adapt the quantile loss as optimization function, and compare various results by q-risk Reo/Roo quantile point 50%/90% More detailed descriptions of probabilistic forecasting are provided in subsection 3.2.
Table includes the Roo losses of all forecasting methods for Electricity and Traffic datasets. The Electricity data set does not have any covariates and is lack of spatial information, whereas the Traffic dataset does have spatial information even without multiple covariates. We observe that ConvTrans and TFT are comparable with each other and both outperform all other methods. We believe that compared with TFT, ConvTrans is able to take advantage of the spatial information in the Traffic dataset. This is not the case for the Electricity data set.
Table 2 and Table 3 include the Rs0 and Rop losses of all multi-horizon methods in the multivariate group. We consider both one-day and seven-day predictions and optimize the hyperparameters of all methods by using grid search. We have several important observations. First, for the one-day prediction, iterative DeepAR outperforms Seq2Seq and MΩRNN due to the use Poisson distribution and weather conditions. Second, for the spatial baselines DMVST and ST ST-MGCN, R3o and Reo losses increasing with longer forecasting days, as such methods may overfit biased weights external covariates. Third, Causal Trans outperforms all other competing methods primarily due to the of the causal estimator DML. For instance, compared with the second best method, Causall l Trans yields maximum 9.3% lower R.50 and 15.2% lower Roo on the Ride-hailing (7d, city A, Supply) dataset. Fourth, Causa 1 Trans achieves lower losses forecasting supply than forecasting demand, since we explicitly model causal relationship between supply and demand in (2). Fifth, as expected, different with the one-day prediction, the seven-day prediction focuses on unbiased distribution estimation in order to alleviate error accumulation. This point of view is further reinforced by the results of the ablation study reported in Appendix C.2, and causal attention is visualized in Appendix C.I.
5 CONCLUSION
Based on causal inference theory, we develop the CausalTrans framework to address collaborative supply and demand forecasting in large-scale two-sided markets. We design the fast multi-head attention to improve the computational complexity to nearly linear O(V). Causa Trans achieves similar performance as TFT based on the two datasets in the univariate group and outperforms all competing methods including TFT the nine different experiments for the multivariate group. In particular, for our Ride-hailing datasets, achieve up 15% error reduction compared with various baseline methods. In the
Causal Trans future, we will continue to integrate causal inferences with existing deep learning methods to deal with large-scale spatio-temporal forecasting problems.

---

Under review conference paper at ICLR 2021
REFERENCES
Jason Abrevaya, Yu-Chin Hsu, and Robert P Lieli. Estimating conditional average treatment effects. Journal of Business & Economic Statistics, 33(4):4855-5) 505,, 2015.
Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al. Gluonts: Probabilistic time series models in python. arXiv preprint Std1::9900005666 2019.
Mohammad Taha Bahadori, Qi Rose Yu, and Yan Liu, Fast multivariate spatio-temporal analysis via low rank tensor learning. In Advances neural information processing systems, pp. 3491-3499, 2014.
Rohitash Chandra, Yew-Soon Ong, and Chi-Keong Goh. Co-evolutionary multi-task learning with predictive recurrence for multi-step chaotic time series prediction. Nearocomputing. 243:21-34, 2017.
Tong Chen, Hongzhi Yin, Hongxu Chen, Lin Wu, Hao Wang, Xiaofang Zhou, and Xue Li. Tada: trend alignment with dual-attention multi-task recurrent neural networks for sales prediction. In 2018 IEEE International Conference on Data Mining (ICDM), pp. 49-58. IEFE, 2018
Victor Chemozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, James Robins, et al. Double/debiased machine learning for treatment and causal parameters. Technical report, 2017.
Victor Chemozhukov. Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. 2018.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 19t4::994..000000 2019.
Zihang Dai, Guokun Lai, Yiming Yang, and Quoc V Le. Funnel-transformer: Filtering out sequential redundancy for efficient language processing. arXiv preprint arXiv:2006.03236. 2020.
Shengdong Du, Tianrui Li, and Shi-Jinn Horng. Time series forecasting using learning framework. 20/8 9th International Symposium on Parallel Architectures, Algorithms and Programming (PAAP), pp. 171-176. IEEE, 2018.
deep
Alireza Ermagun and David Levinson. Spatiotemporal traffic forecasting: review and proposed directions. Transport Reviews, 38(6):76--81 2018
Xu Geng, Yaguang Li, Leye Wang, Lingyu Zhang, Qiang Yang, Jieping Yc, and Yan Liu. Spatiotemporal multi- graph convolution network for ride-hailing demand forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3656-3663 2019.
Felix A Gers and E Schmidhuber. Lstm recurrent networks learn simple context-free and context-sensitive languages. IEEE Transactions on Neural Networks, 12(6): 1340, 2001.
Saurabh Goyal, Anamitra Roy Choudhary, Venkatesan Chakaravarthy, Saurabh ManishRaje, Yogish Sabhar- wal, and Ashish Verma. Power-bert: Accelerating bert inference for classification tasks. arXiv preprint MMXiv:2001.08950 2020.
Garima Jain and Bhawna Mallick. A study of time series models arima and ets. Available at SSRN 2898968, 2017.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. arXiv preprint arXiv:2006. /6236, 2020.
10

---

Under review conference paper at ICLR 2021
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.0444 2020.
Vineet Kosaraju, Amir Sadeghian, Roberto Martin, Ian Reid, Hamid Rezatofighi, and Silvio Savarese. Social-bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. In Advances in Neural Information Processing Systems, pp. 137-146, 2019.
Sören R Künzel, Jasjeet S Sekhon, Peter Bickel, and Bin Yu. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10); 4156-4165, 2019.
Denis Kwiatkowski, Peter CB Phillips, Peter Schmidt, Yongcheol Shin, et al. Testing the null hypothesis of stationarity against the alternative of a unit root. Journal of econometrics, 54(1-3):159-178, 1992.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In Advances in Neural Information Processing Systems, pp. 5243-5253, 2019.
Yuxuan Liang, Songyu Ke, Junbo Zhang, Xiuwen Yi, and Yu Zheng. Geoman: Multi-level attention networks for geo-sensory time series prediction. In IJCAI, pp. 3428-3434, 2018.
Bryan Lim, Sercan O Arik, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. arXiv preprint arXiv:/9/2.09363, 2019.
Xianglong Luo, Danyang Li, Yu Yang, and Shengrui Zhang. Spatiotemporal traffic flow prediction with knn and Istm. Journal of Advanced Transportation, 2019, 2019.
Cheonbok Park, Chunggi Lee, Hyojin Bahng, Kihwan Kim, Seungmin Jin, Sungahn Ko, Jaegul Choo, et Stgrat: A opatio-temporal graph attention network for traffic forecasting. arXiv preprint arXiv.1911.13181, 2019.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. In Advances in neural information processing systems, pp. 7785-7794 2018.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 2020.
Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. arXiv preprint arXiv: 18.001122 2018.
Yongcheol Shin and Peter Schmidt. The kpss stationarity test as unit root test. Economics Letters, 38(4): 387-392, 1992.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Kun Tang, Shuyan Chen, and Aemal J Khattak. A spatial-temporal multitask collaborative learning model for multistep traffic flow prediction. Transportation Research Record, 2672(45):1-13, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10900: 2017.
11

---

Under review conference paper at ICLR 2021
Chenguang Wang, Mu Li, and Alexander J Smola. Language models with transformers. arXiv preprint 1944091904.09408, 2019.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A recurrent forecaster. arXiv preprini arXiv: 1053, 2017.
quantile
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's ' transformers: State-of-the-ar natural language processing. ArXiv, pp. arXiv-1910, 2019.
Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li.
Deep muew spatial-temporal network for taxi demand prediction. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, and Shuai Yi. Spatio-temporal graph transformer networks for pedestrian trajectory prediction. arXiv preprint arXiv:2005.08514 2020.
Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. Temporal regularized matrix factorization for high- dimensional time series prediction. In Advances in neural information processing systems, pp. 847-855, 2016.
Chenhan Zhang, JQ James, and Yi Liu. Spatial-temporal graph attention networks: A deep learning approach for traffic forecasting. IEEE Access, 7:166246-166256 2019a.
G Peter Zhang. Time series forecasting using hybrid arima and neural network model. Neurocomputing, 50: 159-175, 2003.
Weibin Zhang, Yinghao Yu, Yong Qi, Feng Shu, and Yinhai Wang. Short-term traffic flow prediction based on spatio-temporal analysis and enn deep learning. Transportmetrica A: Transport Science, 15(2):1688 1711, 2019b.
Lingxue Zhu and Nikolay Laptev. Deep and confident prediction for time series at uber. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 103-110. IEEE, 2017.
12

---

Under review conference paper at ICLR 2021
A RIDE-HAILING DATASET DETAILS
Taking city A as example, supply, demand, delta' and rainfall trends (January 1st, 2018 to January 1st, 2020) are plotted at daily scale figure 2. We conclude that the variance of demand is bigger than supply, especially in raining rush hours.
Taking August 17th, 2018 8 in city A as another example figure 3, we observe that the delta at dark red regions would not be for long, spatio-temporal supply changed by corresponding demand and reposition of drivers. The ride- tiie-haiingg platform would release useful strategies to promote orders. Collaborative demand and supply implies that the distribution of supply corresponds to the distribution of demand.
supoly delta
40000
35000
30000
1.0
25000
20000
0.5
15000
0.0
-0.5
2018-01
2015:04
2018-07 2018-10
2019-01
2019-04 2010-07
2019-10 2020-01
Figure 2: Supply, demand, delta and rainfall trends in city A ranging from January 1st, 2018 January 1st, 2020. On August 17th, 2018 (Friday), demand increased significantly under heavy rains and evening peak hours. Lack of supply was far from being able to meet explosive demand. At the night peak hours on September 24th, 2018 (Mid-Al Midd-Autunn Festival), drivers' number family reunion. During the National Days (from October 1st to 7th, 2018), commuting drivers and passengers both decreased, resulting in bilateral decline of supply and demand. On December 7th, 2018 (Friday), heavy snow and low temperature stimulated potential demand. At the beginning of New Year's Eve in 2019, people were eager to reunite with families, resulting in low supply and high demand. After that, supply and demand tended to be balanced gradually.
reduced and demand increased due
B TRAINING DETAILS
Empirically, we consider determining optimal hyperparameters via pre-defined random search space. For reproducibility, we include essential hyperparameters on our Ride-hailing dataset Table 4.
Table 4: Optimal hyperparameters on Ride-hailing dataset.
Learning
Dropout
Batch size
Num, multi head
Sliding Window
Chuster K.
Optimizer
Ride-hailing (1d. city Demand)
days days days days
Ride huiling (ld. city A, Supply)
0.001
0.3
64
14
3
Adam
Ride-hailing (1d, city B, Demand)
0.001
128
Adam
Ride haillng city B, Supply)
0.001
0.3
128
14
Atamn
Ride hailing (7d, city A, Demand)
0.001
512
days
Adam
Ride-hailing (7d,
Supply)
0.001
512
days
Adam
Ride-hailing (7d, city B, Demand)
0.001
0.1
512
days
28
Adam
hailing (7d. city Supply)
days
City A a big city in China. "Supply minus demand is delta.
13

---

Under review conference paper at ICLR 2021
(a) 17:00 pm.
(b) 18:00 pm.
(c) 19:00 pm.
Figure 3: The delta heat maps at different o' clock of city A on August 17th, 2018. Panels (a), (b) and (c) show delta maps in peek hour 17:00pm, 18:00pm and 19:00pm, respectively. Delta values are normalized (-100, 0) for data privacy, where 100 and 0 means the maximum and minimum delta value, respectively. As the evening peak process was going on, supply was matched with demand along the direction of red arrows in panels (a), (b) and (c) gradually.
C INTERPRETABILITY CASES
this section, we analyze the impacts of essential components in Causal Trans and focus on what causal atlention learns. First, since causal demand and supply are hardly assembled with unbiased estimation (Figure 2), we demonstrate attention-based interpretability in instance-specific significant events like frequent rainfall, holidays and peek time slots. Second, we perform ablation analysis about target probabilistic distribution causal attention with DML and Uplift, F FastAttens ion and Spati alFus on. Finally, compare fast improvements in multi-head attention CPU (Intel Xeon E5-2630 2.20GHz) and GPU (Tesla P40), respectively.
C.I CAUSAL ATTENTION VISUALIZATION
As one of the most essential components, causal attention employs difference stationary tests and double machine learning estimate coefficients ((s) of treatment effects. In this section, we visualize causal attention distribution through sample-specific cases, including rainfall, weekdays, and time slots. Frequent rainfall the most significant weather event for demand described in Section A. Unlike with plenty of rainfall events, there are only dozen of holidays in one year. If sequential context before one holiday fails to pass Kpss stationary test, causal estimator would not be applied in training attention weights. Large-scale dataset is the fundamental to our method. For the diverse peek time slots, Section 3.1 concludes that demand and supply distributes different at commuting peeks and night hours. addition, seasonal fluctuation and government's 's policies (e.g. traffic restriction in National Day) are considerable factors.
Rainfall, Take demand forecasting at an anonymous region city A as an example, treatment rainfall 8, target is demand and other covariates include regional id, time slots and holidays. For convenience, we select a group of adjacent AB groups from sufficient rainfall cases to give an interpretation. In Figure 4, we backtrack rainfall treatments to fix AB Group 2, and search AB Group / by controlling similar covariates. Similarity means that both first-order differences are stationary, and then we construct a group of simple randomized controlled experiments. Given estimated 0 (z) by running DML, plot the distribution of causal
14

---

Under review conference paper at ICLR 2021
attention on the right side of green line. practice, large amounts of increasing data would enhance the robustness of causal evaluation iteratively.
presanilitic
40-
Group
4B
Group
Pay Dun wy
My
lala
AMM
les
M
20
32
2000-28-25
Figure 4: Causal attention and probabilistic demand forecasting treated by rainfall some days at a anonymous region in city A. A time slice means hour. The black solid line is real demand time series, and orange solid line is temporal differences in next 24 hour slices. The green vertical line means a starting point of forecasting, and subsequent green filled areas describe confidence interval between quantile 10% and 90%. According to causal attention, "AA group 1" and "AA group 2" (two red filled areas) are regarded comparable contexts, as the first-order difference of both groups passes kpss Wwiitkkwwkii al., 1992) stationary test (P > 0.05). "AB group 1" and "AB group 2" (two purple filled areas) is control group and treatment group, respectively. Based on homologous hypothetical controlled experiments, causal attention with future weather information would be added in forecasting to learn inferences.
Collaborative demand and supply. As described in Section 3.1 and equation (2), the distribution of supply is driven by the spatio-tempone patterns of demand. Similar with above Rainfall analysis, we take another anonymous region in city A as an example. In this case, forecasting target is supply, causal treatment is demand, and external variables include weather, time slots and holidays. According to Algorithm I, our method needs to construct AB groups and corresponding lookback AA groups from large-scale historical data. For both AB controlled experiments, the average demands of AB and AA groups should be significant different, while supply is unlimited. In AA experiments, for at least one day. We trace back data to the past, but selected AA groups should satisfy randomization grouping hypothesis passed by 1-fest. Such periods with stable supply are abundant in recent years, which implies that we can easily find proper evaluation dataset for diverse regions. In Figure 5, trained causal attention demonstrates the demand's causal weights reflect supply forecasting. Additionally, more novel causal modules similar with equation (2) can be designed to enhance interpretability and robustness, and such modules support end-to-end training in Causal Trans well.
empirically suggest that the time span maintains
C.2 ABLATION ANALYSIS
This subsection focuses on the performance of CausalTrans when some components are excluded. Proposed essential items contain tricky Pois PoissonOOtupt, Causal Attention(C .A. A.), FastAt tesstttentti and Spat ialllllll Fusion. C.A. can be implemented by different causal algorithms, such as DML and Up 1 ifft (Künzel et al., 2019). As shown in Table 5, we list Rıo (50% quantile point) losses on previous eight Ride- hailing datasets. Table demonstrates that C A. (DML) outperforms all of other components, and causal supply can be clearly influenced by causal demand. Finally, both FastAttent i on and Spat i Fuss i on are not harmful to forecasting performance.
Furthermore, spatial fusion shows tiny improvement (+0.3% on average) in Table 5. We feel that spatial fusion aggregates adjacent hexagonal grids, leading to reducing statistical noises in both demand and supply. For
15

---

Under review conference paper at ICLR 2021
Bupply eUppyy (IPPA)
AF
M
w
1.0
10
2005 24.00
2026 34.00
2012 15
20.5 04 37
2015 04- 22
2015 04
2015 04 27
Figure 5: Causal attention and probabilistic supply forecasting treated by demand in some days at an amonymous region in city A. The black solid line is real supply time series, while other lines and areas are similar with Figure 4. AA group and AA group 2 (two red filled areas) are selected by causal attention, and both AA groups have analogous rainfall, the same weekday and time slots. Mean difference of both AA groups in t-test is not significant. As rainfall can change demand and following demand changes supply, we estimate collaborative demand causal effects by adding rainfall causal effects. Fortunately, there are no rainfall and holidays factors in evaluation periods (right-hand side of the green vertical line), and therefore we can visualize the pure causal attention distributed in (0, 1).
Table 5: Ablation analysis (R5o losses) of various components Causal C Trans. Fach value represents the R50 loss that eliminates specific component, and percentages in brackets are loss variations. For Traffic data set, Poi ssonOutput, and Spatial Fussi on are positive components. For Ride-hailing data set, each component reflects different importance varying difffernnt tasks. For Poi roissonOutput. forecasting demand and supply are not significantly different, but longer predict step leads to a bigger loss increment. Lack of Po Poissone sutOutput increases 2% loss average. The most essential component C. A. (DML) improves collaborative supply forecasting more than demand. Long-term prediction depends on C.A. (DML) as well the causal covariates. C.A. (Uplift) is similar to C. (DML), , but the simpler C. A. (Uplift) ft) means less importance. FastAttents i on (-0.2% on average) and Spat a l Fussion (+0.3% on average) are proposed for reducing time complexity significantly without harming loss.
PoissonOutput
C.A. (DML)
C.A. ((pli(t)
Fast&ltention
Spatial7usios
Traffic
0.098(+3.19)
0.095(-0.4%)
0.097(+2.55)
Ride hailing (1d, ciry A, Demand)
0.444(+2.3%)
0.463()..6%)
0.457(+5.2%)
0..43( (.225))
0.431(-0.8%)
Ride-hading (Id, ciry Supply)
0.400(+1.7%)
0.424(+7.9%)
0.417(+6. 1%)
0.392(-0.39)
0.394(+0.295)
Ride-hoiling (Id, city B, Demand)
0.366(+1.5%)
0.379(45. 19%)
0.361(-0.0%%
0,363(+0.5%)
Ride-hailing (1d, city B, Supply)
0,345(+1.1%)
0.363(+6.4%)
0.360(+5.5%%
0,341(-0.1%)
0.344((0.8%)
Ride hailing (7d, city Demand)
0.633(+3.2%)
0.674(+10.0%)
0.660(+7.7%)
(.641()
0.612( (0.)2)))
hailing (7á, city A. Supply)
0.483(+ 3.39%
0.537(+ (4.8)
0.520(+11.155
0.465
0.472(+0.9%)
Ride hailing (7d, city Demund)
0.549(+1.8%)
0,584++8.4%)
0,575(+6.7%)
0.540(-0.2%)
0,546(+1.3%)
Ride hailing (7d, city B, Supply)
0.460(+1.3%)
0.503)110.55)
(.494(+8.8%)
0.453(
0,457(+0.7%)
Average
+2.0%
+8.8%
+7.0%
-0.2%
+0.3%
instance, in some cases, the boundary (usually around 800 meters) of adjacent grids separates large demand hotpots (e.g.. large shopping malls), resulting in some noise when counting supply and demand. Spatial fusion reduce the influence of such noise, while improving the probabilistic forecasting performance. According to Table 5, the longer forecasting time (c.g.. days versus day), the more significant gain by using spatial trick for enhancing the robustness of forecasting. The
fusion. We consider the use of spatial fusion hyperparameter of spatial fusion is K. used in the kmeans method. In this paper, we set K. E (3, 4,,4,, 5). More ablation analysis about K. is shown in Table 6.
16

---

Under review conference paper at ICLR 2021
Table Ablation analysis (Rso losses) various cluster number in SpatialFus represents the R50 loss specific K, and percentages in brackets are loss variations. Roughly speaking, seven-day prediction needs bigger K than one-day. We conclude that the longer prediction range needs more heterogeneous patterns. Optimal K on various Ride-hailing subdatasets shown in Table 4.
Each value
==5
Ride-hailing (1d, city A, Demand) Ride hailing (7d, city A, Demand)
0.434(+0.00) (.4) 0.630(2..7g
0.446(228%)
0.448(+333) 0.613(+0.05)
0.449(+3448)
0.641(+4..%)
0.617(00.9%
0.629(-2.6%)
C.3 TIME EFFICIENCY IMPROVEMENT
One of innovations proposed in this paper is to shorten running time of attention without losing overall quantile loss. The long experiment cycle suggests that we should choose representative dataset, such as one-day demand prediction in city A. Data size of city A is large enough to reflect robust attention weights. In such dataset, we are only interested in the decrease of running time as the number heads in multi-head attention decreases. As shown in Figure 6, when multi-head
the reduction ratios of CPU (20), GPU (1) and GPU (2) compared with softmax are 58%, 70%, and 68%, respectively. Similarly, when multi-head is equal to 5, the responding reduction ratios are, respectively, 49%, 58% and 60%. An exact time complexity V( (see in Section 3,4), the smaller K, the longer running time. In summary, proposed time-efficien attention outperforms default softmax attention significantly.
60
suftmac
suftoe Fime officet
time aff
342
60
Burenp Smee Po2r
Bundin Tree He00s
630
42-
1
20-
12-
SHUILI
010200
CRAOI
(a) Multi-head = 3.
(b) Multi-head S.
Figure 6: Time efficiency improvements on one-day demand prediction in city A. The numbers in brackets mean logic cores a chip. CPU (20) utilizes multiple processing to accelerate matrix multiplication, The comparison of GPU (1) and GPU (2)) aims to demonstrate possibility of applying powerful GPUs in real world. Setting up (a) 3 heads and (b) heads in head attention is quite different. According to equation (9) and above bar plots, the less number of heads, the shorter running hours. Each running hour result is averaged by using three independent experiments.
D DISCUSSIONS ON LINEAR ATTENTION
In subsection 3.4, propose novel linear attention based approximate Taylor expansion of exponential function. In contrast, other important methods are also developed to reduce attention cost. These attention acceleration methods can be roughly clarified into two groups. The first one is to construct kernel functions to
17

---

Under review conference paper at ICLR 2021
approximate softmax function, denoted as
softmax (Q" K) = =0Q)) <(K),
(14)
where Q and K are query matrices and key matrices, respectively. For instance, Katharopoulos al. (2020)
construct a kernel function with basis function s0(x) = x(z) elu(x) + 1 and reduce the computation complexity from O(N²) to ONN, but such performance just only concluded from image dataset. Shen et al. (2018) further explore a series of kernel forms to dissect Transformer's 's attention. They proposed a new variant of Transformer's s attention by modeling the input as product of symmetric kernels. This approach
replaces the calculation order of softmax, which is equivalent to the basis function ((x) = softmax(()) and (p(x) =
The second one is to modify attention's 's definition. Child et al. (2019) develope sparse factorizations of the attention matrix, which reduce the computation to O(NvN), but its attention hyperparameters are very hard be initialized and actual efficiency is hard to ensure. Kitaev et al. (2020) propose Reformer to replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O(N2) to O (N))) where is the length of the sequence. Furthermore, they use reversible residual layers instead of standard residuals, allowing storing activations only once in the training process instead of L times, where L is the number of layers. However, Reformer is difficult to be implemented and applied in different tasks. Wang et al. (2020) demonstrate that the self-attention mechanism can be approximated by a low-rank matrix, and further propose Linformer mechanism to reduce the overall self-attention complexity to ONN). Linformer uses two additional matrices E and V project K and V, respectively, in order get Attention (Q, K, V)= so soft ftma ((EK))))))))) FV. But the MLM experiment in Linformer does not need to extract long-te dependence and cannot verify its linear time complexity for capturing long- meg-tterm attention. Eliminating redundancy vectors from the self-attention is key design idea. Furthermore, Goyal et al. (2020) exploit redundancy pertaining to word-vectors, and propose PoWER- BERT to achieve up to 4.5x reduction in inference time over BERT with <1% loss in accuracy the standard GLUE benchmark. Similarly, Dai et al. (2020) propose Trans--rran,,rr,rrr which gradually compresses the sequence hidden states to a shorter one, and hence reduces the computation cost. Finally, for our approximate Taylor expansion of softmax attention, feature maps conditions and our task focuses on short-tere dependence, then our linear attention would be useful for this aspect.
(i.e. Q. K and V in self-attention) meet the positive definite and normalization
18


Reviews:
[{"review_id": "sxbDM_LiZ2q", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes a new framework of casual spatial-temporal prediction with high interpretability. Specifically, it first proposes a casual transformer including fast spatial graph fusion, casual attention, and temporal attention units. The authors conduct extensive experiments from different domains (electricity, traffic, etc.) and show the effectiveness, efficiency, and interpretability of their proposed method. \n\nStrength:\n\n+ The authors propose to reduce the complexity of the computation of the attention module. \n\n+ The authors first propose a causal attention method for HTE in large-scale spatio-temporal prediction problems.\n\n+ The experiments on datasets from various domains are adequate, which can support the authors' claim. \n\nWeakness: \n\n- Both equations (1) and (2) are based on the authors' assumptions. Do such assumptions have any support (either from previous literature or from the data)? In my opinion, some of the assumptions do not make sense, for example, one assumption is that $x_v(t + 1)$ is primarily affected by historical demands in $x_v(: t)$ and external covariates in $z$ without historical supply $y$. However, if the historical supply y is not enough, then the demand may raise because more and more demand accumulates. \n\n- It seems that the authors' proposed approximate Taylor's expansion attention can be used for attention models or even softmax functions in any scenarios. Does it have any limitations? For example, some of the coefficients are very large so that the higher-order Taylor terms cannot be overlooked. Please explain why such problems do not exist in the spatial-temporal prediction scenario in this paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "nCqPm9Qsi7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is interested with multivariate probabilistic forecasting applied in the context of ride-hailing forecast.\nThe goal is to be able to handle the spatial aspect, causal effects of external covariates (e.g. the causal impact of rain, Christmas on supply/demand) and dependency between supply and demand. To this end, the authors propose a causal attention mechanism\t(to not just detect correlation with covariates) and a custom transformer architecture where the quadratic attention cost is avoided. Experiments are performed on public and private datasets against a set of (mostly) univariate baselines.\n\nStrong points:\n+ highly-relevant problem\n+ novel consideration of handling causality instead of simple covariate correlation in this context\n\nWeak points:\n- very hard to read, many details missing and notation are not introduced\n- lack of relevant baselines (e.g. baselines using spatial information)\n- lack of relevant metrics to illustrate the benefit of the contribution\n- missing related work discussion for efficient attention computation\n\nI recommend a reject for this paper.\n\nWhile the problem presented by the paper is highly relevant for the community, the paper has several issues that makes it not ready for publication. The first issue comes in the clarity of the description and in particular section 3.3 where many terms and notation are not introduced at all making the paper very hard to read (see detailed comments). \n\nExperiments are also problematic as many details were unclear (see detailed comments) and would be far from being reproducible. But the most problematic bit is in its design: key aspects introduced in the paper are not asserted in the experiments.1) While the method claims ~5% error improvement on the ride-hailing benchmark, only one baseline have access to spatial information while numerous methods have been proposed to handle spatio-temporal forecast, only one baseline (TFT) is provided with spatial information (with no detail). 2) Since you are interested in the probabilistic forecast of the joint demand/supply targets (collaborative), your experimental setup should consider a joint metric (CRPS-sum for instance [1,2]) rather than the average of R50/R90 metrics of demand/supply targets which is \"blind\" to correlation between the two demand and supply targets.\n\nFinally, I had issue with the Taylor expansion as its description was not clear (see detailed comments) but also I found description of related work missing in this aspect. Attention cost has been decreased to $N\\sqrt{N}$, $N\\log{N}$ and $N$ respectively in [3, 4, 5], this discussion is clearly missing to relate your contribution.\n\n# Additional questions to the author\n- 3.1 at no point in the paper you say explicitly what is the loss that you are minimizing, this makes it harder to read. I would recommend to specify in 3.1 directly that you minimize NLE with poisson distribution (which is what I understood) and specify also clearly how predictions are made.\n- 3.3 the beginning of the section mix model description and related work\n- 3.3 Eq. (3) has two unintroduced notation, what is the \\bar? what is \"batch\"?\n- 3.3 Eq. (6) is so confusing, you are mixing norms and vectors (unintroduced), $T$ is not explicitly introduced, there is no clear explanation why the approximation holds also (e.g. why a^T W would be small)\n- 4.2 DeepState is *not* an auto-regressive model, it belongs to the second category.\n- 4.3: \"CausalTrans outperforms all other competing methods primarily due to the use of the causal estimator DML and spatial information\". This is very problematic as you dont compare with methods designed to handle spatial information. \n\n# Additional feedback (not part of the decision assessment)\n- 3.1 eq (1) (2) makes an assumption that only demand impacts supply: it would be good to discuss it at least. One could imagine where it breaks.\n- 3.1 weather features: you should precise whether they are known in advance\n- 3.3 The section will be easier to read if you indicate variable dimensions\n\nFinally, I would recommend to use a public dataset rather than the private one for your benchmark (e.g. NY taxi dataset or Uber), you also have covariates such as date or weather forecast and this would make your work comparable and reproducible in the future.\n\n\n\n[1] High-dimensional multivariate forecasting with low-rank Gaussian Copula Processes http://papers.nips.cc/paper/8907-high-dimensional-multivariate-forecasting-with-low-rank-gaussian-copula-processes\n\n[2] Multi-variate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows https://arxiv.org/abs/2002.06103\n\n[3] Generating Long Sequences with Sparse Transformers https://arxiv.org/pdf/1904.10509.pdf\n\n[4] Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention https://arxiv.org/abs/2006.16236\n\n[5] Reformer: The Efficient Transformer https://arxiv.org/abs/2001.04451\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VlTvPc0dGnc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a Transformer-like framework, named CausalTrans, to tackle the demand and supply problem in the ride-hailing market. The problem is formulated by training two probabilistic models which forecasts collaborative demand and supply, by given historical observations and dynamic covariates. The paper leveragesTransformer encoder-decoder architecture, and proposes submodule (Fast S.F., C.A. and T.A.) for different functionalities. Many experiments and ablation analysis are constructed to show that the proposed method outperforms state-of-the-art. The paper also provides good visualization regarding the casual attention model, facilitating to understand the proposed idea.\n\n+ overall the paper is well organized and easy to read.\n+ the proposed architecture is with merit: Fast S.F. with an approximate Taylor’s expansioninstead of using softmax function, showing lower computational cost with little performance dropped; C.A. is proposed to deal with HTE problem in large-scale spatio-temporal forecasting problems, where a DML algorithm is also proposed to learn the C.A. model.\n+ the experimental results are comprehensive and promising.\n\nStill, I have some questions:\n- The ablation analysis shows that spatial fusion shows tiny improvement (+0.3% on average). I’d like the author to elaborate more on the reason. If there are hyperparameters for the spatial fusion, please do some ablation analysis in this regard.\n\n- From experiments the fast attention improves computation a lot with only 0.2% performance dropped in multi-horizon methods. Is it the same in multi-horizon methods? \n\n- From table 1 the results show that, in case of Electricity, the proposed method can’t outperform the state-of-the-art (TFT) due to lack of covariates and spatial information. I’d like to see the ablation analysis (like table 3) in case of Traffic, showing that in case of the iterative method, the performance can be gained by each submodule.\n\n- Lack of experimental details. For example, learning rate/strategy, batch size, optimizer, architecture settings...etc. If the author(s) plan not to release the code in the future, it’s better to list the experimental details in the appendix, for reproducing the performance. \n\nOverall I think that the proposed causal attention is valuable. By adapting the C.A. to the transformer architecture, the proposed method is comparable to state-of-the-art. Moreover, sufficient experiments demonstrate the proposed method can achieve decnet performance in the demand-supply problem. If the paper can better clarify the points mentioned above, I’ll vote for positive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "E6gZWHZNzad", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an interpretable spatio-temporal fusion transformer for predicting supply and demand in ride-haling platforms.  More generally, the authors claim that their approach extends to other two-sided markets such as electric grids, retail etc by showing empirical results of their approach using data from these markets.  The paper is well-written and easy to follow. \n\nThe assumption that the supply is always conditioned on the demand x_v(, t + \\tau_max) is too strong.  Is there a smoother of this assumption where the demand is dependent on a moving window over the past and future supply?  The experimental results show the efficacy of the proposed ML architecture.  However, one result that will greatly help the conclusions are results that clearly show the interpretability of the predictions, given that the authors state this as one of the main differences of their proposed solution.\n\nThe use of higher-order Taylor terms to approximate the attention procedure is interesting but the time complexity reductions are obvious and therefore does not meet the novelty bar for an ICLR submission.\n\nOther suggestions - make the use of the word 'collaborative' in the model more clear.  The significance of the results is not completely clear and how the interpretability is helping understanding of the results.  Adding clarifications will help in the exposition.\n\nI still have some concerns regarding the paper. The lack of baseline comparisons with spatio-temporal data (as also observed by fellow reviewers). My other concern also remains - from the authors' response, it is not clear how once can clearly attribute explainability of the results from their analysis of the model.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a spatial-temporal prediction framework with causal effects of predictors for better interpretability. The idea is interesting and the touch on modeling causal relations could be useful in practical applications. The paper receives mixed ratings and therefore there has been extensive discussion. We agree that while the paper has some merits, it falls short on the following aspects: 

1, One central issue pointed out by all reviewers is the evaluation. For example, the contribution on efficient attention was not compared to any previous work; most of the baselines do no have access to the spatial information, which makes the comparison unfair. The authors did add two more baselines with access to spatial information. However, there are not enough details and discussions to make the results convincing; In addition, other stronger baselines should be added. 

2. The notation and technical presentation was extremely lacking in the submitted version, the amount of unintroduced notations. Even in their core contribution equations had major issues with norm and vectors mixed together (see the difference between the corrected equation in the Taylor equation and the one in the original submission)

After the discussion, all reviewers agree that the paper fails to provide a fair and convincing evaluation, and the ratings will be adjusted to reflect the discussion. We hope that the reviews can help the authors improve the draft for a stronger submission in the future. 
Decision: Reject