Is Attention Better Than Matrix Decomposition?
Authors: Zhengyang Geng
Meng-Hao Guo
Hongxu Chen
Xia Li
Ke Wei
Zhouchen Lin
Venue: ICLR 2021 Poster (2021)
Keywords: attention models
matrix decomposition
computer vision

Abstract: As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays a vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intriguing finding is that self-attention is not better than the matrix decomposition~(MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context issue as a low-rank completion problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series of Hamburgers, in which we employ the optimization algorithms for solving MDs to factorize the input representations into sub-matrices and reconstruct a low-rank embedding. Hamburgers with different MDs can perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants. Code is available at https://github.com/Gsunshine/Enjoy-Hamburger.

Published as a conference paper at ICLR 2021
IS ATTENTION BETTER THAN MATRIX DECOMPOSITION?
Zhengyang Geng :2, Meng-Hao Guo Hongxu Chen', Xia Li", Ke Wei", Zhouchen 2 !Zhejiang Lab; KKyy Lab. of Machine Perception (MoE), School of EECS, Peking University; Tsinghua University; "School of Data Science, Fudan University; "Pazhou Lab
ABSTRACT
As an essential ingredient of modern deep learning, attention mechanism, especially self-attention, plays vital role in the global correlation discovery. However, is hand-crafted attention irreplaceable when modeling the global context? Our intrigu- ing finding is that self-attention not better than the matrix decomposition (MD) model developed 20 years ago regarding the performance and computational cost for encoding the long-distance dependencies. We model the global context is- sue low-rank completion problem and show that its optimization algorithms can help design global information blocks. This paper then proposes a series Hamburgers, in which we employ the optimization algorithms for solving MDs factorize the input representations into sub-matrices and reconstruct low-rank embedding. Hamburgers with different MDs perform favorably against the popular global context module self-attention when carefully coping with gradients back-propagated through MDs. Comprehensive experiments are conducted in the vision tasks where it is crucial to learn the global context, including semantic segmentation and image generation, demonstrating significant improvements over self-attention and its variants. Code is available.
INTRODUCTION
Since self-attention and transformer (Vaswani al., 2017) showed significant advantages recurrent neural networks and convolutional neural networks in capturing long-distance dependencies, attention has been widely adopted by computer vision (Wang al., 2018: Zhang et al., 2019a) and natural language processing (Devlin et al., 2019) for global information mining. However, is hand-crafted attention irreplaceable when modeling the global context?
This paper focuses on new approach to design global context modules. The key idea is, if we formulate the inductive bias like the global context into an objective function, the optimization algorithm to minimize the objective function can construct computational graph, i.e., the architecture we need in the networks. We partieularize this idea by developing counterpart for the most representative global context module, self-attention. Considering extracting global information in the networks as finding a dictionary and the corresponding codes to capture the inherent correlation, we model the context discovery as low-rank completion of the input tensor and solve it via matrix decomposition. This paper then proposes global correlation block, Hamburger, by employing matrix decomposition to factorize the learned representation into sub-mat aub-matrices low-rank signal subspace. The iterative optimization algorithm to solve matrix decomposition defines the central computational graph, i.e., Hamburger's 's architecture.
to recover the clean
Our work takes advantage of the matrix decomposition models as the foundation of Hamburger, including Vector Quantization (VQ) (Gray & Neuhoff, 1998), Concept Decomposition (CD) (Dhillon & Modha, 2001), and Non-negative Matrix Factorization (NMF) (Lee & Seung, 1999). Additionally, instead of directly applying Back-Prophe Poce-Prrpggtioon Through Time (BPTT) algorithm (Werbos al., 1990) to differentiate the iterative optimization, we adopt a truncated BPTT algorithm, i.e., one-step gradient, to back- propagate the gradient effectively. We illustrate the ad advantages of Hamburger in
"qqual first authorship Corresponning author

---

Published as a conference paper at ICLR 2021
the fundamental vision tasks where global information has been proven erucial, including semantic segmentation and image generation. The experiments prove that ceammiizaiio-desiinnn Hamburger perform competitively with state-of-the-ar attention models when avoiding the unstable gradient back-propagated through the iterative computational graph of MD. Hamburger sets new state-of- the-art records the PASCAL VOC dataset (Everingham al., 2010) and PASCAL Context dataset (Mottaghi et al., 2014) for semantic segmentation and surpasses existing attention modules for GANs in the large scale image generation on ImageNet (Deng et al., 2009).
The contributions of this paper are listed as follows:
We show white-box approach to design global information blocks, i.e., by turning the optimization algorithm that minimizes an objective function, in which modeling the global correlation is formulated as low-rank completion problem, into the architecture.
We propose Hamburger, a light yet powerful global context module with O(n) complexity, surpassing various attention modules on semantic segmentation and image generation.
We figure out that the main obstacle of applying MD in the networks is the unstable backward gradient through its iterative optimization algorithm. As a pragmatic solution, the proposed one-step gradient facilitates the training of Hamburger with MDs.
2 METHODOLLOGY
2.1
WARM UP
Since matrix decomposition is pivotal to the proposed Hamburger, we first review the idea of matrix decomposition. A common view that matrix decomposition factorizes the observed matrix into a product of several aut-matrices, e.g.. Singular Value Decomposition. However, more illuminating perspective is that, by assuming the generation process, matrix decomposition acts as the inverse of the generation, disassembling the atoms that make up the complex data. From the reconstruction of the original matrices, matrix decomposition recovers the latent structure of observed data.
Suppose that the given data arranged as the columns of large matrix X [X1,'.. ... ,x,] Rd A general assumption is that there is a low-dimensional subspace, or union of multiple subspaces hidden in X. That is, there exists dictionary matrix D [d],... ,d,] E dxdr and corresponding
codes C = [c1, ... , cal E Rexn that X can be expressed as generation.
X X DC + E,
decomposition
where X Rdxn is the output low-rank reconstruction, and E E Rdxx is the noise matrix to be discarded. Here we assume that the recovered matrix X has the low-rank property, such that rank(X) < min(rank( (D), rank(C)) S < min(d, nd)
Different MDs can be derived by assuming structures to matrices D, C, and E (Kolda & Bader, 2009; Udell al., 2016). MD is usually formulated as an objective with various constraints and then solved by optimization algorithms, with classic applications to image denoising (Wright et al., 2009; Lu et al., 201 4), inpainting (Mairal al., 2010), and feature extraction (Zhang et al., 2012).
2.2 PROPOSED METHOD
We focus on building global context modules for the networks without painstaking hand-crafted design. Before starting our discussion, we review the representative hand-designed context block self-attention pithily.
The attention mechanism aims at finding a group of concepts for further conscious reasoning from massive unconscious context (Xu et al., 2015; Bengio, 2017; Goyal et al., 2019). As representative, self-attention (Vaswani et al., 2017) is proposed for learning long-range dependencies in machine translation,
QK Vd
Attention (Q, K, V) = softmax
V,
(3)
2

---

Published as a conference paper at ICLR 2021
where Q, V E Rnxd features projected by linear transformations from the input. Self- attention extracts global information via attending all tokens at a time rather than the typical one-by- one processing of recurrent neural networks.
(. Upper Breed (Lnear Translormaiion)
d'
(2)
Hame Marrix Decompostiinn
MXN
Z
Lower Broad (Linerr Tranifomation
Matris Decomposiiion
Figure I:
Overview of Hamburger
Though self-attention and its variants achieved great success, researchers are confronted with (I) developing new global context modules based on self-attention, typically via hand-crafted engineering, and (2) explaining why current attention models work. This paper bypasses both issues and finds method to easily design global context modules via well-defined white-box toolkit. We try to formulate the human inductive bias, like the global context, as an objective function and use the optimization algorithm solve such a problem to design the module's architecture. The optimization algorithm creates a computational graph, takes some input, and finally outputs the solution. We apply computational graph of optimization algorithms for the central part of our context module.
Based on this approach, we need to model the networks' global information issue as an optimization problem. Take the convolutional neural networks (CNN) as an example for further discussion. The networks output tensor X' E C= XW after we feed into an image. Since the tensor can be seen as a set HW C-dimensional hyper-pixels, we unfold the tensor into a matrix X C= When the module learns the long-range dependencies the global context, the hidden assumption is that the hyper-pixels are inherently correlated. For the sake of simplicity, we assume that hyper-pixels linearly dependent, which means that each hyper-pixc in X be expressed the linear combination of bases whose elements are typically much less than HW. In the ideal situation, the global information hidden in X can be low-rank. However, due vanilla CNN's poor ability to model the global context (Wang et al., 2018; Zhang et al., 2019a), the learned X is usually corrupted with redundant information incompleteness. The above analysis suggests potential method to model the global context, i.e., by completing the low-rank part X in the unfolded matrix X and discarding the noise part E, using the classic matrix decomposition models described in Eq. (1), which filters out the redundancy and incompleteness at the same time. We thus model learning the global context as a low-rank completion problem with matrix decomposition as its solution. Using the notion of Sec. 2. the general objective function of matrix decomposition is
min L(X, DC) + (D) + RC(C) D,C
(4)
where is is the reconstruction loss, R₁ and R2 are regularization terms for the dictionary D and the codes C. Denote the optimization algorithm to minimize Eq. (4) M. M is the core architecture we deploy in our global context module. To help readers further understand this modeling, We also provide more intuitive illustration in Appendix G.
In the later sections, we introduce our global context block, Hamburger, and then discuss detailed MD models and optimization algorithms for M. Finally, we handle the gradient issue for back-propagation through matrix decomposition.
2.2.1 HAMBURGER
Hamburger consists of one slice of "ham" (matrix decomposition) and slices of "bread" (linear transformation). As the name implies, Hamburger first maps the input Z E Rddxxn into feature space with linear transformation W, E Rdxé, namely "lower bread", then uses matrix decomposition M to solve low-rank signal subspace, corresponding to the "ham", and finally transforms extracted signals into the output with another linear transformation W. E d.d. xd called "upper bread"'
H(Z)= W..M(WiZ))
(5)
3

---

Published as a conference paper at ICLR 2021
where M is matrix decomposition to recover the clear latent structure, functioning global nonlinearity. Detailed architectures of M. i.e, optimization algorithms to factorize X, are discussed in Sec. 2.2.2. Fig. describes the architecture of Hamburger, where it collaborates with the networks via Batch Normalization (BN) (loffe & Szegedy, 2015), a skip connection, and finally outputs Y,
Y Z (N)H(Z)).
2.2.2 HAMS
This section describes the structure of "ham", i.c., M in Eq. (5). As discussed in the previous section, by formulating the global information discovery as an optimization problem of MD, algorithms to solve MD naturally compose M. .M takes the output of "lower bread" its input and computes low-rank reconstruction as its output, denoted as X and X, respectively.
M(X) X DC.
We investigate two MD models for M, Vector Quantization (VQ), and Non-negative Matrix Factor- ization (NMF) to solve D and C and reconstruct X, while leaving Concept Decomposition (CD) to Appendix B. The selected MD models are introduced briefly because we endeavor to illustrate the importance of the low-rank inductive bias and the optimization-driven designing method for global context modules rather than any specific MD models. It is preferred to abstract the MD part whole, i.e., M in the context of this paper, and focus on how Hamburger can show the superiority in its entirety.
Vector Quantization Vector Quantization (VQ) (Gray Neuhoff, 1998), a classic data compres- sion algorithm, can be formulated as optimization problem in term of matrix decomposition:
min D.C
X DC s.t. C2 E {ci, C2 ... e,,,,
where e; is the canonical basis vector, c; = [0, 1, _OTT. The solution to minimize the
ith
objective in Eq. (8) is K-means (Gray Neuhoff, 1998). However, ensure that VQ differentiable. we replace the hard arg min and Euclidean distance with softmar and cosine similarity, leading to d, x,
Alg. 1, where cosine D, X) and sof somt is applied column-wise and T is the temperature. Further we can obtain hard assignment by one-hot vector when T ->
) is a similarity matrix whose entries satisfy cosine cosine(D,X)) (D,)); = I ||x]
Algorithm 1 Ham: Soft VQ Input X. Initialize D, C. for & from 1 to K do
Algorithm Ham: NMF with MU
Input X. Initialize non-negative D, C
for from 1 to K_do
Cj - Cúj (D X)
C softmar(( (cosine( (D, X)) D t T ddog(C1,))
(DTDDW) DCWs
(XC ho
Dij +- Dij (DCCT). )5
end for Output X = DC.
end for Output X DC.
Non-negative Matrix Factorization If we impose non-negative constraints on the dictionary D and the codes C, it leads Non-negative Matrix Factorization (NMF) (Lee & Seung, 1999):
min
|X DC s.1. Dij 2 0, Cj.- 2 0.
D.C
To satisfy the non-ne non-negative constraints, we add ReLU non-linearity before putting X into NMF. We apply the Multiplicative Update (MU) rules (Lee & Seung, 2001) in Alg. 2 to solve NMF, which guarantees the convergence.
As white-box global context modules, VQ. CD, and NMF are straightforward and light, showing remarkable efficiency. They are formulated into optimization algorithms that mainly consist of matrix multiplications with the complexity O(ndr), much cheaper than complexity (n²2d) in self-attention as r & n. All three MDs are memory-friendly since they avoid generating a large n X n matrix as an intermediate variable, like the product of Q and K of self- self-attention in Eq. (3). In the later section, our experiments prove MDs are at least on par with self-attention, though the architectures of M are created by optimization and look different from classic dot product self-attention.
4

---

Published as a conference paper at ICLR 2021
2.3 OTE-STEP GRADIENT
Since M involves an optimization algorithm as its computational graph, a crux to fuse it into the networks is how the iterative algorithm back-propagates gradient. The RNN-like behavior of optimization suggests Back-Propage Propagation Through Time (BPTT) algorithm (Werbos et al., 1990) as the standard choice differentiate the iterative process. We first review the BPTT algorithm below. However, in practice, the unstable gradient from BPTT does harm Hamburger's 's performances. Hence we build an abstract model to analyze the drawbacks of BPTT and try to find pragmatic solution while considering MD's s nature as an optimization algorithm.
As shown in Fig. 2, x, y and h' denote input, output and intermediate result at time step t, respectively, while F and G are operators. At each time step, the model receives the same input processed by the underlying networks.
h h+ F(h', x).
(10)
The intermediate results h' are all discarded. Only the output of the last step is passed through & for output y,
y = G(h)..
(11)
In the BPPT algorithm, the gradient from output y input x is given, according to the Chain rule:
h(--
h--1
h
y
F
diu'
g
ihx
dy 0x
Oy Th
Oh
0h
II
(12)
x
X
Oh
0x
thought experiment is to consider / - oc, leading to a fully converged result h" and infinite terms in Eq. (12). We suppose that both F and G are Lipschitz with constants La w.r.f h, Ly w.r.t. x, and Lg. and Ln < 1. Note that these assumptions apply to large number of optimization numerical methods. Then we have:
Figure 2: One-step Gradient
Table 1: One-step Gradient & BPTT
Proposition 1
(h') has linear convergence.
Method
One-step
BPTT
Proposition 2
lim By 1000 dx
By Dh
(I
0F 0h
-
aJ dx
VQ
77.7(77.4)
76.6(76.3)
CD
78.1(77.5)
75.0(74.6)
Proposition 3
lim
By
0, lim
By
Lg
NMF
78.3(77.8)
77.4(77.0)
(0000
It is easy to incur gradient vanishing w.r.t. h when Ln is close to O and gradient explosion w.r.t. 8x moreover, suffers from an ill-conditioned term (I- OJF 3h > when the largest eigenvalue of aF 8h i.e., the Lipschitz constant of F w.r.t. h, approaches I and its minimal eigenvalue typically stays near 0, thus restricts the capability of the gradient to search the well-generalized solution in the parameter space. The erratic scale and spectrum of the gradient back through the optimization algorithm indicate the infeasibility to apply BPTT to Hamburger directly, corroborated by the experiments in Tab. 1, using the same ablation settings as Sec. 3.1.
x when L,, is close to 1. The Jacobian matrix dy
The analysis inspires us a possible solution. Note that there are a multiplicatiinn of multiple Jacobian Sh matrices Bh/ and summation of an infinite series in BPTT algorithm, leading to uncontrollable scales of gradients. It enlightens us to drop some minor terms in the gradient while preserving its dominant terms to ensure the direction is approximately right. Considering terms of Eq. (12) as
a series, i..., { āy Lah [T
ab:11 11)
0h" ab'',, h. it makes sense to use the first term of this series to ax
8h.
approximate the gradient if the scale of its terms decays exponentially measured by the operator norm.
The first term of the gradient is from the last step of optimization, leading to the one-step gradient,
Jy
θy Oh
(13)
Ox
Oh' 0x
The one-step gradient is linear approximation of the BPTT algorithm when + oo according to the Proposition 2. It is easy to implement, requiring a no_grad operation in PyTorch (Paszke et al., 2019) or stop_ graddent operation in TensorFlow (Abadi et al., 2016) and reducing the time and space complexity from Ot)) in BPTT to (1). We test adding more terms to the gradient but its performance is worse than using one step. According to experimental results, one-step gradient is acceptable to back pec-pprope gradient through MDs.
5

---

Published as a conference paper at ICLR 2021
Table 2: Ablation components of Hamburger with NMF Ham.
Method
ml)((%)
Params
baseline
75.9(75.7)
32.67M
basic
78.3(77.8)
+0.50M
ham
75.8(75.6)
+0.50M
upper bread
77.0(76.8)
+0.25M
lower bread
77.3(77.2)
+0.25M
only ham
77.0(76.8)
+0M
EXPERIMENTS
In this section we present experimental results demonstrating the techniques described above. Two vision tasks that benefit lot from global information and attention mechanism attract including semantic segmentation (over 50 papers using attention) and deep generative models like GANs (most state-of-tie the-art GANs adopt self-attention since SAGAN (Zhang et al., 2019a)). Both tasks are highly competitive and thus enough for comparing Hamburger with self-attention. Ablation studies show the importance of MD in Hamburger well the necessity of the one-step gradient. We emphasize the superiority of Hamburger on modeling global context over self-attention regarding both performance and computational cost.
3.1 ABLATION EXPERIMENTS
We choose to conduct all ablation experiments on the PASCAL VOC dataset (Everingham et al., 2010) for semantic segmentation, and report mloU of 5 runs the validation set in the form of best(mean). ResNet-50 (He et al., 2016) with output stride 16 is the backbone for all ablation experiments. We employ 3x3 conv with BN (Ioffe & Szegedy, 2015) and ReLU to reduce channels from 2048 to 512 and then add Hamburger, the same location as popular attentions in semantic segmentation. For detailed training settings, please see Appendix E.
Latent dimension
Latent dimension
78.0
78.0
77.5
77.5
77.0
128
256
512
1024
32 48 64
72 96
Figure 3: Ablation on d and r
Breads and Hams We ablate each part of the Hamburger. Removing MD (ham) causes the most severe decay in performance, attesting to the impor- tance of MD. Even if only the parameter-free MD is added (only ham), the performance can visibly im- prove. Parameterization also helps the Hamburger process the extracted features. Bread, especially upper bread, contributes considerable performance.
MEAN
77.25
77.00
12
Latent Dimension and r is worth noting that there is no simple linear relation between d and r with performances measured by mloU, though d 8r is satisfactory choice. Experiments show that even r = performs well, revealing that can be very cheap for modeling the global context.
Figure 4: Ablation K

---

Published as a conference paper at ICLR 2021
Iterations K
We test more optimization steps in the evaluation stage. In general, the same K for training and test is recommended. K = is enough for CD and NMF, while even K is acceptable for VQ. Typically 3~6 steps enough since simple MD's prior is still biased, and full convergence can overfit it. The few iterations are cheap and act as early stopping.
3.2 A CLOSE LOOK HAMBURGER
To understand the behavior of Hamburger in the networks, we visualize the spectrums of representa- tions before and after Hamburger on the PASCAL VOC validation set. The input and output tensors are unfolded to C HW The accumulative ratio of squared largest r singular values over total squared singular values of the unfolded matrix has been shown in Fig. A truncated spectrum is usually observed in classic matrix decomposition models' results due to the low-rank reconstruction. the networks, Hamburger also promotes energy concentration while preserving informative details via the skip connection. Additionally, visualize the feature maps before and after Hamburger in Fig. 6. MD helps Hamburger learn interpretable global information by zeroing out uninformative channels, removing irregular noises, and completing details according the context.
Iopat image
Before Hamburger
After Hamburger
Ground Truth
68
Redundancy
Denoising
0.7
before
Incompletnnes
Completion
100
Figure 5: Accumulative Ratio
Figure 6: Visualization of feature maps
3.3 COMPARISON WITH ATTENTION
This section shows the advantages of MD-based Hamburger over attention-related context modules in computational cost, memory consumption, and inference time. We compare Hamburger (Ham) with self-attention (SA) (Vaswani al., 2017), Dual Attention (DA) module from DANet (Fu et al., 2019), Double Attention module from A Net (Chen et al., 201 8b), APC module from APCNet (He et al., 2019b), DM module from DMNet (He et al., 2019a), ACF module from CFNet (Zhang et al., 2019b), reporting parameters and costs of processing tensor Z E 1 memory usage is the key bottleneck of cooperating with attention in real applications. Hence we also provide the GPU load and inference time NVIDIA TITAN Xp. In general, Hamburger is light in computation and memory compared with attention-relaten global context modules.
in Tab. 3. Excessive
Table 3: Comparisons between Hamburger and context modules.
GPU Load
GPU Time
Method
Params
MACs
Train
Infer
Train
Infer
SA
1.00M
292G
5253MB
2148MB
242.0ms
82.2ms
DA
4.82M
79.5G
2395MB
2203MB
72.6ms
64.4ms
22
1.01M
25.7G
326MB
165MB
22. 22.9ms
8.0ms
APC
2.03M
17.6G
458MB
264MB
26.5ms
11.6ms
DM
3.00M
35.1G
557MB
268MB
65. mss7ms
23.3ms
ACF
0.75M
79.5G
1380MB
627MB
71.0ms
22.6ms
Ham (CD)
0.50M
16.2G
162MB
102MB
20.0ms
13.0ms
Ham (NMF)
0.50M
17.6G
202MB
98MB
15.6ms
7.7ms

---

Published as a conference paper at ICLR 2021
3.4 SEMANTIC SEGMENTATION
We benchmark Hamburger on the PASCAL VOC dataset (Everingham et al., 2010), and the PASCAL Context dataset (Mottaghi al., 2014), against state-of-the-art attentions. We use ResNet-101 (He et al., 2016) as our backbone. The output stride of the backbone is 8. The segmentation head is the same as ablation experiments. NMF is usually better than CD and VQ in ablation studies (see Tab. 1), Therefore. we mainly test NMF in further experiments. We use HamNer to represent ResNet with Hamburger in the following section.
Results on the PASCAL VOC test set, and the PASCAL Context validation set, are illustrated in Tab. and Tab. 5, respectively. We mark all attention-ban based models with in which diverse attentions compose the segmentation heads. Though semantic segmentation is a saturated task, and most contemporary published works have approximate performances, Hamburger shows considerable improvements previous state-of- tfte-of-theeart the-art art attention modules.
Table 4: Comparisons with state-of- setic-of-thh-aae the-art on the PASCAL VOC test set w/o COCO pretraining.
Table 5: Results on the PASCAL-Context Val set.
Method
mloU(%)
Method
mloU(%)
PSPNet (Zhao et al., 2017)
47.8
PSPNet (Zhao et al., 2017)
82.6
SGR* (Liang al., 2018)
50.8
DFN (Yu et al., 2018)
82.7
EncNet (Zhang et al., 2018)
51.7
EncNet (Zhang et al., 2018)
82.9
DANet" (Fu et al., 2019)
52.6
DANet* (Fu et al., 2019)
82.6
EMANet" (Li et al., 2019)
53.1
DMNet (He al., 2019a)
84.4
DMNet" (He et al., 2019a)
54.4
APCNet* (He et al., 2019b)
84.2
APCNet (He et al., 2019b)
54.7
CFNet* (Zhang et al., 2019b)
84.2
CFNet" (Zhang et al., 2019h)
54.0
SpyGR* (Li et al., 2020)
84.2
SpyGR* (Li et al., 2020)
52.8
SANet" (Zhong et al., 2020)
83.2
SANet" (Zhong et al., 2020)
53.0
OCR" (Yuan et al., 2020)
84.3
OCR" (Yuan et al., 2020)
54.8
HamNet
85.9
HamNet
55.2
3.5 IMAGE GENERATION
Attention presents as the global context de- scription block in deep generative models like GANs. Most state-of of- art GANs for condi- tional image generation integrate self-attention into their architectures since SAGAN (Zhang al., 2019a), e.g., BigGAN (Brock et al., 2018). SAGAN (Lučić et al., 2019), and LOGAN (Wu al., 2019). It is convineing to benchmark MD- based Hamburger in the challenging conditional image generation task on ImageNet (Deng et al., 2009).
Table 6: Results on ImageNet 128x 128. are
from Tab. and Tab. 2 of Zhang et al. (2019a).
Method
FID
ING.AN-rojection"
27.62
SAGAN
18.28
HamGAN-baby
16.05
YLG
15.94
HamGAN-strom,
14.77
Experiments are conducted to compare Hamburger with self-attention on ImageNet 128x 128. Self- attention is replaced by Hamburger with NMF ham in both generator and discriminator at feature resolution 32, named as HamGAN-baby HamGAN achieves an appreciable improvement in Fréchet Inception Distance (FID) (Heusel et al., 2017) over SAGAN. Additionally, we compare Hamburger with a recently developed attention variant Your Local GAN (YLG) (Daras et al., 2020) using their codebase and the same training settings, named HamGAN-strons HamGAN-stron over 5% improvement in FID while being 15% faster for the total training time and 3.6x faster for the module time (1.54 iters/sec of HamGAN, 1.31 iters/sec of YLG, and 1.65 iters/sec without both context modules, averaged from 1000 iterations) on the same TPUv3 training platform.
offers
4 RELATED WORK
General Survey for Attention The last five years have witnessed a roaring success of attention mechanisms (Bahdanau al., 2015; Mnih et al., 2014; Xu al., 2015; Luong al., 2015) in
8

---

Published as a conference paper at ICLR 2021
deep learning. Roughly speaking, the attention mechanism is a term of adaptively generating the targets' weights to be attended according to the requests. Its architectures are diverse, and the most well-known one is dot product self-attention (Vaswani et al., 2017). The attention mechanism has a wide range of applications, from a single source (Lin et al., 2017) to multi-source inputs (Luong et al., 2015; Parikh et al., 2016), from global information discovery (Wang et al., 2018; Zhang et al., 2019a) to local feature extraction (Dai et al., 2017; Parmar et al., 2019).
Previous researchers attempt to explain the effectiveness of attention mechanisms from numerous aspects. Capturing long-range dependencies (Wang et al., 201 1).. sequentially decomposing visual scenes (Eslami et al., 2016; Kosiorek et al.. 2018), inferring relationships between the part and the whole (Sabour et al., 2017; Hinton et al., 2018), simulating interactions between objects (Greff et al., 2017; van Steenkiste et al., 2018), and learning the dynamics of environments (Goyal et al., 2019) are often considered as the underlying mechanisms of attention.
One common idea from biology is that attention simulates the emergence of concerns in many unconscious contexts (Xu et al., 2015). Some work tries to interpret the attention mechanism by visualizing or attacking attention weights (Serrano & Smith, 2019; Jain & Wallace, 2019; Wiegreffe & Pinter, 2019), while others formulate attention into non-local operation (Wang et al., 2018) or diffusion models (Tao et al., 2018; Lu et al., 2019) or build attention- models via Expectation Maximization (Greff et al., 2017; Hinton et al., 2018; Li et al., 2019) or Variational Inference (Eslami et al., 2016) 6) on a mixture model. A connection between transformer and graph neural network is discussed well (Liang al., 2018: 8; Zhang al., 2019c). Overall, discussions towards attention still far from reaching agreements or consistent conclusions.
Efficient Attention Recent works develop efficient attention modules via low-rank approximation in both computer vision (Chen et al., 201 8b; Zhu et al., 2019; Chen et al., 2019; Li et al., 2019) and natural language processing (Mehta et al., 2019; Katharopoulos et al., 2020; Wang et al., 2020; Song et al., 2020). Technically, the low-rank approximation usually targets at the correlation matrix, i.c. the product of Q and K after the so so fmaa operation, using product of two smaller matrices to approximate the correlation matrix and applying the associative law to save the memory cost and computation, where the approximation involves kernel functions or other similarity functions. Other works (Babiloni et al., 2020; Ma al., 2019) make efforts to formulate attention into tensor form but may generate large intermediate variables. In this paper, we do not approximate attention or make it efficient. This paper formulates modeling the global context computation and memory efficiency is by-product of the low-rank assumption on the clean signal subspace and optimization algorithms as architectures.
low-rank completion problem. The
Matrix Decomposition in Deep Learning There is a long history of combining MD with deep learning. Researchers focus on reducing the parameters in the networks via factorization on the weights, including the softmax layer (Sainath et al., 2013), the convolutional layer (Zhong et al., 2019), and the embedding layer (Lan et al., 2019). Tariyal et al. (2016) attempts to construct deep dictionary learning for feature extraction and trains the model greedily. This paper tries to factorize the representations to recover a clean signal subspace as the global context and provide a new formulation for modeling the long-range dependencies via matrix decomposition.
CONCLUSION
This paper studies modeling long-range dependencies in the networks. We formulate learning the global context as a low-rank completion problem. Inspired by such a low-rank formulation, we develop the Hamburger module based on well-studied matrix decomposition models. By specializing matrix decomposition's 's objective function, the computational graph created by its optimization algorithm naturally defines ham, Hamburger's core architecture. Hamburger learns interpretable global context via denoising and completing its input and improves the spectrum's 's concentration. It is startling that, when prudently coped with the backward gradient, even simple matrix decomposition proposed 20 years ago is as powerful self-attention in challenging vision tasks semantic segmenta- tion and image generation, as well as light, fast, and memory-efficient We plan to extend Hamburger to natural language processing by integrating positional information and designing decoder like Transformer, build a theoretical foundation for the one-step gradient trick or find a better method to differentiate MDs, and integrate advanced MDs in the future.
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
Zhouchen Lin is supported by NSF China (grant no.s 61625301 and 61731018), Major Scientific Research Project of Zhejiang Lab (grant no.s 2019KB0ACO1 and 2019KB0AB02), Beijing Academy of Artificial Intelligence, and Qualcomm. We thank Google's Tensorflow Research Cloud (TFRC) for providing us Cloud TPUs.
REFERENCES
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg. Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI' 16, pp. 263-283. USENIX Association, 2016. 5,
Francesca Babiloni, Ioannis Marras, Gregory Slabaugh, and Stefanos Zafeiriou. Tesa: Tensor element self-attention via matricization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), June 2020.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, 2015. 8
Yoshua Bengio. The consciousness prior. arXiv preprint
2017. 2
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018. 8
Liang-Chieh
Liang- Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018a. 17, 18
Yunpeng Chen, Yannis Kalantidis, Jianshu Li, Shuicheng Yan, and Jiashi Feng. A*2-nets: Double attention networks. Advances in Neural Information Processing Systems 31, pp. 352-361.
Curran Associates, Inc., 2018b. 7.9
Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis. Graph-based global reasoning networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019. 9
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEFE international conference on computer vision, pp. 764-773, 2017.
Giannis Daras, Augustus Odena, Han Zhang, and A. Dimakis. Your local gan: Designing two dimensional local attention mechanisms for generative models. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14519-14527, 2020. 8, 18
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009. 2, 8, 17, 18
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Inderjit S. Dhillon and Dharmendra S. Modha. Concept decompositions for large sparse text data using clustering. Machine Learning, 42(1):143-115, Jan 2001. ISSN 1573-0565. doi: 10. 00.1033/A110076122299999 1, 15
10

---

Published as a conference paper at ICLR 2021
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, koray kavukcuoglu, and Geoffrey Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems 29, pp. 3225-3233.
Curran Associates, Inc., 2016.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International jourrna journal of computer vision, 88(2):
303-338, 2010. 2, 6, 8, 17
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3146-3154, 2019. 7, 17
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. arXiv preprint arXiv:1909.10893 2019. 2,
R. M. Gray and D. L. Neuhoff. Quantization. IEEE Transactions on Information Theory, 44(6): 2325-2383, Oct 1998. doi:
1,4
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems 30, pp. 6691-6701, Curran Associates, Inc., 2017. 9
Junjun He, Zhongying Deng, and Yu Qiao. Dynamic multi-scale filters for semantic segmentation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3561-3571, 2019a.
Junjun He, Zhongying Deng, L. Zhou, Yali Wang, and Yu Qiao. Adaptive pyramid context network for semantic segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7511-7520, 2019b. 7,8
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016. 6, 8, 17
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in Neural Information Processing Systems, 2017. 8, 18
Geoffrey Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing, In International Conference Learning Representations, 2018. 9
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 456, 2015. 4, 6
Sarthak Jain and Byron Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543-3556, 2019. 9
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and Franccois Fleuret. Transformers are mns: Fast autoregressive transformers with linear attention. ArXiv, abs/2006. 16236. 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR,
abs/ toss/442.6990. 2015. 18
Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM Review, 51(3): 455-500, 2009. doi: 10. 10.1137/07701111X
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling moving objects. Advances in Neural Information Processing Systems, pp. 8606-8616, 2018. 9
11

---

Published as a conference paper at ICLR 2021
Karol Kurach, M. Lucic, Xiaohua Zhai, M. Michalski, and S. Gelly. A large-scale study regular- ization and normalization in gans. In JCML, 2019. 18
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2019. 9
Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factoriza- tion, Nature, 401(6555):788, 1999. 4
Daniel D. Lee and Sebastian Seung. Algorithms for non-negative matrix factorization. In Advances in Neural Information Processing Systems 13. MIT Press, 2001. 4
Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation- maximization attention networks for semantic segmentation. In Proceedings of the IEEE Interna- fional Conference on Computer Vision, pp. 9167-9176, 2019. 8, 17
Xia Li, Y. Yang, Qijie Zhao, Tian cheng Shen, Zhouchen Lin, and Hong-Cheu Liu. Spatial pyramid based graph reasoning for semantic segmentation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8947-8956, 2020.
Xiaodan Liang. Zhiting Hu, Hao Zhang, Liang Lin, and Eric P Xing. Symbolic graph reasoning meets convolutions. In Advances in Neural Information Processing Systems, pp. 1853-1863, 2018. 8,
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint 173.03370.. 2017. 9
Canyi Lu, Jinhui Tang, Shuicheng Yan, and Zhouchen Lin. Generalized nonconvex nonsmooth low-rank minimization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2014. 2
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie- Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv: wwt.:v:1906007777000 2019. 9
Mario Lučić, Michael Tschannen, Marvin Ritter, Xiaohua Zhai Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. In International Conference on Machine Learning, pp. 4183 4192,, 2019. 8
Minh Minn-Thhag Luong, Hieu Pham, and Christopher D Mamning. Effective approaches to attention- based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, 2015. 8,9
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. tensorized transformer for language modeling. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. Online learning for matrix factoriza- tion and sparse coding. Journal of Machine Learning Research, 11(Jan):19-60, 2010. 2
Sneha Mehta, H. Rangwala, and N. Ramakrishnan. Low rank factorization for compact multi-head self-attention. ArXiv, abss/191200082 2019. 9
Takeru Miyato and Masanori Koyama. eGANs with projection discriminator. In International Conference on Learning Representations, 2018. 17, 18
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference Learning Representations, 2018.
18
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. Recurrent models of visual attention. In
Advances in neural information processing systems, pp. 2204-2212, 2014. 8
12

---

Published as a conference paper at ICLR 2021
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016. 17
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam Cho, Seong- Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 891-898, 2014. 2, 8, 17
Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2249-2222 2016." 9
Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In Advances in Neural Information Processing Systems, pp. 68-80, 2019. 9
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pp.
2019.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 1856-3866, 2017. 9
T. Sainath, Brian Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensiona output targets. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6655-6659 2013.
Soñia Serrano and Noah A Smith. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1931-2951, 2019.
Kyungwoo Song, Yohan Jung, Dong-Jun Kim, and I. Moon. Implicit kernel attention. ArXiv, abs/2006.061477, 2020. 9
Yunzhe Tao, Qi Sun, Qiang Du, and Wei Liu. Nonlocal neural networks, nonlocal diffusion and nonlocal modeling. In Advances in Neural Information Processing Systems 31, pp. 506. Curran Associates, Inc., 2018.
Snigdha Tariyal, Majumdar, R. Singh, and Mayank Vatsa. Deep dictionary learning. IEEE Access, 4:10096-10109, 2016. 9
Madeleine Udell, Corinne Horn, Reza Zadeh, and Stephen Boyd. Generalized low rank models. Foundations and Trends in Machine Learning, 91):1-111, 2016. ISSN 1935-8237. doi: 10. 15.1561/ 2200000055.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. International Conference Learning Representations, 2018. 9
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pp. 5998-60098
Curran Associates, Inc., 2017. 1, 7,9
Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. ArXiv, abs/2006.04768. 2020. 9
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794-7803, 2018. 1,
Paul Werbos et al. Backpropagation through time: what does and how to do it. Proceedings of the IEFF, 1 1990. 1,
13

---

Published as a conference paper at ICLR 2021
Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 11-20, 2019. 9
John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. In Advances in neural information processing systems. pp. 2080-2088, 2009. 2
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy P. Lillicrap. Logan: Latent optimisation for generative adversarial networks. ArXiv, abs/1912.00953. 2019. 8
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048 2057, 2015. 8,9
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Learning discriminative feature network for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1857-1866, 2018. 8
Yuhui Yuan and Jingdong Wang. Ocnet: Object context network for scene parsing. arXiv preprint arXiv:1809. 00916, 2018. 17
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-content coeectcoontextull representations for semantic segmentation. arXiv preprint arXiv:1909, 1065, 2020. 8, 17
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. ICML, 2019a. 1, 6, 8, 17, 18
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. ArXiv, 1bs/1991..2007, 2020. 18
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7151-7160, 2018. 8
Hang Zhang, Han Zhang, Chenguang Wang, and Junyuan Xie. Co-occurrent features in semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 548 118-557,22 2019b. 7, 8,
Songyang Zhang, Shipeng Yan, and Xuming He. LatentGNN: Learning efficient non-local relations for visual recognition. In Proceedings of the 36th International Conference Machine Learn- ing(ICML) volume 97 of Proceedings of Machine Learning Research, pp. 7374-7383. PMLR, 2019c. 9
Zhengdong Zhang, Arvind Ganesh, Xiao Liang, and Yi Ma. Tilt: Transform invariant low-rank textures. International journal of computer vision, 99(1):1224, 2012. 2
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 2881-2890, 2017. 8, 18
Zhisheng Zhong, Fangyin Wei, Zhouchen Lin, and Chao Zhang. Ada-tucker: Compressing deep neural networks via adaptive dimension adjustment tucker decomposition. Neural networks: the official journal of the International Neural Network Society, 110:104-115, 2019.
Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei- Shi Zheng, Jonathan Li, and Alexander Wong. Squeeze-and anneeze--nd-attentioo networks for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020. 8
Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 593-602, 2019. 9
14

---

(,(r, Published as a conference paper at ICLR 2021
A
TABLE OF NOTION
Table 7: Summary of notations in this paper
A scalar.
x
A vector.
X
A matrix.
2
A tensor.
1,
A vector whose n elements are all
X;
i-th column of matrix X.
h
Vector h at time step f.
By
Jacobian matrix of y w.r.t X.
X
Operator norm.
LXF
Frobenius norm
diag
Map a vector diagonal matrix.
cosine
Cosine similarity used in Alg. 1.
softmaz
Column softmax function.
normalize
Column-wise normalization by L2 nomn.
B HAMS
Additionally, we introduce another type of ham adopted by Hamburger, Concept Decomposition.
Concept Decomposition the following form:
We first enhance Concept Decomposition (Dhillon & Modha, 2001) to
min D.C
X +
(14)
s.t.
E max QDD.X)
D
This problem has closed solution w.r.t C under given D. iee, C CD X. Since D - 3I is positive definite matrix with a regularized conditional number, the inverse can be more numerically stable than the original where semi-positive definite matrix D D is given under 0. practice, 0.01 or 0.1 makes no difference for 3.
Algorithm 3 Ham: Soft CD Input X. Initialize D, C
for from 1 to K do
C c so softmar( cossine D, X)) D t normalizel (
end for
C - D - D
Output X DC.
The dictionary in CD is given by spherical K-means (Dhillon & Modha, 2001) with objective Q X), mentioned in Eq. (14).
ES S cosine (x, d,)
arg max
(15)
s.t.
|d,||= = 1.
The same strategy as VQ is adopted to make the whole algorithm differentiable, however, in which each column of D is normalized be unit vector and thus differs from VQ.
15

---

Published as a conference paper at ICLR 2021
C PROOF OF PROPOSITIONS
We investigate an abstract RNN model inspired by numerical methods to understand the drawbacks of BPTT algorithm in differentiating the optimization algorithm of MDs, M. We show the propositions in Sec. 2.3 to illustrate the unstable gradient from M when using BPTT algorithm, considering MDs' nature optimization algorithms.
Proposition 1
The iterations of F have linear convergence.
Proof. obvious that JF is a contraction mapping w.r.t h under arbitrary given x. We then conclude {h') } is Cauthy sequence and F(*, x) admits a unique fixed point ht due to Banach Fixed Point Theorem.
++
h"1
[[ x) - F(h", x(lll
(16)
Ln lh' h"ll
Eq. (16) shows the linear convergence.
Proposition 2
lim By Ux
By 2h
0F
)
-F 85 Ox
Proof. Note that F(+,)) admits a unique fixed point h" under arbitrary given i.e.
h F(h", x)
h F(h",x)= =0
(17)
By differentiating the above equation, we can obtain
0F
Oh
0F
(I
(18)
Oh
Ox
Ox
The Jacobian matrix I au Immediately, we have oh
is invertible, which implies the existence of the implicit function h' hx))..
lim oy
Qy
Oh
0y
OF
0F 0x
(I
(19)
0x
oh
8x
Oh
8h
which completes the proof.
Proposition 3 lim
dy 2
O, lim
S
Lake
8x
I- Lb
100
$0000
Proof.
Ty 0h
dy
Oh
dy
oh
=
0h
Oh
2h
0h
S Lg
(20)
2m1
Then we have:
dy
lim
0.
(21)
(-000
7h
dy
dy
0h
oh
0x
Oh
Ohi
dx
1=0
j=i+1
dy
Oh
Oh
||
Oh
0h
0x
jmi41
(22)
< Lg
Le
i=0 Lg Lu (1 Li))
TTh
Then we have:
θy ||
L.g Tox
lim
<
(23)
Ox
Ln
16

---

Published as a conference paper at ICLR 2021
D DATASETS
PASCAL VOC The PASCAL VOC dataset (Everingham et al., 2010) is widely used dataset in both semantic segmentation and detection. For segmentation, it contains 10,582 images for training, 1,449 images for validation and I 1,456 images for testing. PASCAL VOC dataset involves 20 foreground object classes and a background class for segmentation and detection.
The PASCAL Context dataset (Mottaghi et al., 2014) is challenging dataset in semantic segmentation, which provides detailed labels and involves 59 foreground object classes and a background class for segmentation. consists 4,998 and 5,105 images training and validation set, respectively.
PASCAL Context
ILSVRC 2012 The ILSVRC 2012 (ImageNet) (Deng et al., 2009) dataset contains 1.3M training samples and 50k test images, categorized into 1000 object classes. We resize images to resolution 128 x 128, as done in SNGAN with projection (Miyato & Koyama, 2018) and SAGAN (Zhang et al., 2019a).
E DETAILS OF EXPERIMENTS
E.I ABALATION EXPERIMENTS
We use dilated ResNet-50 (He et al., 2016) with the output stride 16 as the backbone. The backhone is pre-trained on ImageNet (Deng et al., 2009). We apply poly-learning rate policy under batch size 12 and 30k iterations (about 35 epochs) for fast experiments (less than 12 hours using NVIDIA TITAN Xp GPU). The initial learning rate is set to 0.009, multiplied by (1 iter iteration, with momentum 0.9 and weight decay 0.0001. Hyperparameters of Hamburger are the same Appendix E.3.
iter iter 0.9 after each
E.2 A COMPARISON WITH ATTENTION MECHANISM
We report MACs according to Molchanov et al. (2016), using torchprofile' accurate profiler for Pytorch. Real-time cost is measured by built-in Pytorch memory tools on NVIDIA TITAN Xp GPU with input tensor Z [g1x512x122..2 128x128 Inference times averaged results from 20 repeats of 100 runs.
E.3 SEMANTIC SEGMENTATION
Architectures We use ResNet-101 (He et al., 2016) with the ouptput strid 8 as our backbone. We adopt dilated convolution (Chen et al., 2018a) preserve more detail spatial information and enlarge receptive field as done in the backbone of state-of-the-aa attention models (Fu et al., 2019; Li et al., 2019; Zhang et al., 2019b). We employ a 3x3 convolution layer with BN and ReLU to reduce channels from 2048 to 512 and then add Hamburger on the top of the backbone. Note that the input of Hamburger is tensor Z C We unfold Z to matrix E CCCHW and set d2 = and HW for Hamburger. Latent dimension d and i.e., the column vectors' dimension of the input matrix X E dx to M and the number atoms in the dictionary D E Rexd are set to 512 and 64. The iterations of MD's optimization algorithm, K, are set to 6. Non-negative Matrix Factorization (NMF) is our default ham for semantic segmentation.
Data augmentation In the training stage, we apply random left-right flipping, random scaling (from 0.5 to 2), and cropping to augment the training data. Images are resized to 513x513 for the PASCAL VOC dataset and the PASCAL Context dataset. In the test stage, the multi-scale and flipping strategy is applied other tfae-oof-the-ar attention-based models (Fu et al., 2019; Yuan & Wang, 2018; Yuan et al., 2020).
Phttps:/ httpl:::///gtthub.cooo www...ow/........ccctot http://iiiaaannitttorrrrrorooooo tu/torchprofile
17

---

Published as a conference paper at ICLR 2021
Optimization We use mini-batch SGD with momentum 0.9 train HamNet. Synchronized Batch Normalization is adopted in experiments on semantic segmentation. All backbones are fine-tuned from ImageNet (Deng et al., 2009) pre-training. Following previous works (Zhao et al., 2017; Chen et al., 20) we apply poly-learning rate policy. The initial learning rate is multiplied by iter y0.9 For the PASCAL VOC dataset, learning rate, weight decay, batch size, iterations set to 0.009, 0.0001, 16, and 60k, respectively. We fine-tune HamNet the PASCAL VOC trainval set with the learning rate down to a tenth. The learning rate, weight decay, batch size, iterations are 0.002, 0.0001, 16, and 25k for the PASCAL-Context dataset.
(1 stermoo
E.4 IMAGE GENERATION
We use the official GAN codebase" from Tensorflow (Abadi al., 2016) and TF-GAN to train HamGAN and evaluate FID.
Architectures Experiments on ImageNet are conducted using the same architecture as SAGAN (Zhang et al., 2019a), and YLG (Daras et al., 2020), including Spectral Normalization (Miy- ato et al., 2018) in both the generator and the discriminator, conditional Batch Normalization in the generator, and class projection in the discriminator (Miyato & Koyama, 2018). Hamburger with NMF ham is placed at feature resolution 32x in both the generator and the discriminator where self-attention can obtain the best FID according to Zhang et al. (2019a). We use = 8r for Hamburger, and d is the same as the input channels, while the optimization steps K are 6, Restricted to expenditures of training GANs on ImageNet, d, r. and K are decided according to the ablation experiments on semantic segmentation without new ablation experiments.
Optimization For all models, we use Adam (Kingma & Ba, 2015) optimizer with TTUR (Heusel al., 2017). HamGAN employs the same training settings as SAGAN (Miyato et al., 2018) and YLG (Daras et al., 2020), respectively.
Evaluation metrics The quality of images generated by GANs are evaluated by Fréchet Inception Distance (FID) (Heusel al., 2017). Lower FID indicates that the model can generate higher-fidelity images. In our experiments, 50k images are sampled from the generator to compute FID. We evaluate HamGAN for 6 runs and report the best FID to approximately match the convention in the modern GAN research like Kurach et al. (2019) and CR-GAN (Zhang et al., 2020), reporting top 5%%15% results in the experiments.
F FURTHER RESULTS FROM ABLATION EXPERIMENTS
Table 8: Ablation on initializations.
Init fixed
NMF 77.4(77.3)
CD
VQ
77.7(77.4)
77.3(76.9)
learned
76.8(76.5)
75.0(73.7)
75.9(75.8)
random
78.3(77.8)
77.9(77.3)
77.7(77.4)
online
77.8(77.5)
78.1(77.5)
78.0(77.2)
Initialization We test four types of initialization for the dictionary D. including fixed initialization, learned initialization, random initialization, and warm start with online update. Usually, random initialization is the best choice that means we can sample each entry of D from given distribution like Uniform(O. 1) as the initialization of the optimization algorithm M. For NMF, after initializing D. we initialize C so sof aofam cosine( D.X)) since K- K-means is usually applied for initializing NMF and this initialization for C is equivalent to a single update in Spherical K K-means. A special reminder is that it is not suitable to initialize either D or C values close to 0 due to the property of the MU rule. So the temperature T is recommended to be a higher value like this
Phttps: //g thub combtenso foow//a
18

---

Published as a conference paper at ICLR 2021
initialization for C. Random initialization also works for C in NMF with scores 77.8(77.6) when sampling Ciy ~ Uniform(0, 1). Note that learned initialization is always the worst one since the BPTT algorithm is employed to learn the initialization that the gradient from M may impede the training of the backhone, instead the one-step gradient. Warm start benefits MD with unit vectors in the dictionary D like CD. In general, random initialization is good enough for all three selected MD models. A possible reason is that it can enforce the network to adapt to the results solved by different initializations during the training process, acting like inner augmentation.
Temperature 7 As we have claimed, when T approaches 0, we can get a solution close to the
Table 9: Influence of temperature T with CD ham.
original problem in both VQ and CD. In VQ and CD experiments, relatively low temperature T more recommended to solve better D for MD. However, will not receive more gains but increase the variance during training we further lower T.
Temperature T
mloU(%)
77.1(77.0)
0.1
78.2(77.5)
0.01
78. 78.1(77.5)
Iterations K We take the iterations K of optimization algorithms M for all three MD models, NMF, CD, and VQ, into our consideration. More iterations and even fully converged results for M are tested in the evaluation stage but worse than little optimization steps. The smaller K, ranging from I to 8, can be treated as early stopping for the optimization algorithm M, obtaining satisfactory performances. For a detailed visualization, see Fig. 7, Fig. 8, Fig.
NMF_MEAM MEAN
CD MEAN
77.75
77.50
77.25
77.25
77,00
77.00
76.75
76.75
922
15
76.50
76,30
30
eval
eval
Figure 7: Impacts of on NMF
Figure 8: Impacts K on CD
VQ MEAN
76.25
12
76.00
10
evall
Figure 9: Impacts of on VQ
19

---

Published as a conference paper at ICLR 2021
G AN INTUITIVE ILLUSTRATION
In this section, we hope to give an example to help our readers develop insight into why the low-rank assumption is useful for modeling the representations' global context.
The low-rank assumption helps because it represents the inductive bias that the low-level repre- seniations contain limited and much less high-level concepts than the scale of the represenlations themselves. Imagine an image in which a person works on the road. Many eyper-pixels extracted by the backbone CNN will describe the road. Note that the road can be considered as repeats of small road patches, which means that we can represent the road via modeling the basic road patches and repeat them. Mathematically, is equivalent to finding a small set of bases corresponding to different road patches and a coefficient matrix C' that captures the relation between the elementary road patches and the hyper-pixels. This example illustrates that the high-level concepts, i.e., the global context, can be low-rank in the ideal situation.
The hyper-pixels describing the road patches have close semantic attributes. However, due to vanilla CNN's inefficiency for modeling the long-range dependencies. the learned representation contains too many local details and incorrect information, lacking global guidance. Imagine that the gloves patch locally, we think that this
the person in the image wears gloves. When patch describes gloves. When we consider the global context, we can understand that this patch part of person. The semantic information is hierarchical, depending on at which level hope to comprehend. This work aims at enabling the networks to understand the context globally via the low-rank completion formulation. We thus model the incorrect information, namely the redundancies and incompleteness, the representations into two parts, a low-rank global information matrix and noise matrix, by employing the optimization algorithm to recover the clean signal subspace, discard the noises, and enhance the global information via the skip connection. It could be learned from the data on how much global information the networks need for specific task.
noise matrix. To emphasize the global context, we decompose
20


Reviews:
[{"review_id": "YXqkKmsdtAD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a matrix decomposition-based method to capture spatial long range correlation in the neural network. The proposed method employs an optimization method, i.e. non-negative matric factorization, to reconstruct a low-rank embedding of the input data. The experimental results show that the proposed method outperforms various popular attention-based methods in recent years for various vision tasks, i.e. semantic segmentation and image generation. This paper is basically well written and easy to follow what they have done.\n\nHowever, I have several concerns that I listed as follows.\n- According to experimental results, I find the proposed matrix decomposition based method outperforms several attention based methods in mIoU for semantic segmentation and FIN for image generation. Nevertheless, the Parameters, FLOPs, memory and running time are only compared with Dual attention network. Please compare more attention based methods to verify the proposed method is more efficient attention based methods.\n- The proposed method is similar with EMA-Net, especially for employing concept decomposition as the optimization algorithm. Please discuss the relation and difference between the proposed method and EMA-Net.\n- I am very interested in the initialization of matrix decomposition. In the supplementary, authors only discuss the initialization of D, so what is the best initialization of C for non-negative matrix factorization? Besides, what is the warm start with online update?\n- The figures 3 and 4 are low quality, the text of coordinate is too small.\n\nAfter rebuttal:\nI appreciate the authors' detailed response to my questions, which largely addresses my previous concerns. \nIt's very pleasure to reviewing this interesting, innovative and well-written paper.\nA clear accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Mw3fzV0D9cH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n1. This paper finds that matrix decomposition (MD) performs well as well as the self-attention.\n2. According to the paper, MD approximate the given matrix with low-rank, and it might be helpful inductive bias\n3. MD can be implemented with vanilla matrix factorization or non-negative matrix factorization.\n4. For stable learning, this paper proposes an additional technique, one-step gradient instead of back-propagation through time (BPTT)\n5. To validate the performance, they experiment on semantic segmentation and image generation.\n\nPlease reply to the below questions.\n1. What is the advantage of MD over attention? According to the paper, the main advantage of MD is a less computational burden, but there are several works for linear time attention. Furthermore, it is hard to find sufficient reasons that MD performs better than attention-based models. \n2. There is no description for Table 1. Could you explain the table 1?\n3. Are there results for MD with BPTT instead of a one-step gradient on segmentation or image generation task?\n4. What is the \"human priors\" in the conclusion section? Does it denote the \"low-rank\"?\nIt is hard to understand that low-rank will be helpful.\nIf MD set r as same as min(d,n) instead of small r, does MD have lower performance?\n5. There are several works about 1) analyzing the low-rank problems in multi-head attention and 2) incorporating the low-rank approximation into attention. The discussion between this paper and related works is not enough.\n6. The title is \"Is Attention Better Than Matrix Decomposition?\", but the paper is only for the \"self\"-attention and MD.\nAre there results for Encoder-Decoder structured tasks (such as Translation)?\n\nI suggest that this paper discusses the relationship between MD and the below papers.\n\na) Factorization and Attention\n1. A Tensorized Transformer for Language Modeling\n\nb) Low-rank problems and attention\n1. Low-Rank Bottleneck in Multi-head Attention Models\n\nc) Low-rank attention\n1. Transformers are rnns: Fast autoregressive transformers with linear attention\n2. Linformer: Self-Attention with Linear Complexity\n3. Implicit Kernel Attention\n4. Compact Multi-Head Self-Attention for Learning Supervised Text Representations\n\n-----\nI read a valuable author's response, and I keep my positive score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "B4crPREsCh1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary:\nThe paper presents a method based on matrix decomposition (MD) for encoding global context in computer vision tasks. In particular, a \"Hamburger\" block is proposed encompassing matrix decomposition as its central part, between two linear projection layers. Direct comparison and relations are drawn between the proposed method and the widely adopted self-attention paradigm. The proposed method leads to improved results when Hamburger blocks are used instead of self-attention blocks, leading at the same time to reduced number of parameters, memory footprint and inference time.\n\n\n# Strengths:\nThe paper presents a novel and simple method for capturing global context, demonstrated on two challenging computer vision tasks, namely semantic segmentation and image generation. Important advantages of the proposed method, with respect to the self-attention method which is usually employed in such tasks, are the fact that it can be easily adapted to a wide range of models and problems, it is more efficient and has reduced memory requirements.\n\nA one-step gradient method is proposed for propagating the gradients through the MD optimization algorithm during training. One-step gradient is shown to overcome the unstable gradient problems of the back-propagation through time (BPTT) algorithm, leading to improved performance. A detailed analysis is provided comparing the two methods (one-step vs BPTT).\n\nThe evaluation is quite comprehensive comparing the proposed hamburger block with respect to similar self-attention blocks under several aspects (accuracy/FID score, GPU load, GPU time, FLOPs, nr. of parameters). A detailed ablation study is also presented showing the effect of the most important factors of the proposed contribution in the final performance.\n\nRegarding writing quality, the paper is clear and easy to read. The main ideas and contributions are clearly stated and presented. Some issues regarding the structure of the sections is discussed below.\n\n# Weaknesses:\nThe paper makes the more general claim that the proposed approach can be used for including any human inductive bias expressed through an optimization problem, however, only the problem of capturing global context, in place of self-attention, is explored. To support this more general claim it would be important to include some representative examples (even without providing a detailed evaluation on those).\n\nPossibly related to the previous point is the observation that non-negative matrix factorization (NMF) seems to always perform better than the other two matrix decomposition methods, Vector Quantization (VQ) and Concept Decomposition (CD). This leads to certain questions, as for example:\n* are there any problems that lend themselves better to the other types of MD?\n* is the performance of VQ and CD degraded because they are rendered soft?\n* what is the divergence of soft with original MD as far as the original optimization problem is concerned? The results of the ablation on temperature T provided in Appendix G partially show that the \"softening\" of the algorithm might negatively affect the accuracy.\n\nI find it also strange and possibly nearly violating formatting that the related work section is provided in the appendix. Some directly related work is discussed in the main text, yet a more detailed discussion considering a broader set of works only appears in the appendix. Also regarding related work, other works employing matrix decomposition in the context of deep learning, are not covered (e.g. (Sainath et al., 2013; Tariyal et al., 2016)).\n\nSainath, T. N., Kingsbury, B., Sindhwani, V., Arisoy, E., & Ramabhadran, B. (2013). Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing (pp. 6655-6659).\n\nTariyal, S., Majumdar, A., Singh, R., & Vatsa, M. (2016). Deep dictionary learning. IEEE Access, 4, 10096-10109.\n\n## Minor Comments\n* Table 1, no details are provided for the metric used for the results\n* Table 6, it would be better to specify the difference between the two entries of HamGAN\n* The text in almost all figure is quite small and very hard to read on typical zoom factors (~100%).\n \n# Rating Justification\nOverall, I think that the idea of using matrix decomposition as architectural element to capture global context is quite interesting and novel. Also the method shows advantages with respect to self-attention as far as efficiency and memory requirements are concerned. There are some issues regarding the generality of the proposed approach and the paper's structure, however, I think that the paper strengths exceed its weaknesses.\n\n# Rating and comments after the rebuttal\nThe authors addressed my concerns in their feedback and the revised manuscript they have provided. In particular, I find the claimed contributions much clearer now. In my view, they have also suitably addressed the concerns raised in the other reviews. As a result, I increase my rating to 8 as I think that this work is interesting, novel and impactful.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ZC0ehYGUAK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use matrix decomposition to construct low-rank representations to find the long-distance correlations in context, which is demonstrated more effective than popular self-attention mechanism. Combining linear transformation and matrix decomposition (core part), authors design Hamburger block to model global dependencies from input as residual output. The authors propose differentiable modified Vector Quantization and Non-negative Matrix Factorization to perform matrix decomposition. They propose one-step gradient, an approximation of Back-Propagation Through Time (BPTT) algorithm, to back-propagate gradient of matrix decomposition. They conduct experiments on semantic segmentation and image generation to demonstrate the superiority of their methods regarding modelling global dependencies and computational cost.\n\nI believe that there is clear novelty in the proposed method. The paper is well written. One weakness is that the experiment analysis is a little weak. It will be great to see stronger experiments in the final.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an alternative to self-attention, based on matrix factorization, and apply it to computer vision problems such as semantic segmentation. The method is simple and novel and obtains competitive results compared to existing approaches. The reviewers found the paper well written and easy to understand. For these reasons, I recommend to accept the paper.
Decision: Accept (Poster)