Sharp Attention for Sequence to Sequence Learning
Authors: Pei Zhang
Hua Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Attention mechanism
sequence to sequence learning
reinforcement learning

Abstract: Attention mechanism has been widely applied to tasks that output some sequence from an input image. Its success comes from the ability to align relevant parts of the encoded image with the target output. However, most of the existing methods fail to build clear alignment because the aligned parts are unable to well represent the target. In this paper we seek clear alignment in attention mechanism through a \emph{sharpener} module. Since it deliberately locates the target in an image region and refines representation to be target-specific, the alignment and interpretability of attention can be significantly improved. Experiments on synthetic handwritten digit as well as real-world scene text recognition datasets show that our approach outperforms the mainstream ones such as soft and hard attention.

Under review as a conference paper at ICLR 2022
SHARP ATTENTION FOR SEQUENCE TO SEQUENCE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Attention mechanism has been widely applicd to tasks that output some sequence from input image. Its success comes from the ability align relevant parts of the encoded image with the target output. However, most of the existing methods fail build clear alignment because the aligned parts are unable well represent the target. In this paper we seek clear alignment in attention mechanism through a sharpener module. Since it deliberately locates the target in an image region and refines representation to be target-specific the alignment and interpretabilityy of attention can significantly improved. Experiments on synthetic handwritten digit as well as real-world scene text recognition datasets show that our approach outperforms the mainstream ones such as soft and hard attention.
3
8
10
11
INTRODUCTION
In modern sequence to sequence learning, attention mechanism has become key building block, 13 because it helps identify relevant parts the input sequence and align them with the target output at each time step. Such alignment resembles fixation in human vision, where only the object of interest
15 falls on the fovea in the retina, leading the visual scene outside to be largely ignored. Thanks to such ability select perceptual information, attention mechanism has been successfully applied many visual tasks, such as scene text recognition (Shi et al. 2019) and image captioning (Xu et al. 2015).
16
17
18
Although variety of attention mechanisms have been proposed build alignment, most of them fail to achieve clear alignment. Soft attention (Bahdanau et al. 20155] Luong et al. (2055), the most popular one, aligns a weighted average of the input sequence with the target output throughout the time. Since the weights are never zeros, irrelevant parts are inevitably involved in the alignment and
19
20
21
22 may introduce distraction. For distinct alignment, hard attention (Xu et al.) 2015) enforces exactly one input part is employed, regardless of whether it represents the target or not, and thus may still
23
24 suffer from irrelevant parts. Besides, existing attention mechanisms often regard the input sequence as fixed during alignment establishment. If the target representation given by the selected part(s)
26 is poor, there is no way to fix it. This is especially the case in visual sequence learning, where features precomputed by a convolutional neural network (CNN) before being fed into attention
28 mechanisms. As each feature only characterises a local fixed image region (i.e. the receptive field),
29 hardly covers the appearance of the target exactly, thus leading noisy representation. See Fig. [b)
30 for an example.
31 In this work address the construction of clear alignment in attention mechanism. This is achieved
32 by aligning the target output with image regions instead of features, which is more natural approach
33 to alignment for visual sequence learning. sharpener module is then used make the aligned
34 region as specific as possible to the target, essentially clear alignment. While it can take any form,
35 the module explored in this paper consists of a localiser and an encoder. The former locates the
36 target in the region, while the latter extracts features from the result for alignment. It is such accurate
37 and specific representation that makes attention mechanism able to pay close attention to the target, leading to improved alignment and thus interpretability. The sharpener can be trained along with
39 any sequence-to seseence-to-sqqueee sequence (Seq2Seq) model through back vack-prooagation without extra supervision. Nonetheless, it is also possible to guide its training to further improve alignment quality if auxiliary 41 information is available (see Sect. 4.I). Therefore, the sharpener naturally lends itself to direct attention manipulation, which is yet not available in most of the existing attention mechanisms.

---

Under review as a conference paper at ICLR 2022
softmax
dtl
DtM
ax+
Woodcraft
tanh
Φ(X, ax.)
Ct
fc
fc
X
h
2M
W
(a)
(b)
Figure 1: (a) An outline of the attention mechanism. A query vector f is compared with an input sequence X figure out where look via a multilayer perceptron (see the grey box), leading to a set of weights ag A context vector C₁ is then computed for alignment by a function ø where attention mechanisms generally differ. Soft attention computes C₁ as weighted average of X while hard attention does it by randomly sampling an element from X. (b) Poor target representation in attention mechanism. Each element of X only represents fixed region defined by the receptive field (see yellow boxes). It is hard to accurately represent the target without the distraction of redundant missing parts (see letters 'f" and "W" for examples).
SHARP ATTENTION
44
2.1 ALIGNMENT
Let x= -xI1...., .M3 be an input sequence of length M, where æ; E R. is feature vector 46 representing a region an input image. Usually, X comes from the last convolutional layer of backbone network, and æ; delineates region of fixed size given by the receptive field of that layer. Similarly, we denote the output sequence of length N as Y = f........ yN), where ye E RK is one-of-K encoded vector indicating discrete token in vocabulary of size K. Our goal is to 50 learn a model that can accurately predict y, by choosing appropriate æ; in a sequential process. This implicitly asks for building alignment between X and Y at each time step. To facilitate the construction, we introduce latent variable A = {aI,..., aN }. where a, E RM is a one-of-M 53 encoded vector indicating the index of the selected feature vector at some time. For example, an 1
45
48
51
52
54 refers to the i-th element of X (i.e, x;) being chosen to predict at time t. Usually, the selected feature is called context vector and denoted as Ct.
55
56 To learn the model, we maximise conditional probability
N p yrr Al y X)
N
p(([X) = Σ AIX) = EII
p arl (1)
A
t=1
t=1
a2
Vice
N
M
II Σ t=1 a:
plarly ven
a,)
p(a::
1/ c X X))((x)lleee...... xz; (2)
d=1
57 where the last two terms in I are obtained by applying chain rule on Y. and by using the assumption
58 that y, only depends on a, at time respectively. Equation (2) clearly defines two major components
59 to compute X)) One is the chance of selecting each element of X, and the other is the likelihood
60 of the target token given the selection. However, the computation of the latter is impractical when
61 is large because every element of X has to be considered. Two typical approximations to are thus
62 proposed, leading to the soft and hard attention mechanisms.
By using the first order Taylor expansion| we obtain the loss function for soft attention.
M
M
log (( = Σ log Σ
C..P .i.) ~ log y. Ce
(3)
'I.tt f(-) he function of some random variahle. By using Taylor's theorem, the first-order approximation to the expectation R[f(-}] is given by Elf(-)) f(E(-)).
2

---

Under review as a conference paper at ICLR 2022
where we define ou = p(ou = 1|y X) to simplify the notation. In 33. the context vector e is 65 given by ΣM d, which means that Yr is no longer predicted by a single element of X but rather 66 weighted average of X, thus leading to the break in alignment. To pursue the alignment such that 67 each target token only depends on one element of X, hard attention instead estimates a variational lower bound on log pY using Jensen's 's inequality,
N
log X
Σ log
Σ
oup(yrlye mi)
ou log p(r mi).
(4)
69 Now the optimisation of log p()))) can be thought repeating the following steps until happy:
70 (i) estimating p(a: yec, X) by fixing all model parameters; (ii) modifying the parameters to maximise (e:,; log ply,ly<c. 2;) using each x,. The underlying idea is to increase log p (X) by iteratively raising
71
the lower bound. Since it is infeasible to consider every æ;; as aforementioned, approximation is often adopted and achieved by Monte Carlo sampling (See Sect. 2.4 for details), thus leading O₁ be the sampled feature vector for hard attention.
72
73
74
75
2.2 LOss FUNCTION
76 Intuitively, if each æ; is an accurate representation of the target token when optimising
4 particularly
77 in the second step, there would be tight gap between log logp(Y|X) and its lower bound. In contrast,
78 any poor zy occurs, the gap may become large and thus result performance degradation. Subject
79 to the fixed local region, it is unlikely for æ; to well characterise the target token, which makes hard
80 attention hardly achieve clear alignment (see Fig. T[)). This motivates us to reformulate (4) for more
81 flexible representation of the target tokens.
82 Suppose we can break down the input image into a set of M local regions, each of which is sufficiently large to cover objects of interest. We would like to maximise the marginal log-likeling likelihood (
84 where R = (ri,..., TM) is the set of regions. Similarly, its lower bound é is given by
N
lYgp(Y A
Σ Σ Ellasleer Rt) log logp(v.lue.. R. a.):) =.
(5)
a
85 where a is still a one-of-M encoded vector but now refers to the index of the selected region at time
t. By working with (5). we not restricted to the representation given by X any more. Consider
87 xe to be a function of r parameterised by the backbone network, e.g., x = (r; 0g)) where 0g
88 denotes all the weights in the network. By plugging X = (f (r1; 0 ())........) ...., Sg (rM; 0 q)) into 44. it
89 is casy to see that the lower bound of log p(Y ( is equivalent to that of log p( RR) when partially
90 parameterising the two terms paalyy < R) and log y R, a.) using 0g. As é is valid for all
91 model parameters, it does not rely on any specific modelling. This allows us to separately model the
92 use of R in each term. For example, we may leverage the same backbone network for the first term to
93 get rough idea on where to look, while deliberately design network for the second term to sharpen
94 the focus. It such flexible parameterisation that makes the construction of clear alignment possible. Below we elaborate the modelling of each term in (5)
2.3 MODELLING
97
We use a variant of VGG (Shi et al. 2017) as the backbone network to not only create the set of regions but also process it in p(a:1e ye R2.. When the backbone network is CNN, we may take advantage of the implicitly defined sliding widow for region generation. For example, our backbone network effectively divides an input image of size 100x32 into 24 86 46 regions when extracting features from the final layer (Araujo et al. 2019). Note that the generation of R is arbitrary and we emphasise clear alignment with log P(yely R, a.). we leverage sharpener module that consists of a localiser and an encoder. The former seeks the target in the selected region while the latter extracts features from the result. While a natural choice of the localiser is object detectors, we instead resort to spatial transformer networks (STNs, Jaderberg
98
39
100
101
102
just use the sliding window for simplicity.
103
104
106
et al. (015)) for both computational and labelling efficiency. STN is a lightweight CNN that is able image region. Its training does not need expensive annotations such as bounding boxes, which are usually unavailable in sequential learning tasks. The encoder can take
to crop and transform
108
109
any form and the same CNN to the backbone to simplify the implementation. Let Z be the
3

---

Under review as a conference paper at ICLR 2022
f1
FM
fc
softmax
9:
Cte
Ti
h=
X
Ct
Input
Backbone
Attention
Sharpener
RNN
h
h
Figure 2: A generic Seq2Seq learning architecture with sharp attention. Given an input image, backbone network breaks down it into a set of regions and extracts features from each region, leading sequence of feature vectors X. An attention mechanism (see Fig. aa)) uses and a hidden state vector - to compute a categorical distribution ax, based on which region is randomly chosen and fed into sharpener module to compute the context vector c2 for clear alignment. recurrent neural network (RNN) takes in h and y/-1 (the token previous time step) to update its internal state, and then outputs current h2 for token prediction and next iteration (dashed line).
110 encoder output, which also a sequence of feature vectors similar to with length dependent on
size of the localisation result. The output of the sharpener is the context vector, whose computation
112 will be detailed in Sect. 2.5
The dependency of the current token y, all previous ones you the time is often modelled by 114 an RNN, c.g., long short-term memory (LSTM, Hochreiter Schmidhuber (1977). Specifically, is defined by
M
hr =
C; ho = fi
1 M
Σ 1=1
æ; 0,
Yo = O,
(6)
116 where f-(-) refers to the non-linear function defined by LSTM with parameters θ,, h4 is a hidden
117 state vector at time that summarises the history tokens Verr and is initialised by fully connected
118 layer fi(-) that takes an average of X as the input, and a zero vector used as the initial token yo-
119
Now we are ready to define the two terms in (5) As we use the backbone network to process R in p(arlyee <a. R), the probability of selecting particular region is given by
120
atii = p(as = 1 ::: R)) = p(a:; = h4-1, = (fftmax(fa(x;; (e-11:0)),
(7)
where (-) is the attention function described in Bahdanau et al. (2015) (Fig. [)). The probability of the output token Ú, given all previous ones as well as the selected region is computed by
122
p(y- Ú 1ytz R, ax) = ply. = D. h c.) softmax(/o( h,; 0e)),
(8)
123
where fe(.) is fully connected layer. An overview of the whole architecture is given in Fig.
2.4
OPTIMISATION
125
The differentiation of E w.r.t all model parameters yields the following learning rule
ae
I Bloggpuu ply, R, ax)
3 log log W <:: R)
ΣΣ p(arly R)
+ logply, yy (yyplyyyee R, a.)
00
00
where 9 is a collection of model parameters, e.g., 0 {0,, 0.. o,, O,, On, 0e) (0, for the sharpener module). To reduce the computational cost as explained in Sect. 2. the derivative at time is often numerically approximated by the Monte Carlo method as follows
127
de. 30
1
s
① log p p( b R, a;) + log logp(y.//e R, ai)
log p ces R)
Σ
p(a yen R)) 8=1
S
00
00
SStrictyy speaking. the alignment in this modelling is no longer conditionally independent throughout the time assumed in Sect. 2.T fact, this the modelling used in both soft and hard attention mechanisms (Bahdanau al. 2015) Xu et 2015). However, discussion on why these mechanisms fail to achieve clear alignment still applics.
We use the trick Vop(x; 0) p(=; Vo log ( 8) in the derivation.
4

---

Under review as a conference paper at ICLR 2022
where S is the number of samples à? drawn from categorical distribution defined by p(a, V<t R) (Xu et 2013). Similar to MMii et al. (2014) and Ba et al. (2013), this approximation yields a hybrid loss function asking for different optimisation strategies for the two terms in the square brackets. The former is optimised by a cross entropy loss together with the ground-truth token at time and
130
132
133
gradient hack-propagatton. By regarding the accumulated sum of ply, y R, a.) over the time
134
as a reward, the latter is achieved by the REINFORCE algorithm (Williams) 1992). To reduce the
135
high variance in gradient estimate caused by the unbounded log ply. (ellyee R, a,) (Ba al. 2015).
136
we follow Xu et al. (2015) to introduce a moving average baseline
S
N
bj = 0.9 x b +01 X
log log elylleee, R, a,),
NS
137
where j is the index of the mini-batch. Finally, use the following learning rule for optimisation
Oe
8 N
I @lgglyy (yc. R, ai) ><(log((y.lyee R. a,) - b) i log p( Me R))
~
pla R) s
09
S
00
00
(9)
138
where X, is learning paramepar We do not add entropy H[a,] to 9 to further reduce gradient
139
variance as in Xu et al. (2015), because it encourages a uniform distribution and breaks the alignment.
140
2.5
CONTEXT VECTOR
141
Rather than compute с from X like both soft and hard attention do, the proposed sharp attention leverages the encoder output. Specifically, we explore three ways to compute et given Z,, the encoder output at time t. The first one is pooling, where an average pooling of Ze is used for c, Inspired by the glimpse idea (Mnih et al. 2014), we combine the result from pooling with é", the feature vector associated with the sampled region, to incorporate both fine and coarse representation of the target token, leading to the second way-chuin. Alternatively, we may compute weighted average of the feature set (Z,, );} for better mixed representation. With 11, the weights can learned in similar way to the soft attention shown in Fig. [[). We call this last approach to c weighting.
142
143
144
145
147
3
RELATED WORK
149
150
In Xu et al. 2015), the alignment is treated as a latent variable to help define loss function, which is optimised by gradually increasing variational lower bound on marginal ligllikllioodd of the target output given the input sequence. Later, variety of variational inference techniques (Lawson et al.)
151
152
153
Deng et al. 2018 Bahuleyan et al. 2018) are proposed to further reduce the gap between the lower bound and the marginal aigliikllhhood Alternative approaches to approximating the marginal log- likelihood can be found in Shankar et al. (2018) and [Shanaar & Sarawagi (2019). The former cherry-picks set of alignments, computes the likeliood conditioned on each alignment and averages the results, while the latter extends the idea by enforcing Markov property on adjacent alignments. Instead of approximation, /Wu al. (2018) attempted to compute exact marginal log- likelihood by assuming that each alignment is conditionally independent across time steps. All of the above methods regard the input sequence as fixed in optimisation and thus cannot tailor the input pa to clear alignment.
2018
155
157
158
159
160
161
162
The idea of using relevant parts to improve attention has been explored in various tasks. Mei et al. 2016) adjusted the weights of the input parts resulting from soft attention to highlight the most relevant ones for selective generation. A similar work can be found in NNalappti al. (0016). where keywords are interwoven with the sentences in which they lie for text summarisation by applying soft attention to sentences and words respectively and rescaling word weights. Instead of reweighting, Cheng et al. (2017) used character-level masks to guide the selection of useful parts for scene text recognition. None of these methods build clear alignment due to the use of soft attention.
163
164
165
186
168
Our work is closely related to Xu et al. (2015) and Ba et al. 20155). We generalise the former's mathematical formulation on hard attention by introducing flexible representation of objects of interest via a sharpener module. While the generalisation appears similar to the latter, use it to tackle the alignment issue in attention mechanisms rather than develop a new Seq2Seq model. Our modelling
170
172
173
also differs. Instead of seeking desired objects within the whole image, a divide-and andnou-end- scheme
5

---

Under review as a conference paper at ICLR 2022
is used to gradually narrow the search range for accurate localisation. Another difference in modelling 175 is that in [B et al. (2013) prediction will not happen until a series of localisation across predefined time steps whereas in our work that immediately follows localisation at each time.
178
177
4
EXPERIMENTS
178
We demonstrate the efficacy of the proposed method in two different scenarios of increasing difficulty: (i) synthetic handwritten digit recognition and (ii) real- world scene text recognition. Rather than strive for state-of-the- af-the-art results, the focus here is to highlight (i) the performance of Seq2Seq models can be boosted if attention mechanism really yields clear alignment, that is, paying attention to the target object, and (ii) the proposed sharp attention is effective approach to reaching the goal. Therefore, our vanilla system is built upon off-the-shelf modules and was trained without sophisticated parameter tuning schemes. Below we describe some common choices for all scenarios.
179
180
181
182
183
194
185
All images are converted to grey scale and resized to 100x32. A variant of VGG Shi et al. (017) is then used as the backbone to extract 24x1 I feature map from each resized image as well as create the set of regions, whose height is clipped to 32. The input sequence X obtained by splitting the feature map along its width, leading the dimensionality of each æ; to be the feature map depth (i.e. 512). Before feeding it into the attention mechanism, we follow Shi et al. (0177) to further process X capture long-range contextual information with 2-layer bidirectional LSTM, where each layer has forward LSTM and a backward LSTM, each having 256 hidden units. The depth of the attention mechanism is set to 256 and so is the number of the hidden units of the associated LSTM, which runs over some time to predict the output sequence Y. The pumber of time steps is set to the maximum transcription length in each scenario. As in Sutskever ct al. 22014), end-of-sequence token is used to indicate the finish of prediction. The STN in the sharpener is composed of a localisation network, a grid generator and a sampler. Given a region, the localisation network, achieved by the described in Liu et al. (2016), uses it to estimate an affine transformation, which is then used by the grid generator to place a set of control points on the region. By sampling the intensity value each control point in a way similar (Shi al. (2019), the sampler produces a patch of given size as the STN output. When multiple STNs are used, the output
Implementation
186
187
188
189
190
191
192
193
194
196
198
199
200
201
of the previous one used as the input to the next one. The output of the last STN is plugged into
202
the encoder in the sharpener to compute Z. The whole system was implemented using TensorFlow Abadi et al. 2016) and the code will be released the near future.
203
204
Three kinds of Seq2Seq models were trained from scratch in terms of the attention mechanism used. Specifically, the soft and hard models were learned via corresponding attention mechanisms respectively. Unlike the previous two baseline models, the sharp models were obtained by the sharpener with the context vector schemes described in Sect. A stochastic gradient descent method, ADADELTA (Zeiler 2012), was used to learn the model parameters until certain number of iterations in different scenarios. The learning rate was constant and set to 1.0 and the decay rate was 0.95. In addition, all model parameters were regularised by an Lg norm with weight decay of 4x 10-5. All experiments were done with a batch size of 192 (per GPU) on a workstation of NVIDIA GEFORCE RTX 2080 Ti GPUs The number of samples S was set to the batch size.
Training
205
206
207
209
211
212
213
Evaluation A prediction is correct if the predicted transcription matches ground truth. We reported the proportion of correct predictions on each testing dataset. As in (Shi et al. (2017), all transcriptions were converted to lower cases and had punctuations ruled out before evaluation if applicable.
214
215
218
4.1
HANDWRITTEN DIGIT RECOGNITION
217
We randomly chose images from the MNIST dataset (Lecun et al., 1998), resized them to 32x32 and concatenated them horizontally, leading to an image of 7 handwritten digits. For each 7 in (5, 7, 9, 11, 13), we created 20,000 images for training and 10,000 for testing by selecting from the
218
219
MNIST training and testing datasets respectively, leading to normal handwritten digit dataset. To introduce some distortion, we repeated the above procedure for a rotated dataset by randomly turning the selected images around y-axis within [-30-. 30%] before concatenation. Examples of the generated images are given in Fig. 3 We trained all models with the resulting datasets for 30,000 iterations by setting X, = 1.0 when applicable. For better localisation, we upsampled the set of regions along
221
223

---

Under review as a conference paper at ICLR 2022
Normal (7 digits)
Rotation (9 digits)
GROUND
1
6
2
4
6
8
2
S
4
6
5
9
6
TRUTH
1
5
9
HARD
6
8
2
FI
TI
III
61
1
1
5
S165
5
5
55
1
3
9
X
6
1
II
4
6
9
J
5
X
6
2
9
6
SHARP
1
2
3
5
9
SHARP+
6
2
6
9
2
3
6
5
9
6
REFERENCE
4
6
2 2
1 1
4 4
1
6
8
2 5 5
4 4
6
5 5
9 9
6
4
2
Figure 3: From top to bottom, we show examples of the created handwritten digit images and transcriptions (row 2), and patches for context vector computation well predicted digits (failures shown in red) at each time step (rows 3-5). Note that the patches are the receptive fields for the hard model and the output of the STN for the pooling based sharp models. The reference images are given below the patches in the last row.
Table 1: Recognition accuracy of all models for the handwritten digit datasets.
Normal
Rotation
Mean Acc.
Model
5
7
9
11
13
7
11
13
Baseline
SOFT
97.2
96.6
95.0
91.9
82.2
96.3
95,4
93.6
89.1
78.3
91.6
HARD
97.2
96.4
94.1
91.6
81.7
96.6
95.4
93.5
89.2
78.6
91.4
Sharp
POOLING
97.8
97.5
96.6
94.9
92.8
97.1
96.4
95.0
92.8
89.6
95.1
CHAIN
97.5
97.3
96.3
95.0
93.7
97.3
96.7
95.5
94.1
91.7
95.5
WEIGHTING 95.0
95.4
95.0
92.6
90.2
94.3
94.9
94.6
91.7
87.9
93.2
Pooling-based Searp+Reference
AFFINE 98.1
97.7
96.8
96.0
94.4
98.0
97.1
96.1
95.2
93.0
96.2
225
the width with a scale factor of 1.8 before plugging them into the sharpener, which was efficiently achieved by running region generation with 180x32 images. The patch output by the STN had a size of 24x32.
226
228
Table [ shows that sharp models significantly outperform the baseline, demonstrating the efficacy of clear alignment in attention mechanism. This can be easily seen from the pooling based model whose context vector purely results from the sharpener. Figure 3 also illustrates how attention can benefit from the sharpener. Take the rotation case for example. To predict digit '8' (the second column from left), hard attention chose a feature corresponding to a region filled with four digits. Due
230
232
to the distraction of irrelevant digits (e.g., '6', '2' and '5'), the feature failed to precisely represent
234 '8' thus giving wrong result. Although sharp attention selected the same region, it avoided most
235 of the distraction by deliberately locating '8' in that region, thus leading more accurate and specific representation as well as correct prediction. Besides, the resulting attention also has better interpretability since it is more focused. The above results testify that the performance of Seq2Seq models can be largely improved when attention mechanism is really focused on the object of interest.
236
237
238
239
To show that the sharpener allows for external supervision, set of x4 32 reference images for each digit (see Fig. 3 was created with the Roboto Bold typeface of font size 36 The supervision was achieved by introducing an L2 image similarity loss of weight of 1.0 to minimise the intensity difference between the patch and the reference. By showing what a desired digit would look like, the
241
242
*The font is available at petps::/fonne http//fooot..gooll.eee//o ctm. wwtmm//ppcciieee/ooottooooo We used Pygame for rendering.

---

Under review as a conference paper at ICLR 2022
Table 2: Recognition accuracy of all models for the scene text recognition datasets.
IIT
SVT
IC03
IC13
IC15
SP
Mean
Model
3000
647
867
857
1811
645
Acc.
Baseline
SOFT
77.9
78,8
87.8
86.1
61.4
62.5
75,8
HARD
77.6
78.8
89.2
86.0
59.9
63.9
75.9
Sharp+One STN
CHAIN
76.9
78.1
88.7
88.8
61.1
65.6
76.5
Sharp+Two STNS
POOLING
73.9
72.6
83.6
83.3
54.2
60.2
71.3
CHAIN
77.9
80.1
89.0
87.7
62.0
65.1
77.0
WEIGHTING
77.7
78.5
89.0
87.4
61.5
64.0
76.4
COTTAGE
10
FE
FITNESS
C
O
t
a
e
f
Atmosphere
2 0
Siorage
O
a m
t
0
5
h
e
r
S
a
9
(a) Success
(b) Failures
Figure 4: Examples of recognition results for scene text recognition. All real-world testing images are shown as is without rescaling and grey scale conversion. We leverage the chain based sharp model generate the patches localised by the sharpener and predicted tokens (failures highlighted in red) across time steps. The results from other sharp models look similar to what has been shown here.
243
pooling based sharpener gives better localisation where all digit shape is well preserved with little distraction, compared to its counterpart without such guidance (see Fig. 3 This, in turn, improves the representation of the context vector for alignment, leading to a boost of the model performance. Table [ shows that such improvement is the most remarkable when handling images more digits (c.g.. 13). This is not surprising because the receptive fields in such case have more digits filled and
245
246
247
248
thus make it difficult for the baseline models to decide where look. Even if compared with the best one amongst all sharp models trained without external supervision, i.e. the chain based model, the improvement is still noticeable, further demonstrating the importance accurate and target-specific representation in attention mechanism.
250
252
4.2
SCENE TEXT RECOGNITION
253
To further show the effectiveness the proposed sharp attention, we applied to a real-world visual
sequence learning task, scene text recognition. The training datasets include MJSynth (Jaderberg et al.
255
2014) and Texth'Text (Jaderberg et al. (2016). while the testing ones consist of HITSK- Words (Mishra al. 2012). Street View Text (Wang et al. 2001), ICDAR2003 (Lucas al.) 2003). ICDAR2013
256
257
Karatzas et al. 2013), ICDAR2015 (Kartzaas et al. 2015) and SVT Perspective (Phan et al. 2013),
258
hich are short for IIIT, SVT, IC03, ICI3, ICI5 and SP respectively. There 8.9 million images in the MJSynth dataset and 5.5 million in Synth Text by cropping text regions and ruling out non- alphanumeric characters. The number of images each testing dataset detailed Table 2 where the three datasets, IC03, IC13 and IC15, were prepared by following the protocol in Baek et al. (2019). The total number of testing images 827. Note that some of these datasets (e.g., IC15 and SP) are quite challenging due to various nuisance factors, such poor lighting and geometry change (Fig.
259
260
262
264
For fast evaluation of various model configurations, we randomly sampled 2 million labelled images from the MJSynth dataset. The sampling was done by following the distribution (i.e. histogram of the transcription length) of the original dataset. We set the maximum transcription length to to
286
enable large batch size (i.e. 192) for robust training. For sharp models, upsampled the regions along the width with a scale factor of 2.0 for better localisation. To explore the effects of using multiple STNs for localisation, first used two STNs train all sharp models, and then just used
288
8

---

Under review as a conference paper at ICLR 2022
the second one to train a chain based model for comparison. To whether localisation benefits from a coarse-to-fim search strategy, the first STN was designed to estimate a simple transformation (i.e. x-direction translation) but output a large patch (i.e. 64x32), while the second STN had the same configuration as used in Sect. 4.1 All models were trained for 400,000 iterations with the sampled dataset, and no reference images were used. To train the hard and sharp models we used X, 0.1.
271
273
274
Table 3: Mean entropy of different attention mechanisms for the scene text recognition datasets.
Model
IITT
SVT
IC03
IC13
IC15
SP
Mean
SOFT
1.06
1.01
1.06
1.00
1.15
1.12
1.07
HARD
0.63
0.61
0.67
0.63
0.63
0.62
0.63
SHARP
0.42
0.36
0.42
0.39
0.38
0.38
0.39
Table 4: Sharp attention vs. soft attention in a published scene text recognition work.
Model
IIT
SVT
IC03
IC13
IC15
SP
Mean Acc.
SOFr(Back et al. 2019)
84.3
83.8
93.1
91.9
70.8
71.9
82.6
SHARP
83.9
85.8
94.3
92.2
71.3
73.6
83.5
From [ we see that the chain based one again works the best amongst all sharp models of two STNs. It beats the baseline models remarkably on most of the datasets, whereas the other two either moderately outperform lag behind the baseline. It also beats its counterpart of single STN, even though the latter is slightly better than the baseline as well. The result from the winning sharp model further testifies our hypothesis on clear alignment in large-scale real-world datasets, which is also revealed by Fig. 4 Whenever prediction is successfully performed, there is sensible localisation of the target token. In fact, the sharpener attempts to highlight the token by placing in the patch centre (see Fig. 4 for examples surrounded by blue boxes). This is an encouraging result given that the sharpener was trained in a data-driven manner with merely sequential labelling (i.e. transcriptions). However, the localisation is by no means satisfactory since all patches have some sort of distractions, such as skewed target tokens and irrelevant parts of adjacent tokens. Both this observation and the benefit of multi-STN suggest a potential increase in accuracy if the sharpener is properly designed such that can produce good localisation as shown in Fig. 3 which is beyond the scope of this paper.
276
277
278
279
280
281
282
283
285
287
In Fig. we have shown sharp attention is more focused and yields better interpretability. This can be evaluated by H[a;), entropy of the categorical distribution defined by a (Shankar & Sarawagi 2019). It measures attention uncertainty, that is, the lower entropy, the better alignment and thus interpretability. Averaging H [a,] across all valid time steps leads to the entropy for an image. We reported the mean of such entropy on each testing dataset for various attention mechanisms in The results clearly show that sharp attention indeed boosts interpretability since entropy is a logarithmic metric. We only reported the entropy from the best sharp model in Table
289
281
282
293
3
294
295 Finally, we used the full datasets (i.c. MJSynth & SenthText) to train a sharp model with the same configuration to the best one in Table 2 To fairly compare the proposed attention with other attention mechanisms in existing scene text recogmition works, we reported the performance of the sharp model and a model (i.e. BGG+BILSTMAAtn based on soft attention trained with the same datasets by 296
296
297
Back et al. (2019) in Table The two models share the same backbone and RNN decoder. They only difter attention mechanism. Table further shows the superiority of our method.
300
5
CONCLUSION
302
We have described novel attention mechanism that is able to build clear alignment between relevant regions in the input image and the target output. This is achieved by generic sharpener module that computes accurate representation of the targets across time steps. Experimental results show that
303
305
vanilla implementation of our method can significantly beat soft and hard attention on both synthetic
and real-world datasets in terms of performance and interpretability. without bells and whistles such
307
as the auxiliary model in Ba et al. (2013) and sophisticated training schemes in Xu et al. (2013). We
308
plan to apply our method to more visual sequence learning tasks in the future.

---

Under review as a conference paper at ICLR 2022
6
REPRODUCIBILITY STATEMENT
310
The implementation details have been given in lines 185-202. Although most of the training parameters have been described in lines 207-212, some task-specific setting can be found in lines 223-221 and 266-274 respectively. The generation of synthetic handwritten digit dataset has been detailed in lines 217-222, while the preparation of real-world scene text recognition datasets has been elaborated in lines 254-261.
312
313
314
315
REFERENCES
316
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design
318
319
320
321
and Implementation, pp. 265-283, 2016.
André Araujo, Wade Norris, and Jack Sim. Computing receptive fields of convolutional neural networks. Distill, 2019. doi:
323
computinng
324
receptive-fields
Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. Multiple object recognition with visual attention. In Proceedings of the International Conference on Learning Representations, 2015.
326
327
Jeonghun Back, Geewook Kim, Junyeop Lee, Sungrae Park. Dongyoon Han. Sangdoo Yun, Scong Joon Oh, and Hwalsuk Lee. What is wrong with scene text recognition model com- parisons? Dataset and model analysis. In Proceedings of IEEE Conference on International Conference Computer Vision, pp. 4715-4723, 2019.
329
330
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), Proceedings of the International Conference Learning Representations, 2015.
332
Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, and Pascal Poupart. Variational attention for sequence-to--sequence models. In Proceedings of the International Conference on Computational Linguistics, pp. 1672-1682. 2018
335
337
Zhanzhan Cheng, Fan Bai, Yunlu Xu, Gang Zheng, Shiliang Pu, and Shuigeng Zhou. Focusing attention: Towards accurate text recognition in natural images. In Proceedings of IEEE Conference International Conference Computer Vision, pp. 5086-5094, 2017.
338
340
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander M. Rush. Latent alignment and variational attention. In Proceedings of Advances in Neural Information Processing Systems, pp-
341
342
9735-9747, 2018.
343
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1780- 1997.
344
345
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Synthetic data and artificial neural networks for natural scene text recognition. In Proceedings of NIPS Workshop on Deep Learning, 2014.
346
347
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett (eds.), Proceed- ings of Advances Neural Information Processing Systems, pp. 2017-2025, 2015.
349
350
Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Reading text in the wild with convolutional neural networks. International Journal of Computer Vision, 116:1-20, 2016.
352
353
Dimosthenis Karatzas, Faisal Shafait, Seichhi Uchida, Masakazu Iwamura, Lluis Gomez Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazán Almazån, and Lluis Pere de las Heras. ICDAR 2013 robust reading competition. In Proceedings of the International Conference on Document Analysis and Recognition, pp. 1484-1493, 2013.
355
356
10

---

Under review as a conference paper at ICLR 2022
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Emesst Valveny. ICDAR 2015 competition robust reading. In Proceedings of the International Conference on Document Analysis and Recognition, pp. 1156-1160, 2015.
358
360
382
Dicterich Lawson, Chung-Choee Chiu, George Tucker, Colin Raffel, Kevin Swersky, and Navdeep Jaitly. Learning hard alignments with variational inference. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, pp- 5799-5803, 2018.
363
364
365
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) (1::)72222222 1998.
387
Wei Liu, Chaofeng Chen, Kwan-Yee K. Wong, Zhizhong Su, and Junyu Han. STAR-Net: A spatial attention residue network for scene text recognition. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of British Machine Vision Conference, pp. 43.1 133.13,
388
389
370
2016.
S.M. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong, and R. Young. ICDAR 2003 robust read- ing competitions. In Proceedings of the International Conference on Document Analysis and Recognition, pp. 682-687, 2003.
372
374
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention- based neural machine translation. In Proceedings of the International Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, 2015.
375
378
377
Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. What to talk about and how? Selective generation using LSTMs with coarse-to-fine alignment. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 720-730, 2016.
378
379
381
Anand Mishra, Karteek Alahari, and C. V. Jawahar. Scene text recognition using higher order language priors. In Richard Bowden, John P. Collomosse, and Krystian Mikolajezyk (eds.), Proceedings of British Machine Vision Conference, pp. 1-11, 2012.
382
383
384
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Proceedings of Advances in Neural Information Processing Systems, pp. 2212-2212, 2014.
386
Ramesh Nallapati. Bowen Zhou, Cicero dos Santos, Çağlar Gülçehre, and Bing Xiang. Abstractive text summarization using sequence-to-sequeeee RNNs and beyond. In Proceedings of the SIGNLL Conference on Computational Natural Language Learning, pp. 280-290, 2016.
388
389
390
Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan. Recognizing text with perspective distortion in natural scenes. In Proceedings of IEEE Conference on International Conference on Computer Vision, pp. 369-576, 2013.
391
382
393
Shiv Shankar and Sunita Sarawagi. Posterior attention models for sequence to sequence learning. In Proceedings of the International Conference on Learning Representations, 2019.
395
Shiv Shankar, Siddhant Garg, and Sunita Sarawagi. Surprisingly easy hard-attention for sequence to sequence learning. In Proceedings of the International Conference on Empirical Methods in Natural Language Processing, pp. 640-645, 2018
396
387
398
Baoguang Shi, Xiang Bai, and Cong Yao. An aad-to-end trainable neural network for image-based sequence recognition and its application scene text recognition. IEEE Trunsactions Patlern Analysis and Machine Intelligence, 39(1 (0(()):228-2340) 23044, 2017.
400
Baoguang Shi. Mingkun Yang, Xinggang Wang. Pengyuan Lyu, Cong Yao, and Xiang Bai. ASTER: An attentional scene text recognizer with flexible rectification. IEEE Transactions on Pattern Analysis and Machine Intelligence, (19):2035-2045 2019.
402

---

Under review as a conference paper at ICLR 2022
Iya Sutskever, Oriol Vinyals, and Quoc V. Le. Sequence sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N.D. Lawrence, and K.Q. Weinberger (eds.), Proceedings of Advances in Neural Information Processing Systems, pp. 3104-3112, 2014.
405
Kai Wang, Boris Babenko, and Serge Belongie. End-to-end scene text recognition. In Proceedings of IEEE Conference on International Conference on Computer Vision, pp. 1457-1464, 2011.
408
409
Ronald J. Williams. Simple statistical gradient-followiig algorithms for connectionist reinforcement
410
learning. Machine Learning. 8(3 4):229 226,, 1992.
411
Shijie Wu, Pamela Shapiro, and Ryan Cotterell. Hard non-monotonic attention for character-level transduction. In Proceedings of the International Conference Empirical Methods in Natural
413
Language Processing, pp. 4425 4438, 2018.
414
Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov. Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning, pp.
415
418
417
2048 2048-2057, 2015.
418
Matthew D. Zeiler. Adadelta: adaptive learning rate method, 2012.
12


Reviews:
[{"review_id": "ixzi-EGSzM", "reviewer": "Reviewer_1RPC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of this paper:\n(1). A new sharp attention mechanism is proposed, by introducing a latent vector into traditional attention framework, so that let the target output token only depends on one element of input sequence. \n(2). The general mathematical design and analysis of the sharp attention idea seems reasonable and feasible.\n\nWeaknesses: \n(1) The proposed method can be somehow regarded as a variant of hard attention, since the authors assume that the output y_t only depends on a_t at time t (line 55-56). I don't think to pursue the alignment such that each target token only depends on one element of the input X is a better way comparing with traditional soft attention mechanism. The reason is two-fold. At one hand, such sharp attention may require that each of local region (broken down by some method, such as sliding window) should large enough to cover objects of interest, however, in practice, we can not always guarantee this since sometime we need to use small region in order to make better attention on small instance in a sequence. On the other hand, contextual information is crucial for some sequence recognition problem like handwritten or scene text recognition. Sharp attention may lose the ability to make use of contextual information which is very helpful during the recognition process. \n\n(2) The implementation of the Sharpener in this paper using STN plus CNN. Actually, STN was designed to crop and transform (eg. under affine transformation) an input image so that the output of STN becomes more regular, although its localization subnet can learn a set of fiducial points that seem localize the real instance of interest. In the field of scene text recognition, STN  has been successfully used as a rectification by many previous methods, together with an attention Seq2Seq recognizer, such as \"Shi, et al., Robust Scene Text Recognition with Automatic Rectification, CVPR 2016\". Therefore, it is doubtful that the performance gain achieved by the proposed Sharpener actually comes from the STN rectification rather than the so called sharpen attention. \n\n(3) The experiments are not solid and convincing enough: \n   - The first experiment using synthetic handwritten digital string is quite simple and toy. To show the proposed method can be really useful, I recommend the authors should use more challenging dataset, such as IAM dataset which is widely used in the community of OCR. \n   - I don’t understand why in section 4.2, the authors just used a sampled 2 million samples from MJSynth dataset to train the different models for comparison. Experiments in this paper are not heavy so I see no reason to use just part of the dataset to train the models. To use either of both of MJSynth and SynthText datasets like existing methods will give more accurate evidence to show if the proposed SHARP model works better than the counterpart. Otherwise, I am wondering if the proposed method only performs better under small training sample setting. \n   - More strict comparison with related methods are expected under the same settings, such as:  \n        * B. Shi, et al., Robust Scene Text Recognition with Automatic Rectification, CVPR 2016; \n        * T. Wang, et al., Decoupled attention network for text recognition, AAAI 2020；\n        * Q. Wang, et al., ReELFA: A scene text recognizer with encoded location and focused attention, ICDAR 2019\n\n(4) It is unclear that the proposed Sharpener module is a general module that can be applied in different existing SOTA Seq2Seq methods, such as ATSTER in the field of scene text recognition, and so on. \n\n(5) It is better to illustrate the attention map generated by the proposed method when the recognition results are not correct, so that we can see that if the proposed method can solve the problem of attention mis-alignment issue in such circumstance. \n", "summary_of_the_paper": "This paper presents a new module technology for seq2seq attention model, addressing the mis-alignment issue which widely exist in most current attention-based methods. A new Sharpener module is proposed to help construct clear (direct) alignment in attention mechanism. Mathematical design and analysis of this idea is given in detail, and the authors also give an implementation of Sharpener module, which consist of a localizer (STN) and a n encoder (CNN), under the design concept that the localizer locates the target in the region and the encoder extracts better feature for alignment. Equipped with the proposed Sharpener module, a new Seq2Seq model is proposed for text recognition. Experiments on both synthetic handwritten digital string and real-world scene text datasets show the superiority of the proposed method, comparing with its counterpart, i.e., traditional hard or soft attention-based methods.", "summary_of_the_review": "In general, this problem addressed in this paper is not a new one. There are many existing methods have been proposed to address the attention mis-alignment (or attention drift) issue. Although the motivation of this paper and the mathematical design of the sharp attention mechanism seem ok and reasonable, the proposed new Sharpener module does not bring me too much new insight and inspiration.  I do not see clear advantage of the proposed method against existing ones. Furthermore, I am not satisfied with the experimental part of this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "gPAe0xciGc", "reviewer": "Reviewer_mESb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThe attention machanism with a sharpener module is better than the baseline attention mechanisms.\n\nWeaknesses:\nThe experimental results are not convincing enough. The results on the scene text recognition benchmark are much lower than the current mainstream results [1,2,3]. There are more than 10 percent gaps on these datasets. Thus, the experimental results are not sufficient to prove the effectiveness of the sharpener module since the baseline results are not convincing.\n\n\n[1] Shi, Baoguang, et al. \"Aster: An attentional scene text recognizer with flexible rectification.\" IEEE transactions on pattern analysis and machine intelligence 41.9 (2018): 2035-2048.\n[2] Qiao, Zhi, et al. \"Seed: Semantics enhanced encoder-decoder framework for scene text recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n[3] Yan, Ruijie, et al. \"Primitive Representation Learning for Scene Text Recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "summary_of_the_paper": "This paper proposed a sharpener module for sequence-to-sequence learning. The motivation is to address the construction of clear alignment in attention mechanism. The authors conducted experiments on handwritting text recognition and scene text recognition text datasets. The experimental results show that the attention mechanism with a sharpener module is better than the soft attention and hard attention.", "summary_of_the_review": "The major concern of this paper is that the baseline results are too bad, making the experimental result not convincing. I suggest the authors choose a SOTA method as baselines.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "agTvgC_j44Y", "reviewer": "Reviewer_5Hay", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\nThe alignment is one issue in the normal attention mechanism, especially for the long sequence. In this paper, the authors propose a sharpener module, which is explicitly designed to seek clear alignment. The motivation is strong, and the idea performs well on the two handwritten recognition tasks. \n\nWeakness\n\n(1) The paper organization maybe is not friendly to read. For example, as the sharpener module is the main part of the paper, the description on this part is not concentrated, Some are in the Section 2.3, and some are in the section 2.5, and some are in the section 4 (Implementation). I suggest the authors can merge all details to create one new section to describe sharpener module. If the paper size is limited, maybe the section 2.4 can be shorten because I think this part is very similar with the previous work. \n\n(2) Some details are not clear. For example, it is not clear to know how the three ways to compute context vector c given Z. Also how to get the r_i in the Fig2 (choosing the region regarding to the largest value alpha_{t,i}?). \n\n(3)The experiments maybe are not sufficient. I am not sure what kind of SOFT attention is used in he baseline. Maybe some kind of improvement on SOFT attention can be compared in the experiments, for example some works introduced in Related work.  \n\n(4)The experiments are not clear. Because some datasets in the scene text recognition are curved images, some works integrated the STN module in the recognition. So if the baseline system is not integrated with STN, while the proposed module uses STN, then this comparison is not fair. Maybe you should use the baseline system integrated with STN. \n\n(5) If the authors can do the experiments on the real handwriting recognition, neither English or Chinese text, this will be much better, because the digits are not connected in synthetic handwritten digit string , and the sequence length is not very long in the scene text.", "summary_of_the_paper": "In this paper, the authors propose a sharpener module to seek clear alignment in the attention mechanism. The sharpener module contains a localiser and an encoder, where the localiser is used to adjust the interested region to align the target better, and the encoder is used to extracted features from the adjusted region, which is used to get the final context vector. \n\nThe authors verify the proposed method on two datasets, including synthetic handwritten digit recognition, and one real-world scene text recognition.  The experimental results show the effectiveness of the proposed method. In addition, the authors show the detailed analysis on the results, which verify the proposed idea further. \n\nIn summary, the idea is well motivated, and the experimental results show its effectiveness.", "summary_of_the_review": "This paper is well motivated to seek clear alignment in the attention mechanism. This idea can be understood overall, although some details can be added. For the experiments, it seems not sufficient, maybe more comparison experiments can be added to verify the effectiveness of the proposed method.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SBIdhYp7c16", "reviewer": "Reviewer_aDJW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-written and easy to follow. The motivation is clear. The key innovation is the sharpener module for better feature alignment, which theoretically sounds according to the equations in the manuscript. Experimental results show that the proposed method consistently outperforms the original attention mechanism over different datasets on scene text recognition task.\n\nMy major concerns are as follows:\n1)\tThis method is claimed to be a novel attention mechanism for sequence-to-sequence learning. As such, more experiments on other tasks are expected (except for the scene text recognition task) for a better demonstration of the proposed method. Otherwise, I’m thinking it’s better to claim that this method is designed for scene text recognition task.\n2)\tFor experiments in Fig. 3 and Table 1, what’s the motivation of conducting experiments with reference patches? Why not conduct such experiments for experiments in Tables 2 and 4?\n3)\tAs the experiments are mainly for scene text recognition task, I’m wondering if the proposed method could help on scene text recognition task. Currently, the performances shown in Table 4 are very far from state-of-the-art.\n", "summary_of_the_paper": "This paper presents a sharp attention module for sequence-to-sequence learning. It introduces a sharpener to compute context vectors for clearer alignment of features. Extensive experiments on scene text recognition task demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "In summary, this paper is well-written and its key contribution technically sounds. Extensive results on the scene text recognition task show the effectiveness of the proposed method as compared with the traditional attention mechanism. However, the experiments are not thorough and enough to support the authors’ claim well. Additionally, some details of the experiments are not very clear to me. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a sharp attention mechanism in the context of image to sequence modeling. It seeks to build a “clear” alignment from the attention in order to improve the performance of the task. I don’t think there is a general consensus in the research community that the “clear” or “hard” attention performs better than the vanilla “soft” attention. Therefore, experiments become the key in justifying such motivation (and the model). However, as all the reviewers point out, the experiments in this paper are not satisfying. The numbers are far from the current mainstream results (Reviewer mESb). The experiments are done on relatively small datasets/tasks (Reviewer 1PRC) and the comparisons aren’t strictly speaking fair (Reviewer 5Hay). I think this alone is enough reason for the rejection of this paper. 

Additionally, on the algorithmic side, the novelty of this mechanism is not that high, given the existence of work such as the hard attention (Xu et al, 2015) and variational attention (Deng et al, 2018). It is also an unanswered question how such a mechanism can be introduced into the modern architectures that use attention (e.g. the multi-head attention). The authors did not respond to the questions of the reviewers.
Decision: Reject