SlowGuess/ACL_2020_OCR huggingface dataset:
We evaluated three types of span representations. Here we give details on how each representation is computed. For sum pooling, we use the implementation of area attention (Li et al., 2019) that allows constant time computation of the representation of each span by using summed area tables. The TensorFlow implementation of the representation
8208
Figure 6: The web interface for annotators to label action phrase spans in an AndroidHOWTo instruction.
is available on  $\mathrm{Gibut}^6$
Algorithm 1: Compute the Start-End Concat span representation for all spans in parallel.
Input: A tensor  $H$  in shape of  $[L, D]$  that represents a sequence of vector with length  $L$  and depth  $D$ .
Output: representation of each span,  $U$ .
1 Hyperparameter: max span width  $M$
2 Init start & end tensor:  $S \gets H$ ,  $E \gets H$ ;
3 for  $m = 1,\dots ,M - 1$  do
4  $S^{\prime}\gets H[: - m,:]$
5  $E^{\prime}\gets H[m,:;]$
6  $S\gets [S S^{\prime}]$  ,concat on the 1st dim;
7  $E\gets [E E^{\prime}]$  ,concat on the 1st dim;
$\mathbf{s}U\gets [S E],$  , concat on the last dim;
9 return  $U$
Algorithm 1 gives the recipe for Start-End Concat (Lee et al., 2016) using Tensor operations. The advanced form (Lee et al., 2017) takes two other features: the weighted sum over all the token embedding vectors within each span and a span length feature. The span length feature is trivial to compute in a constant time. However, computing the weighted sum of each span can be time consuming if not carefully designed. We decompose the computation as a set of summation-based operations (see Algorithm 2 and 3) so as to use summed area tables (Szeliski, 2010), which was been used in Li et al. (2019) for constant time computation of span representations. These pseudocode definitions are designed based on Tensor operations, which are highly optimized and fast.
$^{6}$ https : / / github . com / tensorflow / tensor2tensor/blob/master/tensor2tensor/ layers/area_attention.py
Algorithm 2: Compute the weighted embedding sum of each span in parallel, using ComputeSpanVectorSum defined in Algorithm 3.
Input: Tensors  $H$  and  $E$  are the hidden and embedding vectors of a sequence of tokens respectively, in shape of  $[L,D]$  with length  $L$  and depth  $D$ .
Output: weighted embedding sum,  $\hat{X}$
1 Hyperparameter: max span length  $M$
2 Compute token weights  $A$ :  $A \gets \exp(\phi(H, \theta)W)$  where  $\phi(\cdot)$  is a multi-layer perceptron with trainable parameters  $\theta$ , followed by a linear transformation  $W$ .  $A \in R^{L \times 1}$ ;
3  $E^{\prime}\gets E\otimes A$  where  $\otimes$  is element-wise multiplication. The last dim of  $A$  is broadcast;
4  $\hat{E}\gets$  ComputeSpanVectorSum  $(E^{\prime})$
5  $\hat{A}\gets$  ComputeSpanVectorSum(A);
6  $\hat{X}\gets \hat{E}\oslash \hat{A}$  where  $\oslash$  is element-wise division. The last dim of  $\hat{A}$  is broadcast;
7 return  $\hat{X}$
Algorithm 3: ComputeSpanVectorSum.
Input: A tensor  $G$  in shape of  $[L, D]$ .
Output: Sum of vectors of each span,  $U$ .
1 Hyperparameter: max span length  $M$
2 Compute integral image  $I$  by cumulative sum along the first dimension over  $G$ ;
3  $I\gets [0I]$  , padding zero to the left;
4 for  $m = 0,\dots ,M - 1$  do
5  $I_{1}\gets I[m + 1:,]:$
6  $I_{2}\gets I[: - m - 1,:]$
7  $\bar{I}\gets I_1 - I_2$
8  $U\gets [U\bar{I} ],$  concat on the first dim;
9 return  $U$
C Details for Distance GCN
Given the structural distance between two objects, based on the view hierarchy tree, we compute the strength of how these objects should affect each other by applying a Gaussian kernel to the distance, as shown the following (Equation 13).
$$
\operatorname {A d j a c e n c y} \left(o _ {i}, o _ {j}\right) = \frac {1}{\sqrt {2 \pi \sigma^ {2}}} \exp \left(- \frac {d \left(o _ {i} , o _ {j}\right) ^ {2}}{2 \sigma^ {2}}\right) \tag {13}
$$
where  $d(o_i, o_j)$  is the distance between object  $o_i$  and  $o_j$ , and  $\sigma$  is a constant. With this definition of soft adjacency, the rest of the computation follows the typical GCN (Niepert et al., 2016).
D Hyperparameters & Training
We tuned all the models on a number of hyperparameters, including the token embedding depth, the hidden size and the number of hidden layers, the learning rate and schedule, and the dropout ratios. We ended up using 128 for the embedding and hidden size for all the models. Adding more dimensions does not seem to improve accuracy and slows down training.
For the phrase tuple extraction task, we used 6 hidden layers for Transformer encoder and decoder, with 8-head self and encoder-decoder attention, for all the model configurations. We used  $10\%$  dropout ratio for attention, layer preprocessing and relu dropout in Transformer. We followed the learning rate schedule detailed previously (Vaswani et al., 2017), with an increasing learning rate to 0.001 for the first 8K steps followed by an exponential decay. All the models were trained for 1 million steps with a batch size of 128 on a single Tesla V100 GPU, which took 28 to 30 hours.
For the grounding task, Filter-1 GCN and Distance GCN used 6 hidden layers with ReLU for nonlinear activation and  $10\%$  dropout ratio at each layer. Both GCN models use a smaller peak learning rate of 0.0003. The Transformer screen encoder also uses 6 hidden layers but uses a much larger dropout ratio: ReLU dropout of  $30\%$ , attention dropout of  $40\%$ , and layer preprocessing dropout of  $20\%$ , with a peak learning rate of 0.001. All the grounding models were trained for 250K steps on the same hardware.
8210