SlowGuess/ACL_2020_OCR huggingface dataset:
Conference on Empirical Methods in Natural Language Processing, pages 1364-1373, Copenhagen, Denmark. Association for Computational Linguistics.
Christian Stab and Iryna Gurevych. 2014. Annotating argument components and relations in persuasive essays. In COLING 2014, 25th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland, pages 1501-1510.
Christian Stab and Iryna Gurevych. 2017. Parsing argumentation structures in persuasive essays. Computational Linguistics, 43(3):619-659.
Xinyu Wang, Jingxian Huang, and Kewei Tu. 2019. Second-order semantic dependency parsing with end-to-end neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4609-4618, Florence, Italy. Association for Computational Linguistics.
Yue Zhang, Wei Jiang, Qingrong Xia, Junjie Cao, Rui Wang, Zhenghua Li, and Min Zhang. 2019. SUDA-Alibaba at MRP 2019: Graph-based models with BERT. In Proceedings of the Shared Task on Cross-Framework Meaning Representation Parsing at the 2019 Conference on Natural Language Learning, pages 149-157, Hong Kong. Association for Computational Linguistics.
A.1 Input Representation
Following the work of Kuribayashi et al. (2019) and Potash et al. (2017), we propose incorporating multiple types of token representation to provide rich input features. Specifically, the proposed system combines surface, part-of-speech (POS) tags, GloVe (Pennington et al., 2014) embedding, and ELMo (Peters et al., 2018) as input features for each token. The following descriptions explain how we acquire each input representation:
Surface Tokens are parsed by SpaCy (https:// spacy.io/). Surfaces that appear less than four times are replaced by special UNK tokens.
POS tags We employ POS tags obtained by SpaCy.
GloVe We employ 300-dimensional GloVe vectors (obtained from http://nlp.stanford.edu/data/glove.840B.300d.zip).
ELMo We employ the pretrained ELMo (obtained from https://s3-us-west-2.amazon.com/allennlp/models/elmo/2x4096_512_2048cnn_2xhighway/elmo_2x4096_512_2048cnn_2xhighwayweights.hdf5 and elmo_2x4096_512_2048cnn_2xhighway_
hyperparameter	value or search space
GloVe dimention	300
GloVe embedding linear	100
POS embedding linear	100
ELMo type	2x4096, 512_2048cnn_2xhighway
input dropout	0.25, 0.33, 0.45
BILSTM dimension	200, 300, 400
BILSTM stack	1
BILSTMτ dimension	200, 300, 400
BILSTMτ stack	2, 3
recurrent dropout of all BiLSTMs	0.25, 0.33, 0.45
output dropout of all BiLSTMs	0.25, 0.33, 0.45
dimention of all MLPs	600, 700
dropout of all MLPs	0.25, 0.33, 0.45
activation of all MLPs	ReLU
(λedge, λlabel, λtype)	(0.6, 0.2, 0.2), (0.4, 0.3, 0.3), (0.333, 0.333, 0.333)
learning rate	0.0012, 0.0011, 0.001, 0.0009, 0.0008
Adam β1	0.9
Adam β2	0.999
epoch	100
mini-batch size	16
Table 2: List of hyperparameters. Multiple values indicates that the hyperparameter was tuned within those values. Underlines show the selected hyperparameter by the Optuna framework.
options.json). Following Peters et al. (2018), we mix different layers of ELMo for each token:
$$
\begin{array}{l} \tilde {s _ {k}} = \frac {\exp (s _ {k})}{\sum_ {k ^ {\prime}} \exp (s _ {k ^ {\prime}})}, \\ \mathbf {W} _ {\text {S T A R T} (i): \text {E N D} (i)} ^ {\text {E L M o}} \\ = \sum_ {k} \tilde {s _ {k}} \mathrm {E L M o} _ {\text {S T A R T} (i): \text {E N D} (i)} ^ {k}, \\ \end{array}
$$
where  $\mathrm{ELMo}_{\mathrm{START}(i):\mathrm{END}(i)}^k (0 < k\leq N^{\mathrm{ELMo}})$  is the hidden state of the  $k$  -th layer of the ELMo obtained by START(i) to END(i) tokens,  $\mathrm{ELMo}_{\mathrm{START}(i):\mathrm{END}(i)}^{0}$  are the features from character-level CNN in ELMo, and  $s_k$  are trainable parameters. The ELMo parameters are fixed by truncating backpropagation.
The surface and POS tag of a token are each embedded into a vector. A multi-layered perceptron (MLP) is applied to each surface and POS. All features are then concatenated to form input token representation:
$$
\mathbf {w} _ {t} = \mathbf {w} _ {t} ^ {\text {s u r f a c e}} \oplus \mathbf {w} _ {t} ^ {\text {P O S}} \oplus \mathbf {w} _ {t} ^ {\text {G l o V e}},
$$
Optionally, we can concatenate ELMo:
$$
\mathbf {w} _ {t} = \mathbf {w} _ {t} ^ {\text {s u r f a c e}} \oplus \mathbf {w} _ {t} ^ {\text {P O S}} \oplus \mathbf {w} _ {t} ^ {\text {G l o V e}} \oplus \mathbf {w} _ {t} ^ {\text {E L M o}}.
$$
A.2 Non-TSP Model
For non-TSP model in experiments, we provide a shared representation for both type and edge:
$$
\begin{array}{l} \mathbf {h} _ {\text {t y p e \& e d g e}, i} ^ {\text {s p a n}} = \mathbf {h} _ {\text {t y p e}, i} ^ {\text {s p a n}} = \mathbf {h} _ {\text {e d g e}, i} ^ {\text {s p a n}}, \\ = \mathbf {h} _ {\text {E N D} (i)} \oplus \mathbf {h} _ {\text {t y p e} \& \text {e d g e}, i} ^ {\text {s p a n - a t t}} \oplus \phi (i). \\ \end{array}
$$
$$
\mathbf {s} _ {\text {t y p e \& e d g e}, i} = \mathrm {B I L S T M} _ {\text {t y p e \& e d g e}} (\mathbf {h} _ {\text {t y p e \& e d g e}, i} ^ {\text {s p a n}}).
$$
According to the change above, the non-TSP also requires us to modify the pre-biaffine operations:
$$
\mathbf {e} _ {i} ^ {\left(\text {s r c}\right)} = \operatorname {M L P} _ {\text {e d g e}} ^ {\left(\text {s r c}\right)} \left(\mathbf {s} _ {\text {t y p e} \& \text {e d g e}, i}\right),
$$
$$
\mathbf {e} _ {j} ^ {\left(\operatorname {t r g}\right)} = \operatorname {M L P} _ {\text {e d g e}} ^ {\left(\operatorname {t r g}\right)} \left(\mathbf {s} _ {\text {t y p e} \& \text {e d g e}, j}\right),
$$
$$
\ell_ {i} ^ {\mathrm {(s r c)}} = \mathrm {M L P} _ {\mathrm {l a b e l}} ^ {\mathrm {(s r c)}} (\mathbf {s} _ {\mathrm {t y p e} \& \mathrm {e d g e}, i}),
$$
$$
\ell_ {j} ^ {\left(\operatorname {t r g}\right)} = \operatorname {M L P} _ {\text {l a b e l}} ^ {\left(\operatorname {t r g}\right)} \left(\mathbf {s} _ {\text {t y p e} \& \text {e d g e}, j}\right),
$$
and the proposition type classifier:
$$
\mathrm {t y p e} _ {i} = \operatorname {s o f t m a x} \left(\mathrm {M L P} _ {\text {t y p e}} \left(\mathbf {s} _ {\text {t y p e} \& \text {e d g e}, i}\right)\right).
$$
We tuned the hyperparameters using a subset considering our preliminary experiments. See Table 2 for hyperparameter search space and list of hyperparameters chosen by the Optuna framework (Akiba et al., 2019). We tried 20 hyperparameter sets. As can be seen from the table, the high dropout rate is effective. We estimate this is because the system can prevent an overfitting. We also found stacking BiLSTMs in TSP higher can improve performance, implying the semantics can be captured in upper layers.
A.4 Single-task Setup
For the single-task setup (non-multi-task), we provide each task-specific learning: type, edge, and edge label. Each model was optimized using its objective using the same hyperparameters.