SlowGuess/ACL_2020_OCR huggingface dataset:
Our QUASE is based on the re-implementation of BERT with pytorch (Wolf et al., 2019). Although we might change a bit to fit the memory of GPU sometimes, the common hyper parameters for further pre-training s-QUASE and p-QUASE are as follows:
Further pre-training p-QUASE. For sentence-level QA datasets (QAMR, Large QA-SRL, and QA-RE), we further pre-train BERT for 4 epochs with a learning rate of  $5e - 5$ , a batch size of 32, a maximum sequence length of 128. For paragraph-level QA datasets (SQuAD, TrivaQA, and NewsQA), we further pre-train BERT for 4 epochs with a learning rate of  $5e - 5$ , a batch size of 16, a maximum sequence length of 384.
Further pre-training s-QUASE. For sentence-level QA datasets (QAMR, Large QA-SRL, and QA-RE), we further pre-train s-QUASE for 64 epochs with a learning rate of  $1e - 4$ , a batch size of 72, a maximum sentence length of 128 and a maximum question length of 24. For paragraph-level QA datasets (SQuAD, TrivaQA, and NewsQA), we further pre-train s-QUASE for 32 epochs with a learning rate of  $1e - 4$ , a batch size of 8, a maximum sentence of 384, and a maximum question length of 64. We need to note that s-QUASE contains more architectures than BERT, so the hyper parameters for BERT fine-tuning are not good for s-QUASE further pre-training
A.2 Error Analysis of QUASE
The F1 scores of s-QUASE $_{QAMR}$  and p-QUASE $_{QAMR}$  on the development set are 76.20 and 90.35. In general, the results of s-QUASE are similar to BiDAF (Seo et al., 2017) but are significantly worse than the p-QUASE on QAMR. We conduct thorough error analysis including: sentence length, answer length, question length, question words and the PoS tag of the answer. We find that s-QUASE is not good at dealing with long sentences compared to p-QUASE. The analy
sis of model performance with regard to sentence length is shown in Figure 3(a). The average number of QA pairs is much larger when the sentence is longer as shown in Figure 3(b). The distribution of training set and development set is quite different, which makes the situation more complicated. We further compare s-QUASELarge QA-SRL and p-QUASELarge QA-SRL on Large QA-SRL whose distribution of training and development sets are the same. From the results, s-QUASE is still not as good as p-QUASE on long sentences. We think that the failure of s-QUASE in long sentences is mainly because there are more relations to encode, while p-QUASE only needs to encode information based on specific questions. We believe that there is a trade-off between the quality and the quantity of sentence information that a model can encode in practice, although  $h(S)$  also include the information in  $h_A(S)$  in a perfect world.
A.3 Ablation Analysis for s-QUASE
s-QUASE consists of three basic components: a sentence encoder for the sentence representation, a question encoder for the question representation, an interaction layer between the sentence component and the question component. We carefully designed five variants of s-QUASE with increasing complexity and performance: (I) Basic model: a fixed BERT and one-layer bidirectional transformer for sentence modeling, a fixed BERT and one-layer bidirectional transformer for question modeling, and a two-layer multi-layer perceptron (MLP) for the interaction layer; (II) a fine-tuned BERT; (III) the same as model II, with a bi-directional attention flow added to the question component; (IV) the same as model III, with the interaction layer changed from a two-layer MLP to a bidirectional transformer; (V) the same as model IV, with the sentence modeling layer and question modeling layer changed from a single-layer bi-directional transformer to a two-layer one, and beam search is used in the inference stage. Table 6 shows the results of our models further pre-trained on the development set of the QAMR dataset.
In this section, we show the details of experimental setup in Section 3. Because the corresponding settings are too many, we show some common settings here and more details are in our code. We first show the details of simple models in Section B.1,
8754
(a) Error analysis for QUASE.
(b) Number of QA pairs.
Figure 3: Error analysis of QUASE on the sentence length. We compare the performance of s-QUASE and p-QUASE on examples with different sentence lengths in the development set. The average number of QA pairs corresponding to the sentence length in the train and development sets is also shown.
Models	Model I	Model II	Model III	Model IV	Model V
Average EM	34.97	41.64	55.68	64.18	66.77
Average F1	40.05	45.49	62.98	72.96	76.20
Table 6: The results of five variants of s-QUASE $_{QAMR}$  on the development set of QAMR. We use the average exact match (EM) and average F1 as our evaluation metrics.
Embeddings	ELMo	Flair
Tasks	SRL	Coref	NER
Split	small	full	small	full	small	full
Baselines	78.32	83.87	60.72	66.89	89.86	92.37
s-QUASEQAMR	79.40	84.14	61.54	66.58	90.18	92.54
Table 7: Comparison between s-QUASE $_{QAMR}$  and other STOA embeddings. We use the same experimental settings as Section 3.4 for the three single-sentence tasks, SRL, Coref and NER. We use ELMo embeddings for SRL and Coref, and Flair embeddings for NER as our baselines.
and then show some common experimental settings of QUASE in Section B.2. Finally, we compare s-QUASE with other SOTA embeddings (ELMo and Flair) in Section B.3
B.1 Simple Models
When QUASE is used in the feature-based approach, we need use models for the tasks. For simplicity, we sometimes choose to use some simple models rather than strong baselines in Section 3 in our analysis. Following standard practice, we use a simple BiLSTM model with the input of word embeddings and binary features of predicates for SRL, a simple biaffine model based on BiLSTM for SDP, a simple BiLSTM mode for NER, a simple
CNN baseline with the input of word embeddings and position features for RE, and a simple BiLSTM model for TE.
B.2 Experimental Settings
We use the re-implementation of SRL, NER and Coref from AllenNLP (Gardner et al., 2017) for strong baselines, and we implement the strong baselines of SDP and RE ourselves. As for MRC and TE, we use the re-implementation of BERT with pytorch (Wolf et al., 2019). As for simple models, we implement them by ourselves. As for the hyper parameters for strong baselines of single-sentence tasks, we use the same hyper parameters in the related papers (shown in Section 3). As for the hyper parameters for simple models, we tune them ourselves to find some reasonable hyper parameters. The hyper parameters of MRC and TE for p-QUASE are based on (Wolf et al., 2019).
B.3 Comparison with Other Embeddings
To show whether s-QUASE can also provide extra features than other STOA embeddings<sup>11</sup>, such
11The reported STOA models for SRL and Coref is based on ELMo embeddings and the reported STOA model for NER is based on Flair embeddings.
8755
Sentence	Ann.	Question	Answers
(1) Mr. Agnew was vice president of the U.S. from 1969 until he resigned in 1973 .	INF	What did someone resign from?	vice president of the U.S.
(2) This year, Mr. Wathen says the firm will be able to service debt and still turn a modest profit .	INF	When will something be ser- viced?	this year
(3) Mahlunga has said he did nothing wrong and Judge Horn said he "failed to express genuine remorse".	INF	Who doubted his remorse was genuine?	Judge Horn
(4) Volunteers are presently renovating the former post office in the town of Edwards, Mississippi, United States for the doctor to have an office.	IMP	What country are the volun- teers renovating in?	United States
Table 8: Some examples of question-answer pairs in QA-SRL and QAMR datasets. The first two examples are from QA-SRL dataset and predicates are bolded. The last two examples are from QAMR dataset. We show two phenomena that are not modeled by traditional symbolic representations of predicate-argument structure (e.g SRL and AMR), inferred relations (INF) and implicit arguments (IMP).
Span	IOU ≥ 0.5	Token
Models	Precision	Recall	F1	Precision	Recall	F1	Precision	Recall	F1
Rules + EM	24.31	22.78	23.52	34.34	32.27	33.27	50.46	28.19	36.17
PerArgument + CoDL + Multitask	32.02	12.30	17.77	46.99	18.06	26.09	70.76	17.80	28.45
Argument Detector + Argument Classifier	49.19	43.09	45.94	57.84	50.82	54.10	69.37	47.60	56.45
Mapping: upper-bound	67.82	48.58	56.61	89.09	65.82	75.70	91.57	70.25	79.50
Table 9: Results of learning an SRL parser from question-answer pairs.
as ELMo and Flair, we compare s-QUASE $_{QAMR}$  embeddings with ELMo embeddings on SRL and Coref, and compare s-QUASE $_{QAMR}$  embeddings with Flair on NER. The results are shown in Table 7. We find that s-QUASE $_{QAMR}$  has a better performance than ELMo and Flair, especially in the low-resource setting, which indicates that s-QUASE can provide extra features than ELMo and Flair.
C On the Strength of Distributed Meaning Representations
In this section, we first show more details of the comparison between QUASE with symbolic meaning representations in Section C.1. After that, we show the details of learning an SRL parser from QA-SRL in Section C.2.
C.1 Comparison with Symbolic Meaning Representations
Probing Analysis. We first show the details of our probing analysis on the Xinhua subset $^{12}$  of AMR dataset. Our probing task can be formulated as follows: given two nodes in order, the probing model needs to predict the directed relation from one node to the other. We only consider the cases where there is indeed a relation between them.
There are 741 sentences and 9008 relations in valid alignments with 70 different types of relations in the training set, and 99 sentences with 1098 relations in valid alignments with 43 different types of relations in the development set. We use the same edge probing model as (Tenney et al., 2019), but we train it by minimizing a softmax loss rather than binary cross-entropy loss. Therefore, our probing results are based on the classification accuracy, not binary F1 score.
Systematic Analysis. We use Large QA-SRL as a testbed to analyze the representation ability of p-QUASE $_{QAMR}$ . Our p-QUASE $_{QAMR}$  achieves 85.79 F1 score on the development set of Large QA-SRL, while BERT further pre-trained on SQuAD with the same number of QA pairs only achieves an F1 score of 64.63 (it achieves 86.98 F1 on SQuAD). For reference, BERT further pre-trained on Large QA-SRL can achieve 92.19 F1 on Large QA-SRL. All these numbers indicate that p-QUASE $_{QAMR}$  has a strong ability to answer questions related to SRL.
On the other hand, BERT further pre-trained on Large QA-SRL can only achieve 72.17 F1 on the development set of QAMR, while  $\mathsf{p - QUASE}_{\mathsf{QAMR}}$  can achieve 85.79 F1 on Large QA-SRL (it achieves 90.35 F1 on QAMR). These results show that QAMR can cover the questions related to SRL, but Large QA-SRL cannot cover many questions related to AMR. Therefore,
12Only four subsets in AMR dataset contain both training and development sets, but the other three subsets either use informal languages or templatic and report-like structures, which are quite different from the domain of QAMR.
8756
QAMR is a good choice for QUASE to be further pre-trained on.
Some Examples. He et al. (2015) show that QA pairs in QA-SRL often contain inferred relations, especially for why, when and where questions. These inferred relations are typically correct, but outside the scope of PropBank annotations (Kingsbury and Palmer, 2002). This indicates that QA-SRL contains some extra information about predicates. Some examples are shown in Table 8. We further verify that  $\mathfrak{p}\text{-QUASE}_{QAMR}$  can correctly answer questions in the examples, which means that QUASE can encode some extra information that SRL cannot.
Michael et al. (2017) show that QAMR can capture a variety of phenomena that are not modeled in traditional representations of predicate-argument structure, including instances of co-reference, implicit and inferred arguments, and implicit relations (for example, between nouns and their modifiers). Some examples of QAMR are shown in Table 8. Similar to SRL, we find that p-QUASE precedes traditional representations, such as AMR, by correctly answering questions in the examples and hence encoding extra information.
C.2 Learning an SRL Parser from QA-SRL
C.2.1 Learn an SRL Parser
We consider learning a SRL parser from QA-SRL. It reduces the problem of learning AMR from QAMR to a simplified case.
Challenges. There are three main challenges to learn an SRL parser from Large QA-SRL.
Partial issues. Only  $78\%$  of the arguments have overlapped with answers;  $47\%$  of the arguments are exact match;  $65\%$  of the arguments have Intersection/Union  $\geq 0.5^{13}$ .
Irrelevant question-answer pairs.  $89\%$  of the answers are "covered" by SRL arguments;  $54\%$  of the answers are exact match with arguments;  $73\%$  of the answers have Intersection/Union  $\geq 0.5$ . These statistics show that we also get some irrelevant signals: some of the answers are not really arguments (for the corresponding predicate).
Different guidelines. Even if the arguments and the answer overlap, the overlap is only partial.
A reasonable upperbound. We treat the answers that have overlapped with some arguments as our predicted arguments. If two predicted argu
ments intersect each other, we will use the union of them as new predicted arguments. The results are shown in Table 9. We know from the table that this mapping algorithm achieves a span F1 of 56.61, which is a reasonable upper bound of our SRL system.
Baselines. We consider three models to learn an SRL parser from Large QA-SRL dataset.
Rules + EM. We first use rules to change QA pairs to labels of SRL. We keep the labels with high precision and then use an EM algorithm to do bootstrapping. A simple BiLSTM is used as our model for SRL. The results are shown in Table 9. We think that low token F1 is due to the low partial rate of tokens (37.97%) after initialization.
PerArgument + CoDL + Multitask. We consider a simpler setting here. A small number of gold SRL annotations are provided as seeds. To alleviate the negative impact of low partial rate, we propose to train different BiLSTM models for different arguments (PerArgument) and do global inference to get structured predictions<sup>14</sup>. We first use seeds to train the PerArgument model and then use CoDL (Chang et al., 2007) to introduce constraints, such as SRL constraints, into bootstrapping. At the same time, we train a model to predict the argument type from question-answer pairs. These two tasks (argument type prediction and SRL) are learned together through soft parameter sharing. In this way, we make use of the information from QA pairs for SRL. We use 500 seeds to bootstrap. The span F1 of our method is 17.77 and the span F1 with only seeds is 13.65. More details are in Table 9. The performance of this model has only improved several percent compared to the model trained only on seeds.
Argument Detector + Argument Classifier. Given a small number of gold SRL annotations and a large number of QA pairs, there are two methods to learn an end-to-end SRL $^{15}$  system. One is to assign argument types to answers in the context of corresponding questions using rules, and learn an end-to-end SRL model based on the predicted SRL data. This is exactly our first model, Rules + EM. However, the poor precision of argument classification leads to unsatisfactory results. An-
<sup>14</sup>Given a predicate in the sentence with three arguments and one of them is annotated, the sentence is partial for a traditional SRL model but not partial for a PerArgument model.
<sup>15</sup>Note that an end-to-end SRL system is with gold predicates. This is different from the generic definition.
These statistics of partial issues and irrelevant question-answer pairs are based on the PTB set of QA-SRL.
8757
other method is to learn from small seeds and bootstrap from large number of QA pairs. Thich is our second model, PerArgument + CoDL + Multitask. However, bootstrapping can not improve argument detection much, leading to mediocre results. We also notice that argument detection is hard with a small number of annotated data, but argument classification is easy with little high-quality annotated data. Fortunately, most answers in Large QA-SRL overlap with arguments. Furthermore, the mapping results of argument detection is about 56.61, good enough compared to two baselines. We propose to learn two components for SRL, one is for argument detection and the other is for argument classifier. We use the span-based model in (FitzGerald et al., 2018) for argument detection. The argument classifier is trained on predicates in the PTB set of QA-SRL. The results are shown in Table 9.
C.2.2 Using SRL/AMR Parsers in Downstream Tasks
There have already been some attempts to use semantics in downstream tasks. We discuss three types of application here. Traditionally, semantic parsers can be used to extract semantic abstractions, and can be applied to question answering (Khashabi et al., 2018). Second, dependency graphs, such as SDP, can be incorporated into neural networks. For example, (Marcheggiani and Titov, 2017) encodes semantic information in Graph Convolution Networks (GCN). In order to use constituent based traditional symbolic meaning representations, one can encode related semantic information by multi-task learning (MTL). (Strubell et al., 2018) mentioned such an example of application.
The main difficulty of retrieving SRL/AMR from QA signals for downstream tasks is to learn a good parser for SRL/AMR from question-answer pairs.
8758