SlowGuess/NAACL_2022_OCR huggingface dataset:
Rui Zhang, Cicero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, and Dragomir Radev. 2018. Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 102-107, Melbourne, Australia. Association for Computational Linguistics.
A A Weighted Context-Free Grammar
A.1 The Grammar
In our WCFG  $\langle \Sigma, N, S, R, \rho \rangle$ ,  $\Sigma$  is the set of all tokens in the vocabulary,  $N = \{\mathrm{S}, \mathrm{X}_{\sigma}, \mathrm{X}_{\overline{\sigma}}\}$ , where S is the start symbol,  $\mathrm{X}_{\sigma}$  is the span of interest, and  $\mathrm{X}_{\overline{\sigma}}$  is the spans that are not of interest. The complete set of production rules  $R$  is shown in Tab. 10.
S	→	XσXσ
S	→	XσXσ
S	→	XσXσ
S	→	XσXσ
Xσ	→	XσXσ
Xσ	→	XσXσ
Xσ	→	XσXσ
Xσ	→	XσXσ
Xσ	→	XσXσ
Xσ	→	XσXσ
Xσ	→	XσXσ
Xσ	→	x, ∀x∈Σ
Xσ	→	x, ∀x∈Σ
Table 10: Production rules  $R$  of our WCFG.
We only assign nontrivial weights to rules  $_i\mathrm{X}_k \to _i\mathrm{Y}_{j j}\mathrm{Z}_k$  where  $X$  is  $\mathrm{X}_{\sigma}$ . That is to say, in Eq. (1),  $s_p(_i\mathrm{X}_k \to _i\mathrm{Y}_{j j}\mathrm{Z}_k) = 1$  where  $X$  is not  $\mathrm{X}_{\sigma}$ .
A.2 The Inside-Outside Algorithm
For a span  $[i, \mathrm{X}, k]$ , its inside value can be expressed as
$$
\beta ([ i, \mathrm {X}, k ]) = \sum_ {\mathrm {X} \rightarrow \mathrm {Y Z} \in R} \left(\exp s _ {p} ([ i, \mathrm {X}, k ]) \times \left(\sum_ {j = i + 1} ^ {k - 1} \beta ([ i, \mathrm {Y}, j ]) \times \beta ([ j, \mathrm {Z}, k ])\right)\right) \tag {22}
$$
In the case when  $k = i + 1$ , we have  $\beta([i, \mathrm{X}, k]) = \exp s_p([i, \mathrm{X}, k])$ . The inside value of the entire sentence  $\beta([0, \mathrm{S}, |\mathbf{w}|])$  is exactly  $Z$ , the sum of scores of all parses.
A.3 The CKY Algorithm
We use the CKY algorithm to find an optimal parse tree  $t^{*} \in \mathcal{T}(\mathbf{w})$  that maximizes  $\rho(t)$ . The recursive function when  $i < k$  used is:
$$
\gamma ([ i, \mathrm {X}, k ]) = \max  _ {\mathrm {X} \rightarrow \mathrm {Y Z} \in R} \left\{s _ {p} ([ i, \mathrm {X}, k ]) + \max  _ {i <   j <   k} \left\{\gamma ([ i, \mathrm {Y}, j ]) + \gamma ([ j, \mathrm {Z}, k ]) \right\}\right\} \tag {23}
$$
In the case when  $k = i + 1$ , we have  $\gamma ([i,\mathrm{X},k]) = s_p([i,\mathrm{X},k])$ .
The systems are implemented with PyTorch. We use SpanBERTlarge as text encoder. We train the model for 20 epochs and select the best-performing model on the development set for testing. The documents are split into 512 word segments to fit in SpanBERTlarge. Models used for coreference resolution have 402 million learnable parameters, and models for SRL have 382 million learnable parameters. We closely follow the hyperparameter settings of Joshi et al. (2020) and build our models upon the codebase of Xu and Choi (2020)<sup>9</sup> under Apache License 2.0. The learning rate of SpanBERTlarge parameters is set to
<sup>9</sup>https://github.com/lxucs/coref-hoi
$1 \times 10^{-5}$  with 0.01 decay rate, and the learning rate of task parameters is set to  $3 \times 10^{-4}$ . The dropout rate of feedforward neural network scorers is set to 0.3. When training our model, we randomly sample  $0.1|D|$  negative spans that are not mentions and add their negative log-likelihood  $-\log p(\sigma_{ij})$  to the training objective. This is to prevent  $p(\sigma_{ij})$  from converging to 1. For SRL task, we use a batch size of 32 for 40 epochs and the same learning rate with coreference resolution. The same negative sampling technique is applied. Our models are trained on Nvidia Tesla V100 GPUs with 32GB memory. The average training time is around 8 hours for Joshi et al. (S) baseline and around 9 hours for our model. For SRL models, training takes 25 hours.
C Results on the Development Set
In this section, we report the results that our models get on the development sets of OntoNotes and LitBank.
Avg. P	Avg. R	Avg. F1
Joshi et al. (S)	82.0	78.8	80.4
Sigmoid	81.6	79.4	80.5
Ours	81.1	80.2	80.7
Table 11: Results on CoNLL-2012 coreference resolution development set.
Avg. P	Avg. R	Avg. F1
Joshi et al. (S)	78.8	73.7	76.1
Ours	77.4	75.6	76.6
Table 12: Results on LitBank development set.
Avg. P	Avg. R	Avg. F1
He et al.SpanBERT	87.9	85.0	86.4
Ours	87.8	86.2	87.0
Table 13: Results on CoNLL-2012 semantic role labeling development set.