SlowGuess/NAACL_2022_OCR huggingface dataset:
We use a publicly available English modal dependency data set constructed by Yao et al. (2021), which consists of news articles from the following news media sources: Business Standard, Business Insider, NBC News, The New York Times, Reuters, The Guardian, The Washington Post, CNN, Fox News, Yahoo News and Wikinews. We also use a Chinese modal dependency data set constructed by Liu and Xue (2022) that consists of news articles from Xinhua newswire.
We optimize our models with the BertAdam optimizer of a linear scheduler with a warmup ratio of 0.1. The learning rate is 2e-5. We apply a dropout rate of 0.1 over the last layer of the pretrained language model output to get the contextualized representations. We use a 2-layer FFN with ReLU activations for all models. The hidden unit size of the FFNs is the hidden size of the pretrained language model, i.e. 1024 for bert-large-cased, 768 for xlm-roberta-base. For the proposed prompt-based model, we use a batch size of 12, maximum sequence length of 512 for Chinese, a batch size of 6, maximum sequence length of 384 for English. Sequences that are longer than the maximum sequence length are cut to segments with a stride of 64 for both languages.
We train all the models for 30 epochs on a NVIDIA Tesla V100 (16 GB) GPU. We run all
the models for 3 runs with different seeds, and report the average F-scores across runs. Each epoch takes about 45 minutes for English, 19 minutes for Chinese.
We give more details about the two ranking baselines: baseline (ours) and Yao et al. (2021)-J in Table 3. Given a child node, the two models first generate a candidate parent set for the child, then compute the pair score for each child-parent pair. The candidate parent with the highest pair score is selected as the parent. There are a few differences between the two models. First, Yao et al. (2021) encode a document sentence by sentence, i.e. they add a [CLS] and [SEP] token before and after each sentence and encode them with the language model. We encode all the sentences in a document together, i.e. we add a [CLS] and [SEP] token before and after each document, and encode it with the language model. If a document is longer than the maximum sequence length (T), we split it into segments and encode each segment independently. Each segment has T/2 overlapping tokens with the previous segment. The values of T are the same as the maximum sequence length values in section B. The final token representations are derived by taking the average of the token representations in each segment. Next, we obtain the node representations by simply taking the average token representations in a node, while they take the concatenation of the start token, end token and the span token vector in the node as the node representations. Lastly, even if Yao et al. (2021) propose a multi-task learning model by jointly learning node identification and structure building, they train the structure building stage with gold nodes. Our baseline is trained in an end2end fashion: the model first identifies nodes, then uses the system identified nodes as the input for the structure building stage.