SlowGuess/ACL_2020_OCR huggingface dataset:
To demonstrate the generalization of our framework, we re-implement CONNET and some baselines (MTV-SLM, Crowd-Add, Gold) with Transformer-CRF as the base model. Specifically, the base model takes Transformer as the encoder for CRF, which has shown its effectiveness in many NLP tasks (Vaswani et al., 2017; Devlin et al., 2019). Transformer models sequences with self-attention and eliminates all recurrence. Following the experimental settings from (Vaswani et al., 2017), we set the number of heads for multi-head attention as 8, the dimension of keys and values as 64, and the hidden size of the feed-forward layers as 1024. We conduct experiments with crowd-annotation dataset AMTC on NER task and cross-domain dataset UD on POS task, which are described in Section 5.1. Results are shown in Table 4. We can see our model outperforms over other baselines in both tasks and applications.
1
2	Defender [PER Hassan Abbas] rose to intercept a long ball into the area in the 84th minute but only managed to divert it into the top corner of [PER Bitar] 's goal.
[ORG Plymouth] 4 [ORG Exeter] 1
3	Hosts [LOC UAE] play [LOC Kuwait] and [LOC South Korea] take on [LOC Indonesia] on Saturday in Group A matches .
4	The former [MISC Soviet] republic was playing in an [MISC Asian Cup] finals tie for the first time .
5	[PER Bitar] pulled off fine saves whenever they did .
6	[PER Coste] said he had approached the player two months ago about a comeback .
7	[ORG Goias] 1 [ORG Gremio] 3
8	[ORG Portuguesa] 1 [ORG Atletico Mineiro] 0
9	[LOC Melbourne] 1996-12-06
10	On Friday for their friendly against [LOC Scotland] at [LOC Murray-field] more than a year after the 30-year-old wing announced he was retiring following differences over selection .
11	Scoreboard in the [MISC World Series]
12	Cricket - [MISC Sheffield Shield] score .
13	"He ended the [MISC World Cup] on the wrong note ,” [PER Coste] said .
14	Soccer - [ORG Leeds] ’ [PER Bowyer] fined for part in fast-food fracas .
15	[ORG Rugby Union] - [PER Cut-titta] back for [LOC Italy] after a year .
Table 3: Sample instances in Fig. 3 and Fig. 7 with NER annotations including PER (red), ORG (blue), LOC (violet) and MISC (orange).
Figure 7: Visualizations of (a) the expertise of annotators; (b) attention weights for additional sample sentences to Fig. 3. Details of samples are described in Tab. 3.
Methods	AMTC	UD
Precision(%)	Recall(%)	F1-score(%)	Accuracy(%)
MVT-SLM	72.21(±1.63)	51.72(±3.58)	60.21(±1.87)	87.23(±0.51)
Crowd-Add (Nguyen et al., 2017)	75.32(±1.41)	50.80(±0.30)	60.68(±0.67)	88.20(±0.36)
CONNET (Ours)	76.86(±0.33)	56.43(±3.32)	65.05(±2.32)	89.27(±0.31)
Gold (Upper Bound)	81.24(±1.25)	80.52(±0.37)	80.87(±0.79)	90.45(±0.71)
Table 4: Performance of methods with Transformer-CRF as the base model on crowd-annotation NER dataset AMTC and cross-domain POS dataset UD.