SlowGuess/NAACL_2021_OCR huggingface dataset:
in MSQ-NER as inputs to the CGR encoder and obtain the averaged embedding vectors of all the gazetteer tags, e.g., B-PER and B-CW. To visualize these gazetteer tags, we apply t-SNE (Maaten and Hinton, 2008) and generate their 2D visualization shown in Figure 3. It is clear that the tags in the same type, e.g., B-PROD and I-PROD, are close to each other. This indicates that our CGR encode can identify the semantic meaning of these tags and provide effective gazetteer representation.
Some additional detailed results are included in this section.
Table 16 shows detailed precision, recall and F1 scores for each entity class, comparing the no gazetteer baseline model and the best model for each dataset. We note that the worst performance is on products and creative works, as we hypothesized since the entities are much more linguistically complex. These classes achieve the largest increases with our models, which demonstrates that our methods successfully make up the models' weakness in the complex entities challenge.
Parameter	Search Range	LOWER optimal
BiLSTM input word dimension	[50,200]	50
BiLSTM input charCNN dimension	-	16
BiLSTM input #filters	-	128
BiLSTM input filter size	-	3
BiLSTM hidden	[100, 512]	256
Feedforward #layers	[1,3]	1
Feedforward dimensions	[200,800]	512
Feedforward activation	{linear,tanh,relu}	linear
Matcher dimension	[20,100]	100
Matcher biLSTM hidden	[50, 512]	384
Matcher biLSTM dropout	[0,0.5]	0.1
LearningRate	[1e-4, 1e-3]	1e-3
Optimizer	{Adam,WAdam}	Adam
Epochs	-	50
Batch size	[16, 32]	32
Table 14: Optimal hyperparameters for BiLSTM-MoE models
Parameter	Search Range	LOWER optimal	WNUT17 optimal
Feedforward #layers	[1,3]	1	1
Feedforward dimensions	-	768	768
Feedforward activation	{linear,tanh,relu}	linear	linear
Matcher dimension	[20,100]	50	50
Matcher biLSTM hidden	-	384	384
Matcher biLSTM dropout	[0,0.5]	0.1	0.1
1-Stage lr	[1e-5, 1e-3]	3e-4	1.5e-3
1-Stage Optimizer	{Adam,BERT_Adam}	Adam	BERT_Adam
2-Stage lr	[1e-5, 1e-4]	3e-5	3e-5
2-Stage Optimizer	-	BERT_Adam	BERT_Adam
Epochs	-	50	50
Batch size	[16, 32]	25	32
Table 15: Optimal hyperparameters for BERT-MoE models
Dataset	Model	PER	GRP	LOC	CORP	CW	PROD
P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1	P	R	F1
LOWER	Baseline	93.2	96.3	94.8	80.3	83.1	81.6	91.4	90.4	90.9	82.0	67.7	74.2	74.1	70.9	72.5	54.0	49.7	51.8
Ours	96.8	96.6	96.7	91.3	88.9	90.1	94.0	92.2	93.1	85.9	87.9	86.9	85.3	80.3	82.7	65.0	60.3	62.5
MSQ-NER	Baseline	68.1	85.0	75.6	28.7	23.2	25.7	61.0	39.6	48.0	50.0	18.3	26.8	23.6	15.1	18.4	46.2	8.9	15.0
Ours	97.6	97.3	97.4	82.8	83.2	83.0	92.5	79.5	85.5	85.5	83.6	84.5	87.0	69.3	77.2	92.1	69.4	79.2
ORCAS-NER	Baseline	33.9	64.4	44.4	20.9	13.6	16.5	39.5	23.2	29.2	21.3	12.2	15.5	13.9	10.9	12.2	54.0	8.3	14.4
Ours	78.2	91.9	84.5	69.7	78.2	73.7	81.6	70.6	75.7	66.7	78.2	72.0	72.8	74.3	73.6	91.3	65.7	76.4
Table 16: Per-class performance across entity types. We show the optimal model (Ours) for each dataset, which is BERT+MoE+Two-stage for LOWER, and BiLSTM+MoE for MSQ-NER and ORCAS-NER. The baselines are BERT, BiLSTM and BiLSTM without gazetteer, respectively.