Average	Kp20K	Inspec	Krapivin	NUS	SemEval	DUC
Model	10	50	M	10	50	M	10	50	M	10	50	M	10	50	M	10	50	M	10	50	M
RNN Greedy
RANDOM	0.2	0.2	0.2	0.2	0.2	0.2	0.4	0.4	0.4	0.1	0.1	0.1	0.2	0.2	0.2	0.3	0.3	0.3	0.0	0.0	0.0
ALPHA	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.0	0.0	0.0	0.7	0.7	0.7	0.0	0.0	0.0
ALPHA-REV	0.2	0.2	0.2	0.2	0.2	0.2	0.4	0.4	0.4	0.1	0.1	0.1	0.1	0.1	0.1	0.3	0.3	0.3	0.0	0.0	0.0
S->L	0.1	0.1	0.1	0.2	0.2	0.2	0.1	0.1	0.1	0.0	0.0	0.0	0.0	0.0	0.0	0.2	0.2	0.2	0.0	0.0	0.0
L->S	0.2	0.2	0.2	0.2	0.2	0.2	0.6	0.6	0.6	0.0	0.0	0.0	0.0	0.0	0.0	0.6	0.6	0.6	0.0	0.0	0.0
ORI	0.3	0.3	0.3	0.2	0.2	0.2	0.6	0.6	0.6	0.1	0.1	0.1	0.2	0.2	0.2	0.5	0.5	0.5	0.0	0.0	0.0
ORI-REV	0.1	0.1	0.1	0.2	0.2	0.2	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.3	0.3	0.3	0.0	0.0	0.0
PRES-ABS	0.2	0.2	0.2	0.2	0.2	0.2	0.5	0.5	0.5	0.3	0.3	0.3	0.2	0.2	0.2	0.3	0.3	0.3	0.0	0.0	0.0
ABS-PRES	0.2	0.2	0.2	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.3	0.1	0.1	0.1	0.4	0.4	0.4	0.0	0.0	0.0
TRANS Greedy
RANDOM	1.8	1.8	1.8	2.8	2.8	2.8	2.2	2.2	2.2	0.8	0.8	0.8	3.1	3.1	3.1	2.1	2.1	2.1	0.0	0.0	0.0
ALPHA	1.5	1.5	1.5	3.0	3.0	3.0	3.4	3.4	3.4	0.9	0.9	0.9	0.9	0.9	0.9	1.1	1.1	1.1	0.0	0.0	0.0
ALPHA-REV	1.6	1.6	1.6	2.5	2.5	2.5	3.9	3.9	3.9	1.0	1.0	1.0	1.4	1.4	1.4	0.7	0.7	0.7	0.0	0.0	0.0
S->L	1.4	1.4	1.4	2.8	2.8	2.8	2.5	2.5	2.5	0.3	0.3	0.3	1.8	1.8	1.8	1.3	1.3	1.3	0.0	0.0	0.0
L->S	1.3	1.3	1.3	2.4	2.4	2.4	2.4	2.4	2.4	0.6	0.6	0.6	1.6	1.6	1.6	0.8	0.8	0.8	0.0	0.0	0.0
ORI	1.5	1.5	1.5	2.5	2.5	2.5	3.1	3.1	3.1	1.1	1.1	1.1	1.2	1.2	1.2	0.9	0.9	0.9	0.1	0.1	0.1
ORI-REV	1.1	1.1	1.1	2.5	2.5	2.5	2.0	2.0	2.0	0.5	0.5	0.5	0.8	0.8	0.8	0.8	0.8	0.8	0.0	0.0	0.0
PRES-ABS	1.3	1.3	1.3	2.3	2.3	2.3	2.7	2.7	2.7	0.9	0.9	0.9	1.0	1.0	1.0	1.2	1.2	1.2	0.0	0.0	0.0
ABS-PRES	1.8	1.8	1.8	2.8	2.8	2.8	3.8	3.8	3.8	0.6	0.6	0.6	2.0	2.0	2.0	1.6	1.6	1.6	0.0	0.0	0.0
RNN beam50
RANDOM	1.8	2.0	2.0	2.3	2.7	2.7	2.4	2.7	2.7	3.0	3.2	3.2	1.5	1.9	1.9	1.5	1.6	1.6	0.0	0.0	0.0
ALPHA	2.1	2.3	2.3	2.6	2.8	2.8	2.9	3.2	3.3	2.9	3.3	3.3	2.3	2.6	2.6	1.8	2.0	2.0	0.0	0.0	0.0
ALPHA-REV	1.3	1.4	1.4	2.0	2.0	2.0	1.8	1.9	1.9	1.4	1.5	1.5	2.1	2.2	2.2	0.6	0.8	0.8	0.0	0.0	0.
S->L	1.6	1.8	1.8	2.1	2.2	2.2	2.1	2.2	2.2	2.3	2.9	2.9	1.3	1.4	1.4	1.9	2.0	2.0	0.0	0.0	0.2
L->S	1.4	1.4	1.4	1.8	1.8	1.8	2.1	2.1	2.1	2.5	2.5	2.5	1.1	1.1	1.1	0.8	0.8	0.8	0.0	0.0	0.0
ORI	1.9	2.3	2.3	2.3	2.7	2.7	2.7	3.2	3.2	2.6	3.3	3.3	2.3	2.8	2.8	1.2	1.9	1.9	0.0	0.0	0.0
ORI-REV	1.5	1.5	1.5	1.9	1.9	1.9	2.0	2.0	2.0	2.3	2.3	2.3	1.3	1.4	1.4	1.5	1.6	1.6	0.0	0.0	0.0
PRES-ABS	2.2	2.5	2.5	2.7	3.2	3.3	2.8	3.3	3.3	3.3	3.7	3.7	2.6	2.9	3.0	1.6	1.7	1.7	0.0	0.0	0.0
ABS-PRES	1.1	1.1	1.1	1.3	1.3	1.3	1.6	1.6	1.6	1.5	1.5	1.5	1.0	1.0	1.0	1.0	1.0	1.0	0.0	0.0	0.0
TRANS beam50
RANDOM	6.6	7.9	8.2	10.2	12.5	12.7	11.3	12.9	13.1	3.5	5.4	5.5	10.0	11.2	11.5	4.9	5.7	6.0	0.1	0.1	0.1
ALPHA	6.5	9.0	9.3	10.9	14.6	14.8	12.3	16.6	16.9	3.9	5.5	6.1	7.3	10.6	10.8	4.7	6.8	7.1	0.0	0.0	0.0
ALPHA-REV	6.7	9.4	9.6	10.5	14.7	15.0	10.9	14.8	15.0	2.8	5.7	6.1	11.2	14.6	15.0	4.6	6.5	6.6	0.0	0.0	0.2
S->L	6.3	8.3	8.4	10.2	13.0	13.2	10.8	14.0	14.0	3.1	5.2	5.2	9.0	11.0	11.3	5.0	6.6	6.9	0.0	0.0	0.0
L->S	6.7	9.4	9.7	11.0	15.2	15.6	11.4	14.9	15.4	3.9	6.4	7.0	9.2	12.7	12.8	4.9	7.1	7.2	0.0	0.0	0.0
ORI	6.9	9.7	9.9	11.0	14.7	15.1	11.7	16.5	16.7	5.0	7.4	7.6	8.9	12.2	12.6	4.6	7.3	7.5	0.1	0.1	0.1
ORI-REV	7.2	10.3	10.7	11.2	15.9	16.5	12.0	16.7	17.1	4.7	7.0	7.7	9.8	13.7	14.3	5.3	8.2	8.4	0.0	0.0	0.0
PRES-ABS	7.0	9.8	10.2	11.1	15.0	15.5	12.1	16.6	17.5	4.2	6.7	7.1	9.0	12.5	12.7	5.9	8.3	8.4	0.0	0.0	0.0
ABS-PRES	6.1	6.9	6.9	10.2	11.3	11.4	10.3	11.1	11.1	3.8	5.0	5.0	7.4	7.8	7.8	4.9	6.0	6.0	0.0	0.0	0.0