SlowGuess/ACL_2020_OCR huggingface dataset:
Table 11: Hyper-parameters for WIKISQL and WIKITQ. Values are constrained to either a range  $(a,b)$  or a list  $[a,b,c,\ldots ]$
Parameter	PRETRAIN	SQA	WIKI SQL	WIKI TQ
Training Steps	1,000,000	200,000	50,000	50,000
Learning rate	5e-5	1.25e-5	6.17164e-5	1.93581e-5
Warmup ratio	0.01	0.2	0.142400	0.128960
Temperature	1.0	0.107515	0.0352513
Answer loss cutoff	0.185567	0.664694
Huber loss delta	1265.74	0.121194
Cell selection preference	0.611754	0.207951
Batch size	512	128	512	512
Gradient clipping	10	10
Select one column	1	0	1
Reset cell selection weights	0	0	1
C Pre-training Example
In order to better understand how well the model predicts numbers, we relax our accuracy measure to a soft form of accuracy:
$$
a c c (x, y) = \left\{ \begin{array}{l l} 1 & \text {i f} x = y \\ 0 & \text {i f} x \text {o r} y \text {i s n o t a n u m b e r} \\ 1. 0 - \frac {| x - y |}{\max  (x , y)} & \text {e l s e} \end{array} \right.
$$
With this soft metric we get an overall accuracy of 74.5 (instead of 71.4) and an accuracy of 80.5 (instead of 53.9) for numbers. Showing that the model is pretty good at guessing numbers that are at least close to the target. The following example demonstrates this:
Table 12: Optimal hyper-parameters found for pretraining (PRETRAIN), SQA, WIKISQL and WIKITQ.
Team	Pld	W	D	L	PF	PA	PD	Pts
South Korea	2	1	1	0	33	22	11	5
Spain	2	1	<1>	<0>	31	24	7	5
Zimbabwe	2	0	0	2	22	<43,40>	- <19,18>	2
Table 13: Table example from the Wikipedia page describing the 1997 Rugby World Cup Sevens.  $\langle \mathrm{x}\rangle$  marks a correct prediction and  $\langle \mathrm{x},\mathrm{y}\rangle$  an incorrect prediction.
In the example, the model correctly restores the Draw (D) and Loss (L) numbers for Spain. It fails to restore the Points For (PF) and Points Against (PA) for Zimbabwe, but gives close estimates. Note that the model also does not produce completely consistent results for each row we should have  $\mathrm{PA} + \mathrm{PD} = \mathrm{PF}$  and the column sums of PF and PA should equal.
D The average of stochastic sets
Our approach to estimate aggregates of cells in the table operates directly on latent conditionally independent Bernoulli variables  $G_{c} \sim \mathrm{Bern}(p_{c})$  that indicate whether each cell is included in the aggregation and a latent categorical variable that indicates the chosen aggregation operation op: AVERAGE,
col0Home team	col1Home team score	col2Away team	col3Away team score	col4Venue	col5Crowd
geelong	18.17 (125)	hawthorn	6.7 (43)	corio oval	9,000
footscray	8.18 (66)	south melbourne	11.18 (84)	western oval	12,500
fitzroy	11.5 (71)	richmond	8.12 (60)	brunswick street oval	14,000
north melbourne	6.12 (48)	essendon	14.11 (95)	arden street oval	8,000
st kilda	14.7 (91)	collingwood	17.13 (115)	junction oval	16,000
melbourne	12.11 (83)	carlton	11.11 (77)	mcg	31,481
Question	What was the away team's score when the crowd at Arden Street Oval was larger than 31,481?
SQL Query	SELECT col3 AS result FROM table_2_10767641_15WHERE col5 > 31481.0 AND col4 = "arden street oval"
WIKISQL answer	[14.11 (95)]
Our answer	[ ]
Question	What was the sum of the crowds at Western Oval?
SQL Query	SELECT SUM(col5) AS result FROM table_2_10767641_15WHERE col4 = "western oval"
WIKISQL answer	[12.0]
Our answer	[12500.0]
Table 8: Table "2-10767641-15" from WIKISQL. "col6" was removed. The "Crowd" column is of type "REAL" but the cell values are actually stored as "TEXT". Below we have two questions from the training set with the answer that is produced by the WIKISQL evaluation script and the answer we derive.
SUM or COUNT. Given  $G_{c}$  and the table values  $T$  we can define a random subset  $S \subseteq T$  where  $p_{c} = P(c \in S)$  for each cell  $c \in T$ .
The expected value of  $\mathrm{COUNT}(S) = \sum_{c}G_{c}$  can be computed as  $\sum_{c}p_{c}$  and  $\mathrm{SUM}(S) = \sum_{c}G_{c}T_{c}$  as  $\sum_{c}p_{c}T_{c}$  as described in Table 1. For the average however, this is not straight-forward. We will see in what follows that the quotient of the expected sum and the count, which equals the weighed average of  $T$  by  $p_c$  in general is not the true expected value, which can be written as:
$$
\mathbb {E} \left[ \frac {\sum G _ {c} T _ {c}}{\sum G _ {c}} \right]
$$
This quantity differs from the weighted average, a key difference being that the weighted average is not sensitive to constants scaling all the output probabilities, which could in theory find optima where all the  $p_c$  are below 0.5 for example. By the linearity of the expectation we can write:
$$
\sum_ {c} T _ {c} \mathbb {E} \left[ \frac {G _ {c}}{\sum_ {j} G _ {j}} \right] = \sum_ {c} T _ {c} p _ {c} \mathbb {E} \left[ \frac {1}{1 + \sum_ {j \neq c} G _ {j}} \right]
$$
So it comes down to computing that quantity  $Q_{c} = \mathbb{E}\left[\frac{1}{X_{c}}\right] = \mathbb{E}\left[\frac{1}{1 + \sum_{j\neq c}G_{j}}\right]$ . The key observation is that this is the expectation of a reciprocal
of a Poisson Binomial Distribution  $^5$  (a sum of Bernoulli variables) in the special case where one of the probabilities is 1.
By using the Jensen inequality we get a lower bound on  $Q_{c}$  as  $\frac{1}{\mathbb{E}[X_c]} = \frac{1}{1 + \sum_{j \neq c} p_j}$ . Note that if instead we used  $\frac{1}{\sum_{j} p_j}$  then we recover the weighted average, which is strictly bigger than the lower bound and in general not an upper or lower bound. We can get better approximations by computing the Taylor expansion using the moments of  $X_{c}$  of order  $k$ :
$$
\begin{array}{l} Q _ {c} = \mathbb {E} \left[ \frac {1}{X _ {c}} \right] \simeq \frac {1}{\mathbb {E} [ X _ {c} ]} + \frac {\operatorname {v a r} [ X _ {c} ]}{\mathbb {E} [ X _ {c} ] ^ {3}} + \dots + \\ (- 1) ^ {k} \frac {\mathbb {E} \left[ (X _ {c} - \mathbb {E} [ X _ {c} ]) ^ {k} \right]}{\mathbb {E} [ X _ {c} ] ^ {k + 1}} \\ \end{array}
$$
where  $\operatorname{var}[X_c] = \sum_{j \neq c} p_j (1 - p_j)$ .
The full form for the zero and second order Taylor approximations are:
$^{5}$ wikipedia.org/Poisson_binomial_distribution
$^{6}$ wikipedia.org/Taylor_expansions_for_the_moments
$$
\begin{array}{l} \mathrm {A V E R A G E} _ {0} (T, p) = \sum_ {c} T _ {c} \frac {p _ {c}}{1 + \sum_ {j \neq c} p _ {j}} \\ \text{AVERAGE}_{2}(T,p) = \sum_{c}T_{c}\frac{p_{c}(1 + \epsilon_{c})}{1 + \sum_{j\neq c}p_{j}} \\ \mathrm {w i t h} \epsilon_ {c} = \frac {\sum_ {j \neq c} p _ {j} (1 - p _ {j})}{(1 + \sum_ {j \neq c} p _ {j}) ^ {2}} \\ \end{array}
$$
The approximations are then easy to write in any tensor computation language and will be differentiable. In this work we experimented with the zero and second order approximations and found small improvements over the weighted average baseline. It's worth noting that in the dataset the proportion of average examples is very low. We expect this method to be more relevant in the more general setting.