SlowGuess/ACL_2020_OCR huggingface dataset:
Model Configuration. For our model, we chose a hidden layer of  $h = 256$  dimensions. We optimize using the Adam (Kingma and Ba, 2014) optimizer with the standard values. We let the model train using early stopping with a patience on the validation loss, where the validation set is the left-out fold in 5 fold cross-validation. For the word embeddings, we use the pre-trained FastText embeddings (Bojanowski et al., 2017), which are refined during the training phase.
B Query Complexity
The tables below show more details of the coverage (see Tables 6 and 7) and the average number of joins per query (see Table 8).
C Example of Tree Sampling
Figure 4 shows a randomly sampled tree. During Phase 1 of the annotation procedure, an annotator associated the tree with the question: What is the average movie vote of different movies having an Oscar nominee with a cast character called Jesse and were nominated for an Oscar in the year 1991 or later?. In the second phase of the annotation, the tokens of the questions were associated with the nodes of the tree. The tree is depicted from root to leaves, where the root node is the last operation, and the leave nodes are the GetData-nodes. Here we describe the tree sampling procedure in more detail with the tree in Figure 4 as an example.
1 The query type is selected. There are five different types: list, sum, count, average, and Boolean. In our example, the average was selected. This can be forced manually or randomly sampled.
2 The result type is selected, which, in this case, is movie票_average. This can also be set manually or be sampled at random. Based on the query type, only certain types of results
are allowed. More precisely, for average and sum operations, only numeric result types are allowed.
3 The join path is selected. In the first step, a path length is selected, which can be predefined or randomly sampled. In this case, the path length is set to five. Then, in a second step, a random path of the predefined length is selected. In the current example, the query path is: movie, cast, person, oscar_nominee, oscar. The path always starts with the result type of table.
4 The set operation is selected among union, intersection, or difference. In this example, there is no set operation. After the operation was selected, a subpath is chosen, on which the set operation is performed. For instance, if we wanted to know the movies where Brad Pitt and George Clooney worked together, then the subpath movie, cast, person is selected. Finally, two different filters are inserted, one for each actor.
5 The group by operation is selected. First, the operator is selected among sum, average, or count. Then, the group by attribute and the aggregation attribute are selected. In our example, there is no group by operation.
6 The aggregation operation is selected among the min and max operation. This is relevant for the questions of the type: Which movie has the highest rating. In this example, we have no aggregation operation.
7 The filters are selected. For this, we define the number of total filters and the maximal number of filters per table. In this case, we set the number of filters equal to 2, and the maximal number of filters per table to one. Then, the appropriate number of attributes is selected randomly alongside the path. In this case, the tables oscar and cast were selected. Then, an attribute is selected, followed by a comparison operator and a value, which is randomly sampled from the database. In our example, we have: oscar.year  $\geq 1991$  and cast.character = Jesse.
D Annotation Tool
The annotation process is performed in two phases: writing an NL question for a given OT, and as
$^{6}$ https://spacy.io/
TOTAL	CHINOOK	COLLEGE	DRIVING SCHOOL	FORMULA_1
SPIDER	0.917 (0.87)	0.727	0.909	1	0.692
OUR DATASET	0.949	1	0.818	1	0.923
Table 6: Table Coverage, in % to total amount of existing tables. Our dataset shows better table coverage, except for one database (college), where the coverage differs by one table. The biggest improvement in coverage was achieved on the database formula_1, which is also the most complex database with the largest amount of tables. The number in braces indicates the average table coverage for the databases with more than 5 tables.
TOTAL	CHINOOK	COLLEGE	DRIVING SCHOOL	FORMULA-1
SPIDER	0.621 (0.496)	0.354	0.383	0.730	0.221
OUR DATASET	0.544	0.584	0.384	0.756	0.442
Table 7: Attribute Coverage. Our method gives better attribute coverage in particular for larger datasets, for instance, FORMULA_1. The number in braces indicates the average attribute coverage for the databases with more than 5 tables.
TOTAL	CHINOOK	COLLEGE	DRIVING SCHOOL	FORMULA-1
SPIDER	0.504	0.667	0.412	0.441	0.925
OUR DATASET	1.15	0.95	1.18	0.837	1.2
Table 8: Average number of joins per query.
signing tokens from the NL question to the nodes within the OT. We have built two user interfaces, one for each phase. Figure 5 shows screenshots of both tools.
Phase 1. In the first phase, the annotators are presented with an OT and the constraints. Their task is to write an appropriate question for the OT. For this, they can adapt the constraints, in case that they are nonsensical. Furthermore, the annotators can access the intermediate results for each node in the tree to better understand what the OT does. In cases where the OT cannot be annotated with an appropriate question, the OT can be skipped.
Phase 2. For the second phase (Figure 5 (b)), the annotators are presented with an OT and an NL question, which was written by another annotator in the previous phase. The task is to correct the question, and then assign the tokens of the NL to the nodes (i.e., operations) in the tree. For this task, the tool guides the annotator from node to node in the OT. Moreover, for each node, the annotator can choose the corresponding tokens. In the final step, the annotators can correct their token assignment using drag-and-drop features.
Guidelines. In order to have consistent annotations (especially in the second phase), we provided the annotators with extensive tutorial videos. On average, the annotators took 30 minutes to get used to the tool and start to work efficiently. For the first phase, we instructed the annotators to write an
appropriate question and gave examples, as well as examples of pitfalls.
For the second phase, we introduced stricter guidelines, as we noticed that annotators had trouble with this step. Especially, the join operations were not clear to the annotators. Thus, we decided on the following rules:
- Table: If the table denotes an entity type (e.g., movie), the tokens that denote this entity type are to be assigned (e.g., "movies"). If the table is a bridge table, which denotes a relationship between entities (e.g., production_country), then the tokens that denote this relationship are to be assigned to the operation (e.g., "movies", "produced", "in").
- Joins: For the join operations, the same guidelines as for the bridge-tables are to be followed.
- Filter: For the filter constraints (e.g., "person_name = Tom Cruise") the tokens, which represent the constraint, are to be selected (e.g., "by", "Tom", "Cruise").
- Query type: For each query type (e.g., count, average, sum, ...), the tokens that correspond or trigger this question type are to be selected (e.g., "How", "many").
Annotators. We recruited 22 annotators, which have a basic understanding of database technologies. We paid each annotator  $25\mathrm{\Phi}$  per hour. Each
Figure 4: Example of a randomly sampled tree. The nodes denote the node type with their arguments. The Tokens are assigned during the second phase of the annotation process. This tree is the answer to the question: What is the average movie vote of different movies having an Oscar nominee with a cast character called Jesse and were nominated for an Oscar in the year 1991 or later?
annotator was given access to a set of instruction videos as well as a user manual. Furthermore, the annotators could pose questions in a forum.
Figure 5: The annotation tool. (a) The OT and the constraints are shown to the annotators. For each node, the annotators can inspect the result of the execution. The annotators write a question and (b) assign the tokens of the question to the operations.