SlowGuess/NAACL_2022_OCR huggingface dataset:
Crase: Causal-Enhanced Table-Filler for Question Answering over Large Scale Knowledge Base
Minhao Zhang $^{1\spadesuit}$  Ruoyu Zhang $^{1\spadesuit}$  Yanzeng Li $^{1\spadesuit}$  Lei Zou $^{1,2\spadesuit}$
$^{1}$  Wangxuan Institute of Computer Technology (WICT), Peking University, China;  $^{2}$ Beijing Academy of Artificial Intelligence, Beijing, China;
$\clubsuit$  {zhangminhao,ry_zhang,zoulei}@pku.edu.cn
$\diamond$  liyanzeng@stu.pku.edu.cn
Semantic parsing solves knowledge base (KB) question answering (KBQA) by composing a KB query, which generally involves node extraction (NE) and graph composition (GC) to detect and connect related nodes in a query. Despite the strong causal effects between NE and GC, previous works fail to directly model such causalities in their pipeline, hindering the learning of subtask correlations. Also, the sequence-generation process for GC in previous works induces ambiguity and exposure bias, which further harms accuracy. In this work, we formalize semantic parsing into two stages. In the first stage (graph structure generation), we propose a causal-enhanced table-filler to overcome the issues in sequence-modelling and to learn the internal causalities. In the second stage (relation extraction), an efficient beam-search algorithm is presented to scale complex queries on large-scale KBs. Experiments on LC-QuAD 1.0 indicate that our method surpasses previous state-of-the-arts by a large margin  $(17\%)$  while remaining time and space efficiency. The code and models are available at https://github.com/AOZMH/Crake.
To incorporate knowledge in real-world question-answering systems, knowledge base question answering (KBQA) utilizes a background knowledge base (KB) as the source of answers to factoid natural language questions. Leveraging the versatility of KB query languages like SPARQL (Prud'hommeaux, 2008), many previous works (Unger et al., 2012; Yahya et al., 2012) adopted a semantic parsing paradigm for KBQA, in which questions are converted to equivalent SPARQL queries and answers are given by executing the queries in KB. Regarding the intrinsic graph structure of SPARQLs, some works further reduced such procedure as generating the query graph of SPARQLs w.r.t. questions. However, these methods either
Figure 1: Generating a query graph (bottom) by two stages to represent the SPARQL (right-top). At graph structure generation stage, node-extraction generates all graph nodes while graph-composition adds unlabeled edges between proper nodes. Then, the relation extraction stage decides the specific predicate of each edge.
require auxiliary tools (e.g. AMR in Kapanipathi et al., 2021, constituency tree in Hu et al., 2021, dependency tree in Hu et al., 2017) causing potential cascading errors, or rely on predefined templates (Cui et al., 2017; Athreya et al., 2021) limiting their expressiveness and generalization abilities.
To address these, efforts were made on devising independent pipelines for query graph construction (Lin et al., 2021). As in Figure 1, these pipelines usually involve a node extraction (NE) module to detect the mentions of all nodes in query graph and link entity mentions, a graph composition (GC) module to connect related nodes given by NE, and a relation extraction (RE) module deciding the KB predicate corresponding to each edge added in GC. In this framework, two drawbacks exist in previous works: 1) we observe strong causal effects between NE and GC, e.g. edges connected by GC are valid only between the node mentions extracted in NE, making GC decisions highly dependent on NE. To this regard, previous works (Zhang et al., 2021; Ravishankar et al., 2021) that perform NE and GC separately without causal-modelling may fall short in deeply comprehending the correlated tasks
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1787 - 1798
and accurately generating query graphs. 2) GC is commonly modelled as a sequence-generation in prior methods, either through generative decoder (Shen et al., 2019; Chen et al., 2021) or via stage-transition (Yih et al., 2015; Hu et al., 2018). However, sequence-modelling generally undergoes sequence ambiguity and exposure bias (Zhang et al., 2019) that harms model accuracy.
In this work, we formalize the generation of query graph in a two-staged manner as in Figure 1. At the first stage, we tackle the aforesaid weaknesses by a novel causal-enhanced table-filling model to jointly complete NE and GC, resulting in a query graph structure representing the connectivity of all nodes. More specifically, inspired by Chen et al. (2020a), we utilize a label transfer mechanism to facilitate the acquisition of causality between NE and GC (which solves drawback 1 above). Further, we apply a table-filler to decode all edges simultaneously, which naturally circumvents the ambiguity and bias of iterative decoding (and solves drawback 2). For the second stage, we propose a beam-search-based relation extraction algorithm to determine the predicate that binds to each graph edge. Differ from prior works, we perform candidate predicate retrieval and ranking alternately for each edge, limiting the candidate scale linearly w.r.t. KB degree and making the algorithm scalable for large-scale KBs like DBpedia.
In short, the major contributions of this paper are: 1) to our knowledge, we are the first to model GC as a table-filling process, which prevents the ambiguity and bias in prior works; 2) we model the intrinsic causal effects in KBQA to grasp subtask correlations and improve pipeline integrity; 3) our method outperforms previous state-of-the-arts on LC-QuAD 1.0, a prominent KBQA benchmark, by a large margin ( $\sim 17\%$ ), further experiments verifies the effectiveness of our approach.
We solve KBQA in a semantic parsing way, given a question (left-top in Figure 1), we generate a SPARQL query (right-top in Figure 1) to represent its semantics and answer the question by executing the query in KB. By definition, SPARQL describes a query graph with each triple in its body referring to a graph edge; by matching the graph pattern in KB, certain KB entries binding to the query graph can be processed as query results (e.g. in Table 1
Type	Example SPARQL
JUDGE	ask {dbr:New_York a dbo:City}
COUNT	select count(?) {?x a dbo:City}
SELECT	select ?x {?x a dbo:City}
Table 1: Supported query types.
for SELECT queries, all entries binding to the "select" node are results; for JUDGE queries, the existence of matched entries determines the boolean result). Hence, our task is further specified as constructing the query graph (bottom of Figure 1) of a question to represent its corresponding SPARQL.
2.2 Methodology Overview
Illustrated by Figure 1, we construct the query graph in two stages. In the graph structure generation stage (bottom-left in Figure 1), we extract all graph nodes by finding the mention of each node in question and its tag among {variable, entity, type}, e.g. the mention and tag for the node ?class is "class" and variable, respectively. Further, we link all non-variable nodes to KB entries, e.g. the type node with mention "person" links to dbo:person in Figure 1. Also, we decide the target ("select") node of the graph and add undirected edges between the nodes that are connected in the query graph, resulting in a graph structure representing the connectivity of all nodes.
Since all edges above are undirected and unlabeled, we fill in the exact KB predicate of each edge in the relation extraction stage (bottom-right in Figure 1) to construct a complete query graph.
Finally, we compose a SPARQL w.r.t. the query graph as output. Note that the body of the SPARQL exactly corresponds to the query graph, so only the SPARQL header is yet undetermined. Like Hu et al., 2021, we collect frequent trigger words in the train data to classify questions into COUNT, JUDGE or SELECT queries as in Table 1 (e.g. a question beginning with "is" triggers JUDGE). Thus, an entire SPARQL can now be formed. In the following sections, we expatriate our methodology for the two aforementioned stages.
3 Graph Structure Generation (GSG)
The overview of the model proposed for graph structure generation is illustrated by Figure 2. As discussed in Section 1, the model jointly deals with node extraction and graph composition via causalmodelling, which is detailed in this section below.
Figure 2: Causal-enhanced table-filling model for graph structure generation. The label-to-node and table-to-edge correspondence is illustrated by the poker and fruit symbols respectively.
3.1 Node Extraction (NE)
Node extraction discovers all nodes in the query graph, i.e. {?person, ?class, dbr: Swinhoe's_Crake, dbo: person} in Figure 1. We represent a node by its mention and tag, i.e. ("person", variable), ("class", variable), ("Swinhoe's Crake", entity) and ("person", type) for each node respectively.
This goal can naturally be achieved by multiclass sequence labeling. More specifically, let  $\mathbf{Q} \in \mathbb{N}^n$  be the question (token ids) with length  $n$ , we first encode it into hidden features  $\mathbf{H}_{rb}$  by a RoBERTa (Liu et al., 2019) encoder  $E_{rb}: \mathbb{N}^n \to \mathbb{R}^{n \times h_{rb}}$  with hidden size  $h_{rb}$ :
$$
\mathbf {H} _ {r b} = E _ {r b} (\mathbf {Q}) \in \mathbb {R} ^ {n \times h _ {r b}}
$$
Then,  $\mathbf{H}_{rb}$  is projected by a fully-connected-network (FCN)  $E_{ne}:\mathbb{R}^{n\times h_{rb}}\to \mathbb{R}^{n\times |L|}$  into  $\mathbf{Y}_{ne}$  in label space:
$$
\mathbf {Y} _ {n e} = E _ {n e} (\mathbf {H} _ {r b}) \in \mathbb {R} ^ {n \times | L |}
$$
$L = \{O\} \cup \{B,I\} \times \{V,E,T,VT\}$  is the label set denoting the mention span of variables (V), entities (E), types (T), or overlapping variable and type (VT). Now, the label prediction of each token can be given by  $\mathbf{P}_{ne} = \arg\max(\mathbf{Y}_{ne})$ ; also, given the gold token labels  $\mathbf{G}_{ne} \in \mathbb{N}^n$  (Figure 2 top), a model for NE can be trained by optimizing:
$$
\ell_ {n e} = - \frac {1}{n} \sum_ {i = 1} ^ {n} \log (\operatorname {s o f t m a x} (\mathbf {Y} _ {n e}) [ i; \mathbf {G} _ {n e} [ i ] ])
$$
Where  $[\cdot ]$  denotes tensor indexing.
After detecting all node mentions and tags, we link each non-variable node to KB entries by DBpedia Lookup and a mention-to-type dictionary built on train data to align the graph structure with KB. See Appendix A for more details in node linking.
3.2 Graph Composition (GC)
After node extraction, all nodes in the query graph remain unconnected. To form the structure of the query graph, graph composition inserts unlabeled and undirected edges between the nodes that are related in the query graph, leaving the specific predicate of each edge yet unresolved. Formerly, graph composition is commonly modelled as a edge-sequence-generation process via stage-transition (Yih et al., 2015; Hu et al., 2018) or generative decoders (Shen et al., 2019; Chen et al., 2021). Despite the strong expressiveness, modelling graph composition by a sequence usually suffers from two issues: 1) while the edge sequence is ordered, edges in the query graph are a set without order. For a graph with two edges  $e_1$  and  $e_2$ , both sequences  $e_1 - e_2$  and  $e_2 - e_1$  correctly represents the edges in the graph, but they are distinct from the perspective of sequence-generation. As a result, the edge set itself becomes ambiguous for the sequence, which confuses the model when comprehending a sequence and potentially decelerates the convergence. 2) As discussed by Zhang et al., 2019, without extra augmentation, sequence-generation
generally endures an exposure bias between training and inference, harming the model's accuracy when predicting. Hence, a robust model should address the issues above properly.
Here, we model graph composition by a table-filling process to decide all edges simultaneously involving no sequence-generation, which naturally circumvents all issues above. Let  $\mathbf{H}_{gc} \in \mathbb{R}^{n \times h_{gc}}$  be the hidden features for graph composition (the full definition of  $\mathbf{H}_{gc}$  with causal-modelling is given in Section 3.3; without causal-modelling, we simply have  $\mathbf{H}_{gc} = \mathbf{H}_{rb}$ ), we adopt a biaffine attention model (Dozat and Manning, 2017; Wang et al., 2021) to convert  $\mathbf{H}_{gc}$  into a table denoting the relationship between each token pair. More specifically, through two multi-layer-perceptrons (MLP)  $E_{head}$  and  $E_{tail}: \mathbb{R}^{n \times h_{gc}} \to \mathbb{R}^{n \times h_{bi}}$ , we first project  $\mathbf{H}_{gc}$  into head ( $\mathbf{H}_{head}$ ) and tail ( $\mathbf{H}_{tail}$ ) features:
$$
\mathbf {H} _ {\{h e a d, t a i l \}} = E _ {\{h e a d, t a i l \}} (\mathbf {H} _ {g c}) \in \mathbb {R} ^ {n \times h _ {b i}}
$$
Then, for  $\forall 1\leq i,j\leq n$  , the biaffine attention is performed between the head features of the  $\mathbf{i}^{\mathrm{th}}$  token  $\mathbf{h}_{head}^{(i)}$  and the tail features of the  $\mathbf{j}^{\mathrm{th}}$  token  $\mathbf{h}_{tail}^{(j)}$  producing  $\mathbf{s}_{i,j}\in \mathbb{R}^2$  representing the probability that an edge exists between the  $\mathbf{i}^{\mathrm{th}}$  and  $\mathbf{j}^{\mathrm{th}}$  token:
$$
\mathbf {s} _ {i, j} = \operatorname {s o f t m a x} \left(\operatorname {B i a f f} \left(\mathbf {h} _ {\text {h e a d}} ^ {(i)}, \mathbf {h} _ {\text {t a i l}} ^ {(j)}\right)\right)
$$
$$
\operatorname {B i a f f} (\mathbf {x}, \mathbf {y}) := \mathbf {x} ^ {T} \mathbf {U} _ {1} \mathbf {y} + \mathbf {U} _ {2} (\mathbf {x} \oplus \mathbf {y}) + \mathbf {b}
$$
As  $\mathbf{U}_1\in \mathbb{R}^{2\times h_{bi}\times h_{bi}}$ $\mathbf{U}_2\in \mathbb{R}^{2\times 2h_{bi}}$  and  $\mathbf{b}\in \mathbb{R}^2$  are trainable parameters,  $\oplus$  denotes concatenation. Combining all scores by  $\mathbf{Y}_{gc} = (\mathbf{s}_{i,j})_{(1\leq i,j\leq n)}\in$ $\mathbb{R}^{n\times n\times 2}$  , we now have a table describing the edge existence likelihood between any two tokens.
At training, we first obtain the boolean gold table  $\mathbf{G}_{gc} \in \mathbb{B}^{n \times n}$ , for every connected node pair in the query graph, the element in  $\mathbf{G}_{gc}$  corresponding to any pair of tokens belonging to the mentions of the two nodes respectively is set to 1 (resulting in several rectangles of 1s). Also, we prefix the question with a special [CLS] token and connect it with the target node to represent the "select" edge; for ASK queries without target nodes, a [SEP] token is suffixed and connected with [CLS]. Note that since the graph structure is undirected,  $\mathbf{G}_{gc}$  is a symmetric matrix. An example of  $\mathbf{G}_{gc}$  can be found in Figure 2. With  $\mathbf{G}_{gc}$ , we can train the table-filler by  $\ell_{tb}$ :
$$
\ell_ {t b} = - \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} l o g (\mathbf {Y} _ {g c} [ i; j; \mathbf {G} _ {g c} [ i; j ] ])
$$
(c)
Figure 3: Modelling NE and GC with (a) and without (b) causality, as X, YNE, and YGC denotes question, NE predictions, and GC predictions. Model (c) learns the causal effects by a label transfer module.
Following Wang et al., 2021, we also introduce  $\ell_{sym}$  to grasp the table symmetry. Finally, we optimize  $\ell_{gc} = \ell_{tb} + \ell_{sym}$  to train a model for GC.
$$
\ell_ {s y m} = \frac {1}{n ^ {2}} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {n} \sum_ {k = 1} ^ {2} | {\bf Y} _ {g c} [ i; j; k ] - {\bf Y} _ {g c} [ j; i; k ] |
$$
At inference, for each pair of nodes given by NE, we average the rectangle area in  $\mathbf{Y}_{gc}$  corresponding to the mentions of the node pair as its edge existence probability. The node pairs with a probability higher than 0.5 are connected. This threshold is selected intuitively to denote an edge is more likely to exist against to not exist, though we argue that the prediction is insensitive to any threshold in reasonable range (e.g.  $0.3\sim 0.7$ ).
3.3 Causal Modelling NE and GC
Up to now, NE and GC are treated as separate tasks that fail to model the intrinsic causal effects between them (e.g. edges in  $\mathrm{Y}_{\mathrm{GC}}$  only exist between the mentions detected in NE). Here, we model such causality by a mediation assumption in Figure 3(b) denoting the causal dependence of GC on both question and NE prediction by edge  $\mathrm{X} \rightarrow \mathrm{Y}_{\mathrm{GC}}$  and  $\mathrm{Y}_{\mathrm{NE}} \rightarrow \mathrm{Y}_{\mathrm{GC}}$  respectively. To grasp this causal graph, we devise a label transfer (Chen et al., 2020a) module to enable the transfer of NE predictions to GC, i.e. representing  $\mathrm{Y}_{\mathrm{NE}} \rightarrow \mathrm{Y}_{\mathrm{GC}}$ , in Figure 3(c).
In detail, we sample NE predictions  $\widetilde{\mathbf{Y}}_{ne}$  by gumbel softmax (Nie et al., 2019) with  $g\sim Gumbel(0,1)$
Figure 4: Candidate retrieval and raking framework for relation extraction.
and temperature  $\tau$
$$
\widetilde {\mathbf {Y} _ {n e}} = \operatorname {s o f t m a x} \left(\left(\mathbf {Y} _ {n e} + \boldsymbol {g}\right) / \tau\right) \in \mathbb {R} ^ {n \times | L |}
$$
$\widetilde{\mathbf{Y}}_{ne}$  is then embedded by label embedding  $\mathbf{W}_{le} \in \mathbb{R}^{|L| \times h_{le}}$  and concatenated with  $\mathbf{H}_{rb}$  to form  $\mathbf{H}_{gc}$  in Section 3.2 with  $h_{gc} = h_{rb} + h_{le}$ :
$$
\mathbf {H} _ {g c} = \mathbf {H} _ {r b} \oplus (\widetilde {\mathbf {Y}} _ {n e} \mathbf {W} _ {l e}) \in \mathbb {R} ^ {n \times h _ {g c}}
$$
Now, by minimizing  $\ell_{gsg} = \ell_{ne} + \ell_{gc}$ , a joint model for NE and GC can be obtained. In this model, GC receives NE labels to learn the causal effects from NE, while NE gets feedback through differentiable label transfer to further aid GC decision. In this sense, our model improves the integrity of graph structure generation compared with separately modelling each subtask or simple multitasking.
4 Relation Extraction (RE)
As shown in Figure 4, relation extraction (RE) conducts candidate retrieval and ranking in turn for each edge in graph structure  $S$  to decide its predicate. For a question  $q$ , an edge  $e$  connecting nodes  $n_1$  and  $n_2$  with mention  $m_1, m_2$  respectively, candidate retrieval recalls a set of predicates  $P$  that can be bound to  $e$ . Note that unlike  $e$ , each predicate in  $P$  is directional. Then, candidate ranking  $\operatorname{Rank}(P, q, m_1, m_2)$  gives each predicate a score. This section details this procedure.
Candidate Ranking For each  $p_i \in P$ , we encode it together with  $q, m_1, m_2$  by a RoBERTa encoder and pool them to  $0 \leq s_i \leq 1$  to score the predicate. If the direction of  $p_i$  is  $n_1 \to n_2$ , we join  $q, m_1, m_2, p_i$  sequentially by [SEP] token as model input; otherwise (direction  $n_2 \to n_1$ ), the
join order is  $q, m_2, m_1, p_i$ . By giving  $s_i$  to each candidate, we can get the most proper predicates for  $e$  by selecting those with highest scores. More details on training the ranking model can be found in Appendix B.
Candidate Retrieval Zhang et al., 2021 proposed a straightforward way to retrieve candidates: if either  $n_1$  or  $n_2$  is a non-variable node, the predicates around that node in KB are viewed as candidates; otherwise, they trace  $n_1$  or  $n_2$  in other graph edges with non-variable nodes and view the predicates k-hop away from that node in KB as candidates (e.g. predicates 2-hop away from dbo:person are candidates for ?class-?person in Figure 4). We view this as the baseline in latter experiments.
However, this results in a candidate scale  $\mathrm{O}(n^{k})^{1}$ , making it unscalable to multi-hop queries ( $\mathrm{k\uparrow}$ ) and large KBs ( $\mathrm{n\uparrow}$ ). Here, we propose Algorithm 1 to limit the scale to  $\mathrm{O}(\mathfrak{n})$ . We start by selecting an edge between  $n_1^a$  and  $n_1^b$  containing a non-variable node (e.g. edge ?class-dbr:Swinhoe's_Crake in Figure 4), retrieving all adjacent predicates of that node in KB and use Rank to select the most proper predicate  $p_1$  (e.g. dbp:named_by) of score  $s_1$ , this forms a subgraph  $G = \{(n_{1}^{a},n_{1}^{b},p_{1})\}$  with only one edge whose score is  $s_1$ . Then, we sample another edge between  $n_2^a$  and  $n_2^b$  (e.g. ?class-?person) and retrieve its candidates  $P$  based on  $G$  (e.g.  $G$  already entails ?class=dbr:bird, so all neighbors of dbr:bird forms  $P$ ), this process is denoted as Retrieve  $(G,n_2^a,n_2^b)$ . Now, we use Rank to se
${}^{1}\mathrm{n}$  is the node degree in  $\mathrm{{KB}},\mathrm{k}$  is the edge number in  $S$
Type	Methods	P	R	F1
I	NSQA (Kapanipathi et al., 2021)	.448	.458	.445
EDGQA (Hu et al., 2021)	.505	.560	.531
II	QAmp (Vakulenko et al., 2019)	.250	.500	.330
NAMER (Zhang et al., 2021)	.438	.438	.435
STaG-QA (Ravishankar et al., 2021)	.745	.548	.536
Crake (ours)	.722	.731	.715
Table 2: End-to-end performance on LC-QuAD 1.0 test set. I/II stands for methods with/without aux tools. We re-implement NAMER since its results on LC-QuAD is not provided; however, NAMER suffers from severe timeout issues on DBpedia to limit its performance, so we restrict each candidate query to run at most 45s in practice (which already requires  $\sim$ 15h for a complete evaluation run).
lect  $p_2$  of score  $s_2$  from  $P$ , add  $(n_2^a, n_2^b, p_2)$  to subgraph  $G$  and update its score as  $s_1 * s_2$ . Repeating this loop until all edges are bound with a predicate, we finally form a query graph.
Note that for each edge, the candidate scale given by Retrieve is  $\mathrm{O}(\mathfrak{n})$ , since it is always among the neighbors of one or several KB nodes. Also, to improve the recall of query graphs, this process can trivially be extended as a beam search with each step maintaining a beam of subgraphs  $B$ , ordering each subgraph by  $\prod_{i} s_{i}$  as in Algorithm 1.
Dataset We adopt LC-QuAD 1.0 (Trivedi et al., 2017), a predominant open-domain English KBQA benchmark based on DBpedia (Auer et al., 2007) 2016-04, to test the performance of our system. We randomly sample 200 questions from train data as dev set and follow the raw test set, resulting in a 4800/200/1000 train/dev/test split. More details on the dataset can be found in Appendix C. Like Zhang et al., 2021, we do not experiment on multiple datasets due to the high annotation cost involved, however, we conduct no dataset-specific optimizations in this work, so we consider the large improvements on LC-QuAD and detailed discussions sufficient to prove our effectiveness.
Annotation We annotate the dataset with the mention of each node in query graph, e.g. the mention "class" and "person" for the node ?class and dbo:person respectively in Figure 1. With the annotation, we obtain the gold data  $(G_{ne}, G_{gc})$  to train our models. Appendix D details the annotation process.
Figure 5: EM accuracy of GSG during training. See the meaning of each series in Table 3.
Baselines We evaluate our method against existing works both with and without auxiliary tools. With aux tools, Kapanipathi et al., 2021 constructs query graphs based on the AMR of questions; Hu et al., 2021 designs rules on constituency tree to aid query graph formation. For independent pipelines without aux tools, Vakulenko et al., 2019 parses URI mentions from the question to match with KB via confidence score passing; Ravishankar et al., 2021 combines a generative graph-skeleton decoder with entity and relation detector to form a query; Zhang et al., 2021 co-trains a pointer generator with the node extractor to build a query graph, it's worth to note that this work also requires the node-to-mention Annotation for training.
Setup We utilize the RoBERTa-large released by huggingface (Wolf et al., 2020) as our encoder. All experiments are averaged on two runs on an NVIDIA A40 GPU. For the GSG model, we train for at most 500 epochs ( $\sim$ 6 GPU-hours) and report the best checkpoint on dev set; for the RE model, we train for 20 epochs ( $\sim$ 16 GPU-hours) and report the final checkpoint. For hyperparameters, we find no apparent performance variance on dev set as long as the values are in reasonable range (e.g.  $64 \leq h_{le} \leq 1024$ ,  $1e-6 \leq lr_{sgs} \leq 2e-5$ ) so no further tuning is involved. See the full setting in Appendix E.
5.1 End-to-end Evaluation