Wav2SQL: Direct Generalizable Speech-To-SQL Parsing (2024)

Abstract: Speech-to-SQL (S2SQL) aims to convert spoken questions into SQL queries given relational databases, which has been traditionally implemented in a cascaded manner while facing the following challenges: 1) model training is faced with the major issue of data scarcity, where limited parallel data is available; and 2) the systems should be robust enough to handle diverse out-of-domain speech samples that differ from the source data. In this work, we propose the direct generalizable speech-to-SQL parsing model Wav2SQL which avoids error compounding across cascaded systems. Specifically, 1) to accelerate speech-driven SQL parsing research in the community, we release a large-scale and multi-accent dataset MASpider; 2) leveraging the recent progress in the largescale pre-training, we show that it alleviates the data scarcity issue and allow for direct speechto-SQL parsing; and 3) we include the speech re-programming and gradient reversal classifier techniques to reduce acoustic variance and learned style-agnostic representation, improving generalization to unseen out-of-domain custom data. Experimental results demonstrate that Wav2SQL avoids error compounding and achieves state-of-the-art results by up to 4.7% accuracy improvement over the baseline. 1 2

heading: 1 Introduction
text: Speech-to-SQL parsing (S2SQL) aims to generate the SQL query from a spoken question based on relational databases. This technology is highly beneficial as it breaks down barriers among those who lack proficiency in SQL queries and are unable to perform screen inputs while driving or exercising. Furthermore, S2SQL provides flexible and convenient ways of interaction, which opens up a host of
∗Equal Contribution. †Corresponding Author.
1Dataset samples are available at https://Wav2SQL. github.io/
2Code is available at https://github.com/liuhuadai/ Wav2SQL
practical applications in fields such as vehicle terminals, smart watches, smart speakers, and the medical industry. Conventional S2SQL systems (Kumar et al., 2013; Song et al., 2022) are often composed of a cascade of two components: automatic speech recognition (ASR) (Yu and Deng, 2016; Schneider et al., 2019; Hsu et al., 2021) and text-to-SQL parsing (Bogin et al., 2019b,a; Chen et al., 2020; Guo et al., 2019). Compared to cascaded systems, work on direct S2SQL is very limited, with the potential benefits of 1) working on languages without written form (Campbell, 2008), where an estimated half of the 7,000 languages in the world actually do not have written forms; 2) avoiding error compounding across sub-systems (Nakamura et al., 2006; Jia et al., 2019).
The recent development of direct S2SQL parsing still faces several challenges: 1) despite the benefits of direct approaches, model training is faced with the major issue of data scarcity. Humanlabeled speech data is expensive to create, there are very few data resources providing parallel speech, and the data amount is quite limited, 2) increasing demand for SQL parsing from personalized speech challenges models especially in unseen scenarios. When the distributions of custom voice (speaker and accent) differ from training data, the system performance deteriorates due to distribution gaps, and 3) the modality gap between the spoken question and text schema hinders the ability of the scheme linking, making it difficult to align question speech to the intended tables.
To accelerate S2SQL research, we assemble an open-source, multi-speaker, and multi-accent S2SQL corpus MASpider. To the best of our knowledge, MASpider is the first open-source speechto-SQL parsing dataset. We have attached part of MASpider to the supplementary materials, and we will release the entire dataset after the paper publication. To overcome the aforementioned challenges in this paper, we propose Wav2SQL for
4230
direct speech-to-SQL parsing, which is generalizable to unseen acoustic conditions (speaker and accent) in custom data. To be more specific, 1) leveraging self-supervised learning (SSL) (Baevski et al., 2020; Hsu et al., 2021), it alleviates the data scarcity issue and benefits S2SQL model training, 2) we introduce speech re-programming and gradient reverse technique to effectively eliminate the style attributes in representation, which promote the model generalization to unseen speakers and accents in custom data.
Experimental results on the MASpider dataset demonstrate that our Wav2SQL model surpasses the cascaded system in the exact match accuracy and achieves competitive performance with our model trained on the TTS dataset. The main contributions of this work are summarized as follows:
• We introduce the first cross-domain speech-toSQL parsing benchmark dataset MASpider3.
• Leveraging self-supervised learning, we propose the direct generalizable speech-to-SQL parsing model and show that the large-scale pre-training alleviates the data scarcity issue.
• Through introducing speech reprogramming and gradient reversal technique, we effectively eliminate the style attributes in speech representation and predict the style-agnostic variation, which significantly improves the model generalization to unseen speakers and accents in custom data.
• Experimental results on the MASpider dataset demonstrate that our model outperforms the cascaded systems and achieves state-of-the-art performances.

heading: 2 Related Works

heading: 2.1 Text-to-SQL Parsing
text: Semantic parsing of natural language to SQL query recently surged in popularity because of the release of two cross-domain datasets-WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018). IRNet (Guo et al., 2019) encodes the question and schema via bi-LSTM and proposes the string match strategy for schema linking. RATSQL (Wang et al., 2019) presents a unified framework with
3Audio samples are available at https://Wav2SQL. github.io/
a relation-aware transformer(RAT) to encode relational databases and NL questions. SADGA (Cai et al., 2021) adopts the graph structure to provide a unified encoding model for both the NL question and databases. In recent years, speech-to-SQL systems usually adopt cascaded automatic speech recognition with text-based SQL parsing. However, the error propagation hurts model performance, not to mention that numerous languages do not have written forms. In this work, we present the first direct speech-to-SQL parsing model without using text, which demonstrates the generalization to different accents and speakers.

heading: 2.2 Self-Supervised Learning in Speech
text: Self-supervised speech representation learning encodes the speech feature into context representations. TERA (Liu et al., 2021) learns speech representation by reconstructing acoustic frames from their altered counterparts. Vq-wav2vec (Baevski et al., 2019) learns discrete representations via a context prediction task using contrastive loss. Similarly, wav2vec 2.0 (Baevski et al., 2020) is an end-to-end version of vq-wav2vec, while HuBERT (Hsu et al., 2021) predicts masked frames pre-quantized using k-means. In this work, we leverage the recent success of self-supervised learning in speech and show that large-scale pre-training alleviates the data scarcity issue and benefits model training.

heading: 2.3 Domain Generalization
text: Domain generalization aims to learn domaininvariant knowledge which can be generalized to the target distribution, which attracts attention from researchers (Zhou et al., 2020; Shi et al., 2021; Tian et al., 2022; Liu et al., 2023). Li et al. (2018b) propose a conditional invariant adversarial network to learn class-wise adversarial networks and Zhao et al. (2020) learns domain-invariant features by introducing additional entropy regularization to minimize the KL divergence between the conditional distributions of different source domains. For spoken language understanding, unseen speakers and accents in custom data significantly hurt model performance due to the distribution gaps. In this work, we introduce speech reprogramming and gradient reverse to disentangle semantically irrelevant information, leading to the significant promotion of model generalization to custom scenarios.

heading: 3 Dataset Construction
text: We build MASpider upon the Spider (Yu et al., 2018), which has 8659/1034 train/evaluation splits and an unreleased test set. MASpider consists of 9693 spoken utterances recorded by eleven speakers from six different countries. MASpider consists of 15 hours of speech samples recorded in a professional recording studio, including 8.1 hours from 6 females and 6.9 hours from 5 males apart from the person-of-interest (POI). Figures 1 summarize the distribution of dataset split, gender, and country. More details on MASpider are available in the appendix D. The major features of MASpider include:
• Open source. A lack of data could hinder the construction of speech-to-SQL systems, so we release our corpus to accelerate research in the community.
• Diversity. Since the distribution of custom voice could be different from training data, we construct a dataset with different gender, accent, and language background to improve model generalization.
• High quality. High-quality audios without excessive noise or error annotation are essential for S2SQL training. A strict verification ensures high-quality utterances in MASpider.

heading: 3.1 Data Collection and Verification
text: Collection Procedure For all 9693 utterance-SQL pairs in MASpider, we ensure that each speaker is assigned no more than 1500 sentences to avoid excessive data distribution bias. Next, we collect the audio sample of the given text utterances in a professional recording studio. Finally, the spoken utterances are saved in wav format, sampled at 16kHz, and quantized by 16 bits.
Data Labeling For further study, we tag additional statistics such as the native language and age of the speakers, and the year of their English study. Following this, the dataset is split into 12-hour spoken questions for training, additional 3-hour utterances with unseen accents, speakers, and databases for testing, which enable the evaluation of model generalization to custom data. Figure 1(a) illustrates the distribution of the training and test sets on MASpider.
Data Verification Firstly, we check that the accent in the recording matches the speaker’s country. Then, We listen to every recording to check for mispronounced errors and re-record the recording with more than two mispronunciations. Finally, we run the preliminary qualified recordings through an ASR system to control the recorded audio quality. In our case, we used the fine-tuned wav2vec 2.0 ASR model to filter out recordings with their character error rates higher than 25%. For audio with these error rates above the threshold, it is discarded and recollected again until passed.

heading: 3.2 Dataset Statistics
text: After the data collection and processing procedure, we check for audio quality and conduct the statistical evaluation.
Gender The visualization of gender statistics is displayed as Figure 1(b). As we can see, the ratio of male to female speakers is relatively average.
Country The recorders mainly include 4 English native speakers and 7 non-native speakers from Japan, China, Thailand, and Korea. We count the proportion of utterances recorded by these speakers and visualize it as shown in Figure 1(c)
Group By Difficulty Following the common practice (Yu et al., 2018) to better demonstrate the model performance on different SQL queries, we group the difficulty of each spoken question into 4 levels according to the number of SQL compo-
nents, selections, and conditions. Specifically, SQL queries that contain more keywords (e.g., GROUP BY, ORDER BY, INTERSECT, etc.) will be considered harder. In the end, The test set of MASpider consists of 25.5% easy, 37.9% medium, 20.9% hard, and 15.7% extra hard SQL queries.

heading: 4 Proposed Method

heading: 4.1 Overview
text: The overall architecture has been presented in Figure 2b. To alleviate the data scarcity issue (Liu et al., 2023; Huang et al., 2023), we leverage the large-scale self-supervised models including Hubert (Hsu et al., 2021) for the spoken question and language model for the textual schema to derive discriminative representation, enabling direct speech to SQL parsing. For generalizable speech to SQL parsing, we propose several techniques to promote model robustness for unseen (speaker and accent) custom data: 1) we re-program acoustic attributes and perturb the style information in speech, selectively extracting only the semantic information for domain-agnostic modeling; 2) we include gradient reversal classifier to eliminate speaker information with an auxiliary gradient reversal classifier.
In the end, the tree-structure decoder produces results with an abstract syntax tree (AST) in depthfirst traversal order. The training procedures are included in Section 4.5.

heading: 4.2 Enhanced Speech Encoder

heading: 4.2.1 Self-Supervised Pre-training
text: To alleviate the data scarcity issue (He et al., 2023; Liu et al., 2024) and learn high-level and semantic representations from raw waveform (Huang et al., 2022b,a; Cheng et al., 2023), we leverage recent progress in large-scale self-supervised learning with Hubert (Hsu et al., 2021), with a multi-layer convolution waveform encoder to generate the feature sequence followed by a Transformer (Vaswani et al., 2017) context encoder to build the contextualized representations.
We adopt the Hubert-Base model as speech representation, which is pre-trained on 960 hours LibriSpeech (Panayotov et al., 2015). Notably, speech representations (Choi et al., 2021) are found to merge not only rich acoustic information but also acoustic attributes related to accents and speakers. To reduce domain-specific variations for better generalization, we investigate a novel technique in the following parts, which effectively eliminates accent and speaker information in speech representations while preserving linguistic content.

heading: 4.2.2 Analysis: Speech Quality Across Layers
text: Before introducing our techniques for removing accent and speaker information, we first discuss the impact of the selection of different layers of Hubert on the model performance. Similar to natural language understanding, exploring the transformer layers of the BERT model shows that the under-
lying blocks encode syntactic information, while high-level semantic information appears in higher blocks. To make a more intuitive sense of this, we separately extract frozen representations of Hubert’s 12 layers as audio features. We then input these audio features into the S2SQL model and evaluate their performance by exact match accuracy. Figure 3 demonstrates that the first 7 layers as well as the last two layers have poor performance compared to layers 8 to 10 whose accuracy is higher than 39.0 %. Layer 9, with an accuracy of 41.5%, is selected for its superior representation as the final feature.

heading: 4.2.3 Re-program on Acoustic Condition
text: An intuitive way (Li et al., 2018a; Bui et al., 2021) to achieve better generalization is to decompose a model into the domain-agnostic and domainspecific parts via disentangled representation learning and eliminate the domain-specific variations.
In contrast, the representation derived from selfsupervised models contains not only rich semantic content but also information related to pitch and speaker, which are style-specific attributes that may decrease model generalization. As such, we conduct re-programming on speech attributes and perturb the rhythm, pitch, and energy information, which disentangle acoustic variation and selectively extract only the linguistic-related information, exhibiting better generalization to unseen custom data. As shown in Figure 2a, we apply bottlenecks on acoustic conditions and create re-programmed speech samples. Additional details have been attached in Appendix B.
Rhythm Rhythm characterizes how fast the speaker utters each syllable. To perturb rhythm
information, we adopt random resampling RR to divide the input into segments of random lengths, and we randomly stretch or squeeze each piece along the time dimension.
Pitch Pitch is an indispensable component of intonation. First, We normalize the pitch contour to a common mean and standard deviation, removing the timbre variations in speech. Secondly, a chain function is adopted to randomly shift the pitch contour.
Energy Energy represents the magnitude of the raw waveforms and visually reflects the volume of the speech. We re-program energy attributes and create samples with different energy distributions.

heading: 4.3 Gradient Reversal Classifier
text: To address the challenge of speaker identity obfuscation within speech representations, we have adopted a gradient reversal layer (GRL) within the framework of the speaker classifier (Ganin et al., 2016). This approach reconceptualizes speaker variation as an adversarial classification task, with the GRL serving as a pivotal component that actively inverts the gradient signal during backpropagation, thereby maximizing the classification loss of the speaker-discriminative domain.
The operational mechanics of the GRL during the backpropagation process can be succinctly encapsulated by the following mathematical formulation:
R(x) = x, dR
dx = −I, (1)
where I symbolizes the identity matrix. By implementing this strategic gradient inversion, the GRL ensures that the model promotes indistinguishability in feature distributions across diverse speaker identities. As a direct consequence, the resultant features are rendered speakeragnostic—thereby mitigating unwanted biases associated with accent or speaker-specific nuances. These sanitized audio features retain semantic integrity and exhibit an augmented capacity for generalization when deployed in the decoding of SQL at custom data, thus underscoring the efficacy of the GRL in enhancing the robustness of S2SQL systems against speaker variability.

heading: 4.4 SQL Decoder
text: The SQL decoder follows the grammar-based architecture of Yin and Neubig (2017), which generates
the SQL as an abstract syntax tree(AST) in depthfirst traversal order. The generation process of SQL AST is factorized into sequential actions, which are divided into two cases: (1) APPLYRULE which expands the last generated node according to the grammar rules or completes a leaf node, (2) SELECTCOLUMN and SELECTTABLE represent that selects a column or table item from the schema respectively.
Firstly, the probability of generating a SQL y is defined as:
p(y|x) = ∏
t
p(at|x, a<t)
where x is the encoded memory of questions, columns, and tables, at is the action token at time step t, and a<t is the sequential actions before time step t. Then in the tree-structured LSTM decoder, the hidden states at each time step t are updated as mt, ht = LSTM([at−1; pt; ct; nt],mt−1,ht−1), where mt is the cell state of time step t, ht is the hidden state, at−1 is the previous action embedding, pt is the parent information of the current node, ct is the context vector, and nt is the embedding of the current node type. Finally, how action probabilities p(at|x, a<t are computed are explained as follows: For APPLYRULE action,
p(at = AR[r]|x, a<t) = softmaxR(g(ht)) (2)
where AR is the APPLYRULE action and g(·) is the feed-forward network that is composed of two linear layers and a tanh activation function. For SELECTTABLE action,
γj = softmaxj( (htWQ)(xjWK +Rij) T )√ d ),
(3) p(at = ST[i]|x, a<t) = ∑
j
γj (4)
where ST denotes SELECTTABLE action. The calculation of SELECTCOLUMN action is similar.

heading: 4.5 Training and Inference
text: We formulate speech-to-SQL parsing as a sequenceto-tree generation problem. The input is the question (audio) and schema (text), which belong to two different modalities, while the output is the SQL query. We adopt the pre-trained self-supervised speech representation model Hubert (Hsu et al., 2021) and language model GloVe (Pennington et al., 2014) as the backbone of our model.
Training. The final loss terms in training are composed of the two parts:1) domain classification loss LCE : cross-entropy loss between the predicted speaker ID and the ground-truth; 2) SQL generation loss LMLE : maximum likelihood estimation(MLE) based on the given SQL query to maximize the predicted probability p(y|x, a<t) based on a given SQL query. Note that, the domain classification loss LCE is trained to remove speaker information but preserve semantic information, which is helpful for the final objective LMLE to generate more accurate SQL query and the loss weight of LCE is set to be 0.01. Inference. After training, for each pair of the spoken question and database schema, we generate the target SQL query according to grammar rules with heuristics decode. We replace the special tokens in the target sequences with the SQL keywords.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: Evaluation Metrics Following the common practice (Yu et al., 2018), we evaluate the performance by exact match accuracy and component matching accuracy provided by (Yu et al., 2018), where exact match accuracy measures whether the predicted query is equivalent to the gold query as a whole while component matching measures the average exact match between the prediction and ground truth on different SQL components.
Training and Inference We train our model on a single 80G NVIDIA A100 GPU with a batch size of 20 for 100 epochs using the AdamW optimizer. The learning rate is 5e − 4 and the weight decay coefficient is 1e− 4. We preprocess column names and table names for tokenization and lemmatization using Stanza toolkit (Qi et al., 2020). In inference, we adopt beam search decoding with beam size 5.
Model Configurations In the encoder, the hidden size is 256 and the number of RAT layers is 8. In the SQL decoder, we set the rule embedding size as 128 and the node type embedding size as 128. Following (Huang et al., 2022b; Lee et al., 2021), the ASR model in our work is Hubert-LargeLS960-FT. A comprehensive table of hyperparameters is available in Appendix C in the supplementary materials.
Baseline models We compare generated SQL queries of our Wav2SQL with other systems, in-
cluding: (1) Cascaded: the cascaded model composed of automatic speech recognition(ASR) and text-to-SQL parsing model, which adopts the Hubert-Large-LS960-FT (Hsu et al., 2021) and RATSQL (Wang et al., 2019). (2) S2SQL-TTS: the S2SQL model trained on the dataset synthesized by TTS model, where S2SQL means our Wav2SQL with Hubert Only. The TTS model we adopt here is FastSpeech 2 (Ren et al., 2020).

heading: 5.2 Model Performance
text: For in-domain evaluation, we prepare spoken questions with seen accents and speakers according to different SQL components, including SELECT, WHERE, GROUP, ORDER, AND/OR, and KEYWORD following (Yu et al., 2018). The results are compiled and presented in Table 1, and we have the following observations: Wav2SQL surpasses the cascaded system across all SQL component matching except GROUP component and exact match accuracy on all SQL queries. Specifically, the WHERE and ORDER component has increased significantly by 5.4% and 8.1% respectively, and exact match accuracy has increased by 4.7%, demonstrating the effectiveness of our direct speech-to-SQL parsing model. It indicates that our direct S2SQL model avoids error compounding across subsystems. Compare to the upper bound less variance dataset constructed by a singlespeaker single-accent TTS system, we still achieve competitive performance, indicating the efficiency of our proposed techniques for reducing acoustic attributes and promoting generalization.
To further verify the effectiveness of our methods, we compare our model with the cascaded sys-
tem. We group the parsing difficulty into easy, medium, hard, and extra according to the number of component selections and conditions of the target SQL queries. As illustrated in Table 2, we have the following observations:
1) As the parsing difficulty increases, a distinct degradation could be witnessed in generation accuracy; and 2) our direct speech-to-SQL parsing model outperforms the cascade baseline since it avoids error compounding across subsystems, demonstrating a large margin improvement by 7.8% in the extra hard part.

heading: 5.3 Generalization To Custom Data
text: For out-of-domain testing, we prepare spoken questions with databases, accents, and speakers that are unseen in custom data. The results are summarized in Table 1, and we have the following observations: 1) Under the challenge of invisible accents, Wav2SQL can still achieve better performance with a 3.4% exact match accuracy increase compared with the cascaded system, which validates the superiority of our model by exploiting speech re-program and adversarial training to get deterministic representations invariant to accents and speakers; 2) Although we are pleasantly surprised to find that Wav2SQL maintains comparative results with S2SQL-TTS in SELECT, GROUP BY
and AND/OR component accuracy, there is still a certain gap with it in exact match accuracy due to the limited acoustic information in the TTS dataset.

heading: 5.4 Ablation Studies
text: As shown in Table 1, we conduct ablation studies to demonstrate the effectiveness of several designs in our model, including speech re-programming, gradient reversal classifier technique, and selfsupervised model. The results have been presented in Table 1, and we have the following discovering: 1) the removal of the speech re-programming method shows an degradation in exact match accuracy by 1.4% and a significant decrease of 3.5% in SELECT component matching, indicating its efficiency in reducing acoustic variance and learning deterministic representations; 2) Removing the gradient reversal classifier has witnessed a distinct degradation in model performance by 2.1% accuracy especially in GROUP component matching (8.9%), showing its superiority in learning speakeragnostic speech representation; 3) Keeping only the Hubert module(i.e. removing both the speech re-programming and gradient reversal classifier) results in a significant performance decrease compared to adding each of them. This once again proves that both of the methods we propose are able to effectively preserve only semantic information in audio to improve model performance. 4) Upon replacing the Hubert model with the Whisper Encoder, there is a marked decline in exact match accuracy, plummeting by 10.9%. Moreover, comparisons reveal that the "w Hubert Only" setting outperforms the Whisper Encoder with a considerable advantage in component matching for all SQL elements. These results underscore that Hubert effectively encodes a wide variety of high-level
semantic features, which proves critical for the generation of precise SQL queries.

heading: 5.5 Case Study
text: We compare the SQL query generated by Wav2SQL with the cascaded system in Table 3. The results demonstrate that Wav2SQL outperforms the baseline system. For example, in the first and third cases, the cascaded fails to fill the values into the correct slots, thus, it stupidly forgets the ’Name’ of table ’Singer’ and is unable to select the correct column name ’Age’. In addition, Wav2SQL successfully completes the averaging operation on "Age" in the third case. Unfortunately, in the second example, the cascaded system incorrectly constructs the WHERE clause so that it fails to pick singers who are older than the average age.

heading: 6 Conclusion
text: We released MASpider, the first large-scale, multispeaker, and multi-accent S2SQL parsing dataset, which we hope would accelerate S2SQL research in the community. In this work, we presented the first direct speech-to-SQL model Wav2SQL which avoided error compounding across cascaded systems. To tackle the data scarcity issue, we leveraged recent progress in large-scale pre-training and utilized self-supervised models to derive discriminate representation. To promote model generalization and robustness to custom out-of-distribution data, we further introduced speech re-programming and gradient-reversal classifier techniques which reduced acoustic variance and learned style-agnostic representations. Experimental results demonstrated that our approach achieved new state-of-the-art results by up to 4.7% accuracy improvement over baseline. In the future, we will investigate techniques to further enhance the model generalization
in direct Speech-to-SQL parsing.

heading: 7 Limitation and Potential Risks
text: As mentioned in the model performance, there is still a certain gap between Wav2SQL and S2SQLTTS. One of our future directions is to further remove accent and speaker information to improve generation performance. In addition, our experiments find that the schema linking we adopt is still rough compared to text schema linking, which seriously affects the performance of our model. In future work, we will study how to obtain accurate and fine-grained schema linking.
Wav2SQL lowers the requirements for speechto-SQL generation, which may cause unemployment for people with related occupations database developers, and SQL programmers. Furthermore, there is the potential for leading to the misuse of databases than they expect.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China under Grant No. 62222211 and STATE GRID Corporation of China Project No.5211YF220006.

heading: A Domain Classifier
text: Domain classifier effectively captures the audio’s long-term speaker identity and predicts the speaker ID for the spoken question. After training on augmented data, the domain classifier could attain robust representations that capture an ample speaker identity space. Combined with gradient reversal, we can get deterministic representation agnostic to speaker discrepancy, significantly reducing introspeaker variance and making it possible for treestructured depth-first decoding.

heading: B Acoustic Perturbation
text: To obtain speech samples with acoustic information enhancement, we adopt the following functions (Qian et al., 2020; Choi et al., 2021) to perturb the acoustic features, that is 1) random resampling RR, and 2) formant shifting fs, and 3)pitch randomization pr, 4) random frequency shaping using a parametric equalizer peq. Next, we feed augmented audios into the model along with original audios.
• For RR, a random resampling is adopted to modify the rhythm. The raw waveform is divided into segments, whose length is randomly uniformly drawn from 19 frames to 32 frames (Polyak and Wolf, 2019). Each segment is resampled using linear interpolation with a resampling factor randomly drawn from 0.5 to 1.5.
• For fs, a formant shifting ratio is sampled uniformly from Uniform(1, 1.4). After sampling the ratio, we again randomly decided whether to take the reciprocal of the sampled ratio or not.
• For pr, a pitch shift ratio, and a pitch range ratio are sampled uniformly from Uniform(1, 2) and Uniform(1, 1.5), respectively. Again, we randomly decide whether to take the reciprocal of the sampled ratios or not. For more details on formant shifting and pitch randomization, please refer to Parselmouth https://github. com/YannickJadoul/Parselmouth.
• Lastly, peq denotes a serial composition of lowshelving, peaking, and high-shelving filters. We use one low-shelving HLS, one high-shelving HHS, and eight peaking filters HPeak.

heading: C Model Architectures
text: We list the model hyperparameters of Wav2SQL in Table 4 and illustrate the architecture for the
relational-aware transformer(RAT), SQL decoder, and domain classifier in Figure 4. The schema linking used by RAT in the train set is borrowed from RATSQL(Wang et al., 2019) while the schema linking of the test set comes from string matching between the ASR text and the schema. The ASR text is obtained through Whisper. The reason why Hubert is not selected here is that the numbers generated by it are in English and cannot be matched correctly.
Hubert The Hubert feature extractor consists of seven blocks and the temporal convolutions in each block have 512 channels with strides (5,2,2,2,2,2,2) and kernel widths (10,3,3,3,3,2,2), and 12 transformer blocks, model dimension 768, inner dimension (FFN) 3,072 and 8 attention heads.
Relation-Aware Transformer The relationaware encoder consists of 8 transformer layers. Each layer contains a relation-aware self-attention module, the final output passes through a feedforward layer.
C.1 Further Comparisons with Cascaded Systems
From the table, we observe that while employing the speech reprogram data augmentation method for fine-tuning the ASR model controls error propagation in the cascaded model, as evidenced by the decrease in ASR WER, ultimately resulting in slightly better performance than Wav2SQL, this improvement is not solely attributed to the proposed speech reprogram. It is also influenced by
the gain obtained from fine-tuning the ASR model using audio-text pairs from MASpider. Notably, Wav2SQL utilizes a frozen Hubert representation model.
To further elucidate, we conducted additional experiments by fine-tuning Hubert with audio-text pairs, revealing that Wav2SQL outperforms the cascaded model, providing additional confirmation of the effectiveness of our proposed direct model.

heading: D Dataset Annotation
text: We outsource the hiring of annotators and handle the data verification process internally. For the verification of the ASR model, we also utilize Wav2vec 2.0 Large(LV-60) + Self Training / 960 hours / Libri-Light + LibriSpeech. Each sentence is recorded by a single speaker, with a minimum, maximum, and average number of recorded utterances
per speaker at 747, 1232, and 881 respectively.
In-domain and Out-of-domain Test Set The division of in-domain is based on the Spider, resulting in 8659/1034 train/test sets, where speakers and accents are seen during training. Conversely, in the custom out-of-domain split, neither the speaker nor the accent is visible during training, and the train/test split is 8001/1692. It is worth noting that the databases used in these two divisions are invisible.

heading: E Ethical Considerations
text: Our MASpider benchmark presented in this work is a free and open source for the community to study speech-to-SQL parsing. We collect and annotate recordings from the mainstream text-to-SQL dataset, Spider (Yu et al., 2018), which is also a free and open dataset for research use. For audio recording, we hire annotators from different countries to record audio in a quiet environment. We pay the annotators an average of 80 dollars per hour.

heading: Future Work
text: One of our future directions is to further remove accent and speaker information to improve generation performance. In addition, our experiments find that the schema linking we adopt is still rough compared to text schema linking, which seriously affects the performance of our model. In future work, we will study how to obtain accurate and fine-grained schema linking. Wav2SQL lowers the requirements for speechto-SQL generation, which may cause unemployment for people with related occupations database developers, and SQL programmers. Furthermore, there is the potential for leading to the misuse of databases than they expect