UQE: A Query Engine for Unstructured Databases

Authors: hanjun  dai
bethany yixin wang
xingchen  wan
bo  dai
sherry  yang
azade  nova
pengcheng  yin
phitchaya  mangpo phothilimthana
charles  sutton
dale  schuurmans
google  deepmind
google  cloud
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:03:49

Abstract: Analytics on structured data is a mature field with many successful methods.
However, most real world data exists in unstructured form, such as images and conversations.
We investigate the potential of Large Language Models (LLMs) to enable unstructured data analytics.
In particular, we propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.
This engine accepts queries in a Universal Query Language (UQL), a dialect of SQL that provides full natural language flexibility in specifying conditions and operators.
The new engine leverages the ability of LLMs to conduct analysis of unstructured data, while also allowing us to exploit advances in sampling and optimization techniques to achieve efficient and accurate query execution.
In addition, we borrow techniques from classical compiler theory to better orchestrate the workflow between sampling methods and foundation model calls.
We demonstrate the efficiency of UQE on data analytics across different modalities, including images, dialogs and reviews, across a range of useful query types, including conditional aggregation, semantic retrieval and abstraction aggregation.

Main paper: 
 introduction: 
 Data analysis [13] is essential for making well founded decisions and enabling businesses and society to function more effectively. Relational databases [12, 32] and the Structured Query Language (SQL) [7] have delivered huge successes in structured data management and analysis. Typically, such data is collected and organized in a pre-defined schema [14] , where the data properties and relationships have been pre-specified, and downstream analysis is restricted to this schema.In most real-world applications, however, data exists in unstructured formats, such as images, documents and audio recordings. Without preprocessing such data into structured forms, traditional SQL engines can only support limited queries. Preprocessing, including document entity retreival [45] and form understanding [42] , also require training on downstream tasks given a predefined taxonomy. This naturally motivates the question we consider in this paper:How can one perform unstructured data analysis in a flexible and efficient way?In the literature, full-text search engines [20] support scalable regexp-matching search on unstructured data, but this becomes infeasible for more complex semantic reasoning queries. Retrieval-Augmented Generation (RAG) [39, 24, 18] allows question answering on a subset of related data, but is not directly applicable to generic analytical tasks with aggregation and semantic queries that spans over an entire large database. Recent advances in Large Language Models (LLMs) [4, 2] unlock the ability to perform flexible question answering, especially with recent long-context models [33] . However, setting aside the cost per query, data analytics can still be challenging for LLMs without fine-tuning [25] or few-shot demonstrations [9] , even given structured tables. Very entertaining...No purposeful plot, ...Bold, brash, exciting... Recently, a promising line of work considered marrying LLMs with programming frameworks [38] , where logical or arithmetic operations are offloaded to program interpreters [17] . A most relevant example for analytics and table understanding tasks is Cheng et al. [11] , which augments classical SQL semantics with LLMs as user-defined functions (UDF). While promising, the execution of such SQL programs that embed LLM calls still requires sweeping over the entire database, which is too costly for large collections of unstructured content. To overcome this barrier, we leverage the synergy between LLMs and programmatic execution to define an Unstructured Query Language (UQL) that augments SQL for flexible semantic queries, with a focus on improving scalability and efficiency.A key observation is that the efficiency of classical SQL engines relies on (1) indexing structures that avoid the need to scan the entire database, and (2) a compilation system that determines the best execution order for operations. Based on these ideas, we propose the Unstructured Query Engine (UQE), which refines and extends this design principle to unstructured data analytics. To achieve similar effect to indexing, UQE casts the problem as learning to search or sample, seeking to avoid a full database scan with statistically sound methods. Additionally, a compilation system is developed that determines the best execution order and operator combination for different clauses in a UQL query, with the goal of minimizing LLM calls while preserving query semantics.As part of this project, we have created four new benchmark datasets, with both text and image modalities, along with three common analytic tasks. Compared to baseline methods, such as longcontext LLMs and embedding based retrieval, UQE achieves significant improvements in terms of the accuracy and cost reduction on these benchmarks. 
problem: 
 Before defining the problem we are solving, we first establish the terminology and notation we will use throughout the paper. A concrete illustration of the following terms is given in Figure 1 .• Table / database: We define a table T = {T i } N i=1 as an unordered set of |T | = N rows, where each row T i = [T i,1 , T i,2 , . . . , T i,M ] is an array of M elements such that M is the total number of columns in the table. Each row can consist of elements T i,• of heterogeneous types (e.g., datetime, float, enum) with different modalities (e.g., text, image), while elements in each column T •,j must be of the same format and modality.• Structured data: A column T •,j is structured w.r.t. a query if it can be accessed quantitatively, such as by algebraic operations over numeric data, comparison over string labels with predefined vocabulary (e.g., categorical labels), datetime functions, etc.. • Unstructured data: A column T •,j is unstructured if a query cannot access it using standard quantitative access. Typically, such a column does not belong to a predefined taxonomy. Examples include text (e.g., dialogs), images, videos, and other forms of data that usually require semantic understanding and preprocessing before performing any algebraic operations. • Concrete column: A column is concrete if it already exists in the table.• Virtual column: A column is virtual if it does not already exist in the table, but a query is able to operate on it. Conceptually, one needs to derive (partial rows of) these columns by processing the data from concrete columns. In our work, we bypass this step by creating such columns lazily and selectively, which is the key to achieving efficiency and performance gains.SQL engines can perform analytic queries on databases by manipulating structured data in concrete columns. The focus of this paper is to propose a new query engine that can perform analytics on databases with both structured and unstructured data, with queries that operate over both concrete and virtual columns. Standard analytics tasks that we seek to enable over unstructured data are:• Conditional aggregation: perform aggregation operations on a sub-table filtered by a condition.• Semantic retrieval: collect relevant rows specified by semantic filters.• Abstraction and aggregation: group the rows based on abstractions and then performs aggregation.Since optimizing queries on structured data within concrete columns is well-studied, we focus instead on techniques for handling queries on unstructured data over virtual columns. However, the UQE implementation also supports operations over structured data within concrete columns. In the following, unless stated otherwise, the term unstructured databases refer to databases containing both structured and unstructured data.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We discussed that in the conclusion section and we pointed out three major limitations.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
unstructured query language: 
 First, we need to formally define the query language, UQL, that talks to the unstructured databases.The idea of defining a natural query language for unstructured data is not completely new (e.g., Cheng et al. [11] , even though UQL has richer semantics), nor is the specific syntax or design of UQL the main focus of this paper. However, we need to define the scope of queries that the engine can handle, and breakdown the semantic meaning of each clause.We assume a basic familiarity of SQL, upon which UQL is based. UQL can be considered to be a dialect of SQL that has augmented functionalities for handling unstructured and virtual column queries. The SQL clauses that we support in UQL, along with necessary modifications to support unstructured semantic queries, are described as follows.SELECT is a mapping function that maps the operand (usually a row or collection of rows in a grouped query) to a new row of elements. In traditional SQL, this mapping is usually a subset selection over concrete columns, or algebraic operators over those columns. UQL provides additional semantic mapping capability as:SELECT "the attribute specified by natural language" AS attribute_name For example, one can write SELECT "the sentiment of the movie review" over an unstructured movie review column, and retrieve "positive" or "negative" as a structured output.FROM specifies the source of the table. In SQL one can additionally specify table joins, but we limit our attention to sourcing from a single table in this paper.WHERE intrinsically specifies a binary classifier over rows, which is used to retrieve a subset of the database. In addition to comparator operators on structured columns, we also allow semantic specifications in the form of:WHERE "the row satisfies some natural language specifications"The predicates in WHERE are organized in disjunctive normal form (DNF) with AND and OR syntax, so a user can arbitrarily express predicates over concrete and virtual columns.GROUP BY partitions the table into groups, where rows within each group share the same attributes over the keys being grouped by. UQL allows partitioning over virtual columns via natural language:GROUP BY "the abstraction criteria specified in natural language"Similar to WHERE, one can GROUP BY over both concrete and virtual columns by concatenating multiple criteria, with the resulting partition corresponding to grouping by a tuple of these keys.We also reuse other clauses from SQL including: ORDER BY, which simply inherits the SQL semantics to rank the resulting rows according to a specific concrete column. In most analytics tasks, sorting is applied over structured columns with well defined ordering comparators. LIMIT is applied during processing in the form of LIMIT num_rows, which limits the number of output rows.Assumptions: We rely on the ability of an LLM to perform intra-row semantic understanding and analysis tasks. For example, we assume that LLMs are able to correctly judge the specification in WHERE for a single row. Similarly, LLMs should be able to extract the information specified by SELECT or GROUP BY for a single row. We build programmatic functionality on top of this fundamental ability of LLMs to handle analytics for large databases. A UQL query is a composition of clauses that can be categorized as an aggregation or a nonaggregation, as illustrated in Figure 2 . Aggregation queries perform a summary on groups of aggregated rows, such as COUNT the number of rows, or summarize a common attribute in a group of rows (as defined in GROUP BY above). Non-aggregation queries perform operations on individual rows, which usually means the rows can be processed in parallel. A UQL query will only belong to one of the above two types and we do not consider nested queries for now. The query type determines how UQE will optimize and execute the query.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The data and prompts are all public and we have provided the information. We are working on open sourcing the code after going through the internal approval process.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
unstructured query engine: 
 One straightforward way to run UQL is to use an interpreter that executes queries imperatively. Cheng et al. [11] implements an engine in this form, which is able to handle tables of relatively small size. By analogy, this is similar to executing an SQL program using a linear database scan. While it is valid, the latency and cost are prohibitive and generally prevent scaling to real world scenarios.There are (at least) two key techniques in SQL databases for making query execution efficient:• Indexing, which organizes concrete columns via data structures for fast search with sublinear cost.• Compilation, which considers alternative query plans and executes the most efficient one.UQL queries over virtual columns pose challenges in both indexing and compilation. In this section, we present effective approaches to indexing (Section 4.1) and compilation (Section 4.2) for unstructured databases, along with the implementation details of low-level primitives (Section 4.2.2).Executing traditional SQL queries over indexed columns can be made efficient by avoiding an entire database scan to find the relevant rows to process. However, for UQL queries over virtual columns, it is hard to predict or predefine an index that can enable efficient searching, since these columns are not concrete and defined via arbitrary natural language specifications.Our first key contribution is to introduce a proxy for "indexing" that allows one to leverage the intrinsic semantic content of a virtual column to efficiently execute queries without scanning the entire database. The main idea is to use statistically sound sampling techniques to approximately retrieve relevant rows for processing. Based on the two types of queries defined in Section 3.2, we develop corresponding "indexing" counterparts.We use a simple query to illustrate the idea of unbiased estimation to obtain a query result without scanning over an entire virtual column.SELECT COUNT(*) as count FROM movie_reviews WHERE "the review is positive" Given a row T i (a natural language movie review), it is relatively easy for an LLM to tell whether it satisfies the WHERE condition. If we use f : (T i , cond) → {0, 1} to represent the LLM's classification of whether row T i satisfies the conditions specified in cond, then the goal is to estimate the quantityN i=1 f (T i , cond) = N N i=1 1 N f (T i , cond) = N E i∈{1...N } [f (T i , cond)](1) There are many approaches that can be used to estimate the finite sum in the above equation, with different tradeoffs between bias and variance. One unbiased but potentially high variance estimator is to simply use Monte Carlo samples from a uniform distribution over 1 . . . N . A typical technique for reducing variance is to use importance sampling with a proposal p, according toEQUATIONA theoretically optimal proposal p is given as follows:Algorithm 1 Stratified sampling for unbiased aggregation • Embed each row T i as ⃗ e i , using a multi-modal embedding over the unstructured columns of T i .• Cluster the embeddings{⃗ e i } N i=1 into K disjoint groups {C k } K k=1 , C k ⊆ {1 . . . N },where k can be a predefined constant or automatically selected [48] . Each group has size |C k | andK k=1 |C k | = N .We use c : {1 . . . N } → {1 . . . K} to denote the cluster index of each row.• Perform stratified sampling over these groups and obtain samples S ⊆ {1 . . . N }. Then we can obtain the following estimator for the expectation:EQUATIONAlgorithm 2 Online active learning for non-aggregation retrieval 1. Embed each row T i as ⃗ e i , using multi-modal embedding over the unstructured columns of T i . 2. Maintain ĝ(i) that approximates f (T i , cond). Initialize ĝ(i) ∝ U (0, 1) uniformly. Proposition 1 The optimal proposal distribution p that minimizes the variance of estimation in Eq 2 is p i ∝ f (T i , cond), which achieves zero variance.Prop 1 indicates that an ideal proposal should sample rows that have positive f with equal probability, while sampling negative rows with zero probability. However, given that f is the response of an LLM it is expensive to execute over all rows, forcing us to consider efficient approximations.Stratified sampling leverages the ability to partition a population into homogeneous subpopulations.As shown in Prop 1, a good proposal p i should predict whether a row T i satisfies the target property cond. To trade-off between cost and variance reduction, we propose Algorithm 1. In Eq 3 we normalize the importance weights w i to further reduce the estimation variance.Extension The above estimator can be used for other aggregation operations such as SUM and AVERAGE, including GROUP BY, and allowing concrete columns as operands as well. However, some aggregations such as MAX does not admit such an estimator. UQE in this case can only provide estimates with greater effort. We discuss limitations in Section 7.A non-aggregation query can be viewed as a search problem where we want to find a relevant subset of rows to process. As before, we begin with a concrete example:SELECT dialog_ID FROM dialogs WHERE "the customer is unhappy with the agent's manner"In practice, we aim to identify as many rows as possible that satisfy the given condition while adhering to budget constraints (e.g., the total number of tokens allowed to expense). This can be formulated as an online learning problem: given the token budget (or approximately, the number of LLM calls over individual rows, denoted as B), we seek to balance exploration (to better understand the semantic landscape of all rows) with exploitation (to maximize recall). Drawing inspiration from Bayesian optimization, which employs a surrogate model learned on-the-fly to inform sequential decision-making [36, 19, 22] , we use a cheap proxy ĝ as a surrogate for f (T i , cond). At each step t, we re-train ĝ with the observed data and select its maximizer to query in the next step. See Algorithm 2. Here we use random noise ϵ t,i to allow some degree of exploration that decays with t.Compared to the typical max inner-product search method prevalent in RAG systems, we rely on the online learning to adjust the beliefs, instead of solely relying on predefined embedding similarities.Classically, the goal of a compiler is to translate a high level program to low-level machine code, maintaining exact execution results with improved execution speed. Such a lowering process is usually accompanied by optimizations e.g., fusing, selecting the optimal instructions, and kernel optimizations. Our goal is similar: we would like to compile high-level UQL into low-level machine code, with the distinction that the "machine" is an LLM, and the "low-level code" is the orchestration of prompt calls to the LLM. Given that the primary bottleneck is the LLM API calls, we attempt to maintain the execution semantics while minimizing the cost of LLM calls, as in Figure 3 . Lowering a query into sequences of concrete execution units is a planning problem: The action space includes the order of clause execution, as well as ways to fuse clauses to execute together.The objective is to minimize the (estimated) LLM cost. Figuring out the best decomposition and combination is usually an NP-hard problem. Fortunately, the number of clauses is very limited for a single query, so we can enumerate possible combinations of ordering and fusions with little overhead.The outcome of planning is a specification of a sequence of kernel executions. The input and output of each kernel can be one of the following: In the following 3 sections we will explain the building blocks of the compiler, including the kernel implementation, the cost estimation and final instantiation in detail.Each kernel is an standalone execution unit that reads and produces a (stochastic) table.SELECT on structured columns is straightforward. When operating on unstructured columns, we prompt the LLM to extract semantic attributes from the input data. If several extractions share the same source column, we can also group these together into a single prompt to reduce cost.WHERE takes a logical formula in disjunctive normal form, such that each conjunction can contain predicates over both unstructured and structured columns. One optimization we make in this case is to perform evaluations over the structured columns first, then simplify (e.g., remove a conjunction if any of the structured column evaluates to false) the logical formula. Any remaining predicates over unstructured columns are then executed on the table filtered by predicates over structured columns.GROUP BY first gathers a representative subset of rows from the table, then calls an LLM to extract a taxonomy (i.e., the description of each cluster) for a cluster abstraction. Then the taxonomy is used to classify rows sampled according to the methods defined in Section 4.1. Finally, each row is classified into one of the clusters with the corresponding cluster description in the taxonomy.Other standard kernels like ORDER BY are implemented as-is since they are efficient to execute.Kernel fusion: Certain clauses can be fused together to achieve significant efficiency gains.• WHERE + LIMIT can be terminated earlier for non-aggregation queries, once the number of rows specified by LIMIT are retrieved. This is particularly useful for rare event finding. • SELECT + GROUPBY when executed together, the semantic attribute extraction of SELECT and taxonomy classification in GROUPBY can be done in the same LLM call to save cost. • GROUPBY + WHERE can share the same sampling proposal for the aggregation queries.When and how to fuse clauses relies on the planning technique introduced in Section 4.2.1.We only consider the cost of calling the LLM, as this dominates the overall cost per query. Assuming the length of each row in the unstructured data is more or less uniform across rows, then the cost is proportional to the number of rows that fed to the LLM, which we use as the surrogate for estimation.• SELECT maps each row, hence the cost is |T | for the table T fed to SELECT.• GROUP BY consists of two steps, where taxonomy construction consumes a subset of the input tableT , and classification runs |T | LLM calls in parallel. • WHERE depends on the proposal p. In practice we set a budget B and try to minimize the variance of unbiased estimator or maximize the recall in online learning, as explained in Section 4.1. • Whenever clauses are fused together, each implementation is responsible for providing a reasonable cost estimate. For example when SELECT and GROUP BY are fused, the estimated cost is the same as GROUP BY alone, as the classification stage of GROUP BY shares the input tokens with SELECT.The last step of compilation is to generate the machine specific code (e.g., x86 assembly code) from the intermediate representations (IR). For UQE, this is the process of generating the LLM-specific prompts. For example, when GPT is deployed as the "machine", a system prompt like "You are a helpful assistant" will be added to the queries. This step also sets the correct context (e.g., the correct structured/unstructured column to associate to, the description of the databases) for the LLM. When such information is not available, one can also leverage the LLM to provide a good suggestion. 
related work: 
 While the unstructured data analytics engine is relatively new, there are several related works in the context of unstructured data query and analysis. Approaches like pattern or regexp matching [20] is scalable but not feasible for complex semantic reasoning. RAG [24, 18] based approaches rely on the retrieval quality and is not directly suitable for aggregation queries over entire database.LLMs [4, 2, 33] depict the ability of table analytics [15] to some extent [9, 25] , but are still not reliable for large unstructured database analytics yet.Our work is closely related to neural symboilic approaches for unstructured data analytics. Early attempts in this line aim to design specialized neural architectures with inductive biases (e.g., attention) to capture a particular form of operation (e.g., filtering a list of objects based on a natural language predicate by their attention scores) [43, 29, 3] . Those differentiable neural "operators" can then be chained together to model more compositional queries, and trained end-to-end using gradient descent. Another direction, in line with our work, is to augment symbolic programs with learnable operators parameterized by neural networks [10] . Those programs are often modeled as discrete latent variables, which can be hard to optimize. In contrast, UQE leverages predictions from LLMs as supervision to train an efficient proxy query model in an online fashion. Similar to UQE, some recent work [11, 38] also adopts LLMs as fuzzy query operators. However, the generated programs treat LLMs as an UDF in a SQL program, which can be very expensive to execute on large databases. Our UQE implements similar but augmented semantics with the focus on the cost efficiency and scalability. Liu et al. [26] optimizes a similar query engine from the system perspective like cache optimization and deduplication, while our work mainly considers algorithmic improvements and is considered as an approximate query engine [28] . These system and algorithm optimizations are actually orthogonal and can be beneficial to jointly consider both for future works.In a distantly related topic, text2SQL [47, 46, 21, 35 ] also leverages models talking to databases, but is mainly for semantic parsing purpose. While it also leverages the advances in LLMs [44, 16, 31, 37] , the execution is still on pure SQL and thus is not suitable for unstructured databases. There are also works on leveraging formal query languages to better query LLMs [34, 6] , with the focus on controllability of the LLM itself rather than performing analytics on external unstructured data. 
experiments: 
 We benchmark the accuracy and incurred cost of UQE on multimodal unstructured data analytics tasks, with the goal to show and understand when and why UQE can improve accuracy while keeping the cost low. Since the unstructured database analytics is a relatively new task, we construct and compare against several baseline approaches, on a set of tasks created from existing datasets. Baselines: We design the following baselines for comparison• lc-LLM denotes the long-context LLMs that can directly take a subset of database and a natural language question as input, and produce the desired analysis. We mainly evaluate against several model families, including GPT-4 [2] and Claude-3 [4] . Of course, when evaluating the lc-LLM based approaches, we use the natural language instead of UQL as the prompt. • RAG-based can be applied to some non-aggregation queries, such as semantic retrieval. For the retrieval part we use max inner-product search (MIPS) on top of the same embeddings that are used by UQE, for a controlled experiment. • UDF simply treats the LLM calls as User-defined function of an SQL engine, with the same budget as UQE by default. This approach will not have the advanced sampling / search algorithm as used in UQE, which also serves as an ablation for the effectiveness of our UQE.We evaluate different approaches on common analytical tasks in three widely used application domains. We use the datasets that were previously created for discriminative tasks, as these datasets contain both the unstructured columns and the structured ones (the labels in the corresponding dataset). We then hide these structured label columns and perform analytical tasks on the unstructured columns, where these hidden structured columns will be used to compute the ground-truth. The text based tasks include IMDB [27] movie reviews, customer service dialogs including Action-Based Conversations Dataset (ABCD [8] ) and AirDialog [41] , and image based Clevr [23] dataset. Please refer to Appendix C.1 for more information.Setup: We use voyage-2 [1] to embed the text-based unstructured columns, and Vertex [40] for multimodal embeddings. For budget constraint queries, we allow different approachces to access at most 128 rows in the database by default. We run queries on different datasets by instantiating the template shown in each of the sections below.The exact natural language and UQL queries can be found in Appendix D and more information including statistics of conditions we used for query and hyperparameters (for UQE we simply use the default hyperparameters for sampling and online learning) can be found in Appendix C.This task provides aggregated statistics over databases with specified conditions, with the template as:SELECT COUNT(*) FROM {table} WHERE {"satisfies natural language specified condition"}We report the relative estimation error (i.e., |predict-true_count|/true_count) and its standard deviation in Table 1 . For lc-LLM baselines we estimate the count based on groups of unbiased data samples that fed into the prompt.For text based aggregation we use claude-3-haiku as the backbone model, where UQE deploys 10× reduction in relative errors while reducing the cost by a factor of 20× or more. For the image dataset, since only limited set of LLMs are capable right now, we use gpt-4o as the backbone, and compare with lc-LLM baselines. Thanks to the improved sampling method in UQE, the same gpt-4o consistently achieves improved performance out-of-the-box. To verify this, we feed one image at a time to gpt-4o and manually aggregate the count, the estimation error would be 17.10% ± 13.95 and 19.35% ± 13.81% for the two queries of Clevr, which is twice higher than UQE in the worst case.This task filters rows in databases that satisfy specified conditions, with the template as:SELECT * FROM {table} WHERE {"satisfies natural language specified condition"} LIMIT BWhile we limit the output size to be B = 256 to keep the total cost within a reasonable budget. The challenging scenarios are when the number of rows that satisfy the predicate is few (i.e., "rare event finding"). Table 2 shows similar sets of comparison, but the metric is F1 score which evaluates the quality of SELECT-ed rows. Overall UQE (with claude-3-haiku as backbone LLM) consistently achieves comparable or better performance than the baseline methods. MIPS which uses the same embedding of unstructured data as UQE, has high variance across different types of queries. The queries such as "dialogs with account access issues" would be very suitable for MIPS as the embedding similarity is able to capture that well. For queries involving reasoning (e.g., find the images with less than 4 objects), it is pretty hard for pretrained embeddings to express this.This task abstracts the intrinsics of each row, and then performs semantics-based GROUP BY, grouping the common intrinsics across all rows. Finally, it provides aggregated statistics over each group:SELECT derived_attribute, COUNT(*) FROM {table} GROUP BY {"extract an abstract intrinsic attribute specified in natural language"} AS derived_attribute LIMIT 10 The challenging problems in this task are (i) building a taxonomy with good coverage, and (2) bias and variance reduction for groups with small population. The result of this query is a list of tuples of derived attributes and their number of occurrences in the dataset. We use the earth mover's distance (EMD [30] ) as the evaluation metric to compare the extracted tuples and ground-truth tuples.The distance between a pair of attributes is defined by one minus the cosine similarity of their text embeddings. We can see from Table 3 that UQE consistently outperforms baselines while achieving much lower cost. We also show in Appendix C with more qualitative results comparisons.We study the effectiveness of UQE for aggregation queries in Section 6.2.1 and non-aggregation queries in Section 6.2.2, and the quality/cost trade-off of UQE in Section 6.2.3. In appendix we provide more results on other modalities C.3, consistency C.4 and latency C.5.To decouple the variance introduced by the algorithm and the bias introduced by the LLM based predictors, here we use the ground-truth label as the predictive result and focus on the effectiveness of variance reduction. Figure 4 shows the box plot of different sampling methods. We can see using stratified sampling over the embeddings of unstructured content achieves significant lower variance compared to the uniform random sampling. Also both of these achieve similar expected values, which also justifies the correctness or unbiasedness.We show the effectiveness of the online learning in terms of the recall as a function of the iteration steps in Figure 5 . Compared to the dashed line in the figure which indicates the results of uniform random sampling, the online learning can achieve significant boost in terms of the recall. While for some queries the variance at early iterations can be high, these all converge well in the end.Generally the larger compute budget B the better quality UQE will get, and we verify this in Table 4 5. UQE can achieve pretty good quality even with very low budget, and notably compared to the baseline, it achieves similar quality with 16x reduction of the compute needed. We show more results in Section C.5 regarding the compute efficiency.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided the information in the appendix and main paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 This paper proposed an unstructured query engine that leverages 1) the flexibility of LLMs for data understanding; 2) the advances in sampling and online learning for efficient data scanning; 3) and the compiler that bridges these algorithmic workflows with LLMs. We demonstrated its efficiency and accuracy over three analytic tasks on four datasets with two different modalities. However the current work is still very limited in terms of 1) the semantics it lacks, including table join and other types of aggregations; 2) an automated selection of LLMs and sampling configurations; 3) and scaling to even larger databases. We hope to investigate these further in future works.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We provided the variance of the results in almost all tables when applicable.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: No risk as far as we can see.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have properly cited the models (GPT4, Claude3) we used in the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Proposition: The optimal proposal distribution p that minimizes the variance of estimation in Eq 2 is p i ∝ f (T i , cond). The variance gets 0 with this proposal.Let's simplify the notation a bit and use f (x) for f (T i , cond) and prove the variance reduction in general cases for binary function f . For the simplicity let's omit the constant |T | and focus on the estimation of the expectation term. If we come up with a new proposal distribution p : T → [0, 1] where x∈T p(x) = 1, then we get a new estimator in the following form:EQUATIONWe hope this new estimator would have lower variance. Let's define u(x) = q(x) p(x) f (x) for the ease of notation, and look at its variance first:EQUATIONSince our estimator is unbiased,E 2 p [u(x)] = E 2 q [f (x)] and thus has nothing to do with p, let's focus on minimizing the first term. More specifically we have the optimization as:EQUATIONLet:EQUATIONand we can find the saddle point of min p max λxλ2 (L(p, {λ x }, λ 2 )) using K.K.T condition.     -(q(x)f (x)) 2 p(x) 2 -λ x + λ 2 = 0, ∀x ∈ T λ x p(x) = 0, p(x) ⩾ (0), ∀x λ 2 x (p(x) -1) = 0, x p(x) = 1and we can get the optimal solution of pEQUATIONand put it back into Eq(1) we can see the optimal variance would beEQUATIONwhich means one sample would be good enough! In our context, q(x) is usually just a constant (e.g., q(x) = 1 T for the [Case Count]), and f (x) ∈ {0, 1}. In this case, a simplified optimal proposal would be: EQUATIONWe evaluate different approaches on common analytical tasks in three widely used application domains. We use the datasets that were previously created for discriminative tasks, as these datasets contain both the unstructured columns and the structured ones (the labels in the corresponding dataset). We then hiden these structured label columns and perform analytical tasks on the unstructured ones, where these hidden structured columns will be used to compute the groundtruth.• User review mining. We use IMDB [27] for semantic analysis of user sentiment. The entire dataset contains 50K highly polar movie reviews with positive and negative sentiment labels. • Goal-oriented (customer service) dialogue systems. We use two datasets for this category, including 1) Action-Based Conversations Dataset (ABCD [8] ) for intent identification, which has 10,042 dialogs with 10 distinct user intents requiring unique sequences of actions constrained by policies to achieve task success; and 2) AirDialog [41] for conversation outcome understanding. It contains 402,037 goal-oriented conversation on flight booking with 5 possible ground-truth states. • Image understanding. We use Clevr [23] dataset for multimodal data understanding and retrieving.Specifically we use the val split of the dataset, which contains 15,000 images of objects containing different number of cylinders, cubes and spheres with different sizes/colors. We down-sample the image to size no more than 128 × 128, and only feed the images to the LLMs while holding out scene metadata for evaluation only.Also see Table 6 for the detailed statistics of different query conditions. Rare events like cancel in AirDialog is typically challenging to find and aggregate on.For the embeddings, we use the voyage-2 for text and for images, we use Google Vertex API with dimensionality of 512. We preprocess the embeddings for all the datasets and keep them static during the queries.For the aggregation queries, we use faiss2 to cluster the embeddings into 10 groups, and perform stratified sampling on top.For the online learning setting for non-aggregation queries, in our experiment we simply set ϵ t,i to be 0. We start training the function g when we collect at least one possible and one negative example labeled by the LLM. Then after every minibatch of samples collected, we train g via linear logistic regression and simply leverage sklearn for that.Other parameters that might matter include: the sampling budget B for aggregation queries is 128 and for non-aggregation queries it is 256. For group-by queries the UQE needs a step in building the taxonomy, where the budget we use for that is 16. We set these parameters based on educated guess and keep them as default across all the queries over all the datasets.We use the audio MNIST [5] data for this experiment. This dataset contains 30k wav files from 60 speakers pronouncing digits 0-9. We perform the audio semantic retrieval experiments. The query is first converted to audio space using TTS and the corresponding audio embedding is used for MIPS search. We use Gemini Pro 1.5 as the backend model, as it supports the audio inputs nicely. As is shown in Table 7 , the proposed UQE consistently does better than alternatives, and it also allows complex queries that require reasoning, while embedding based MIPS is limited to certain types of queries.While UQE is able to leverage the foundation models to analyze the unstructured databases directly, the execution result is not deterministic due to the nature of the stochasticity of the algorithm and the LLMs themselves. In our paper we aim at reducing the variance so as to improve the consistency. However when the backend LLM gets updated, it might also cause the potential inconsistency. Below we analyze the effect of different LLM backends.Table 8 shows the effect on the IMDB dataset, where we see little variation. Actually the difference caused by the model switching is much lower than using a worse query engine. Of course, this behavior shift would be task related, but with the advances of LLMs we believe this variation would converge and be more stable to different prompts in the future.We report the runtime of UQE with claude-3-haiku as backbone, and lc-gpt-4-turbo as the baseline method in Table 9 . We can see UQE achieves low latency in aggregation operations, but higher latency in retrieval. This is due to the online update and re-evaluation of the g function described in Section 4.1.2. The experiments were run on MacBook Pro CPU, so we expect this bottleneck would be alleviated with better engineered system, which we will focus in our future works.For single_item_query in ABCD, the items mentioned in the dialogs found by UQE is: Which is very close to the ground truth (recover_username, reset_2fa, recover_password).[ [ ' b o o t s ' ' 3 3 8 ' ] [ ' j a c k e tFor airdialog, the outcomes found by UQE is Where the ground truth has one more additional outcome (cancel). But since the percentage of cancellation is very small, it is expected that this might be missing from the group by abstraction when number of occurs are very limited. The following dialogs between a customer service agent and a customer. Dialogs start by headers such as **Dialog 1**, **Dialog 2**, and so on. Your task is to classify whether the dialog content is about the theme "account access issue". Then count how many dialogs are talking about this theme. In the end, output the count as a single number.Here is more detailed explanation about Theme "<THEME>". Be sure to use this information when you classify.Theme "<THEME>" dialogs content is about THEME_EXPLANATION. Please perform thorough analysis for each of the dialog. In the end, please **only** output a single number, which is [the total number of dialogs] that talks about the theme "account access issue". <THEME> = account access issue <THEME> = requesting detailed specifications of a certain item sold on the website <THEME_EXPLANATION> = the customer could not access the account and was locked out, such as couldn t recall their username, couldn't perform two-factor authentication, or forgot their password and couldn't access their account <THEME_EXPLANATION> = the customer needed help with the detailed specifications of a certain item sold on the website, such as inquiries about detailed information of a specific retail item about materials, whether it shrinks, stock availability, etc., but NOT inquiries about promotions, order status, shipping status, questions about how to use the website to purchase an item, or difficulties on using the website to add to cart or purchase an item, or inquiries about subscriptionBelow are the dialogs: **Dialog 0**:[agent]: Hello, how can i help you today [customer]: Hello my name is Alessandro Phoenix and I need to make sure the shipping cost is included on my order ...The following are dialogs between a airline ticketing agent and a customer. Dialogs start by headers such as **Dialog 1**, **Dialog 2**, and so on. The outcome of the dialog will be one of the following 5 categories:[book]: the agent has booked a flight for the customer (not including the flight change);[cancel]: the agent canceled the existing valid reservation for the customer;[no_reservation]: the customer wants to change or cancel the flight 1037 but there is no valid reservation under this customer;[no_flight]: the customer aims to book a flight from departure to destination but finds no flights between departure and destination; Your task is to count how many dialogs have outcome <OUTCOME>. Please **only** output a single number, which is [the total number of dialogs] that satisfied the above requirements. The following are dialogs between a customer service agent and a customer. Dialogs start by headers such as **dialog_1**, **dialog_2**, and so on. Your task is to find out the dialogs where <CONDITION>. Please only output a list of indices in the format of [dialog_3, dialog_7, ...] <CONDITION> = the customer could not access the account and was locked out, such as couldn t recall their username, couldn't perform two-factor authentication, or forgot their password and couldn't access their account <CONDITION> = the customer needed help with the detailed specifications of a certain item sold on the website, such as inquiries about detailed information of a specific retail item about materials, whether it shrinks, stock availability, etc., but NOT inquiries about promotions, order status, shipping status, questions about how to use the website to purchase an item, or difficulties on using the website to add to cart or purchase an item, or inquiries about subscription The following are dialogs between a customer service agent and a customer. Dialogs start with headers such as **dialog_1**, **dialog_2**, and so on. Your task is to analyze all the dialogs, and summarize "<ABSTRACT_ATTRIBUTE>" into groups. Please output the table of your analysis, in the format of pairs of ("<ABSTRACT_ATTRIBUTE>", number_of_dialogs belong to that). Specifically in the format as: group 1,number_of_dialogs group 2,number_of_dialogs ... The following are dialogs between a airline ticketing agent and a customer. Dialogs start with headers such as **dialog_1**, **dialog_2**, and so on. Your task is to analyze all the dialogs, and summarize "<ABSTRACT_ATTRIBUTE>" into groups. Please output the table of your analysis, in the format of pairs of ("<ABSTRACT_ATTRIBUTE>", number_of_dialogs belong to that). Specifically in the format as: group 1,number_of_dialogs group 2,number_of_dialogs ... Read the following customer support dialog between an agent and a customer, and only reply <True> if <WHERE_CLAUSE>, or <False> otherwise. <WHERE_CLAUSE> = the customer could not access the account and was locked out, such as couldn t recall their username, couldn't perform two-factor authentication, or forgot their password and couldn't access their account <WHERE_CLAUSE> = the customer needed help with the detailed specifications of a certain item sold on the website, such as inquiries about detailed information of a specific retail item about materials, whether it shrinks, stock availability, etc., but NOT inquiries about promotions, order status, shipping status, questions about how to use the website to purchase an item, or difficulties on using the website to add to cart or purchase an item, or inquiries about subscription[Dialog]: [agent]: Hi! Thank you for contacting us today. How can I help you? [customer]: I'm pretty upset that a jacket that I ordered is now saying that it is out of stock. Do you know when it will be back in stock?[agent]: I am so sorry that happened to you [agent]: Yes, let me look in to that for you [action]: Searching the FAQ pages ... ...Read the following airline ticketing dialog between the customer and the agent, and only reply <True> if <WHERE_CLAUSE>, or <False> otherwise. <WHERE_CLAUSE> = the agent has booked a flight for the customer (not including the flight change) <WHERE_CLAUSE> = the agent canceled the existing valid reservation for the customer <WHERE_CLAUSE> = the customer aims to book a flight from departure to destination but finds no flights between departure and destination <WHERE_CLAUSE> = the customer wants to change or cancel the flight but there is no valid reservation under this customer[Dialog]: customer: Hello. agent: Hello, how can I help you? customer: Please book a flight ticket from CLT to DEN. agent: Sure, let me know your travelling dates. ...Read the following image, and only reply <True> if <WHERE_CLAUSE>, or <False> otherwise. The following are dialogs between a customer service agent and a customer. Dialogs start with headers such as **dialog_1**, **dialog_2**, and so on. Your task is to analyze all the dialogs, and summarize "<ABSTRACT_ATTRIBUTE>" into groups. Please output the table of your analysis, in the format of pairs of ("<ABSTRACT_ATTRIBUTE>", number_of_dialogs belong to that). Specifically in the format as: Justification: We provided thorough experiments and explanations of the algorithms to jusify the claim.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/facebookresearch/faiss 


Decision: The paper presents a system that can be used to conduct data analytics on unstructured data. The system uses, of course, LLMs :) The system is tested and thoroughly assessed.

Reviewers have given average scores but in general we are all favorable for this paper to appear as poster at NeurIPS.

Reviews: review: paper_summary: paper_summary: This paper proposes an unstructured query language based on a small segment of SQL. Its key feature is that it further supports unstructured texts and images because the authors assume that LLMs work on intra-row texts. During the query, the authors propose using online learning on LLM outputs over batches to improve the sample quality in the upcoming batches. Empirical results convey important information; that is, feeding longer context into larger LLMs does not work as well as using smaller LLMs in a more organized way.
strength_weakness: strength_weakness: - This paper is more like a technical report or a guideline for building certain products. The overall product presented is simplified from the introduction's proposed question. Realizing this system under the strong assumption that LLMs work on intra-row semantic understanding is not too difficult: the virtual column, although it seems to be filled by LLMs, can be realized in several prompts, such as closed-domain classification for WHERE and open-domain classification for GROUPBY.
questions: questions: - Given that the output in the virtual column will work with other parts of the SQL, what is the failure rate in output formatting in the experiments?
limitations: limitations: Yes,
paper_summary: paper_summary: This paper proposes an ambitious new framework for analytics on unstructured databases, the Universal Query Engine (UQE). The authors first present a list of semantics and clauses for querying unstructured databases and propose methods for implementing these functionalities, including indexing and compiling. Experiments on multimodal unstructured data analytics tasks show that UQE outperforms other methods, demonstrating the promise of utilizing UQE in database analytics.
strength_weakness: strength_weakness: - Could you elaborate more on the difference between BINDER and this approach (line 39)? Is it due to the lack of indexing and compatibility with unstructured information?
- Similar to the first point, the baselines seem a bit weak. Why have the authors not used BINDER as their baseline?
- I want to hear the authors' opinions on how advancements in LLMs and vision models might affect the guarantee that databases consistently return the same results. Traditional database engines, which are widely used, provide this consistency over time. Could these technological advancements affect the reliability of database analytics and, subsequently, compromise their reliability?
questions: questions: I have listed my questions in the Weaknesses section.
limitations: limitations: The limitations of the work are properly stated.
paper_summary: paper_summary: This paper propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections.
Further, a Universal Query Language (UQL) is proposed as a dialect of SQL that provides full natural language flexibility in specifying conditions and operators.
UQE leverages the ability of LLMs to conduct analysis of unstructured data and achieve efficient and accurate query execution.
strength_weakness: strength_weakness: 1. Many details about the design and implementation of UQE are missing. For example, the description of the generation of machine-specific code (assembly instructions) during the compilation stage (L251) is not clear. Please clarify the details of this strategy. Additionally, the authors claim that calling the LLM dominates the overall cost per query (L239). Providing detailed information about the LLM-calling stage would be beneficial.
2. The indexing technique introduced by UQE is related to the sampling strategy based on specific queries. However, database indexing is traditionally meant to facilitate quick searches. If each query requires sampling, will this impact efficiency? Additionally, doesn't this compromise accuracy?
3. This paper primarily addresses unstructured datasets, including video and audio. However, the experiments only involve text and images, lacking validation for broader generalizability. 
4. Furthermore, there are plenty of multi-modal vector databases, yet the authors only compare the RAG-based query. Can UQE outperform recent multi-modal query databases, such as Milvus and MillenniumDB? If so, demonstrating this would make the evaluation more convincing.
5. Does the query support join operations, such as joining across multiple different modal files?
6. It's hard to validate the results without supplying code/models.
questions: questions: Please see the weakness section.
limitations: limitations: No discussion on limitations, I suggest the authors to add one.
paper_summary: paper_summary: This paper proposed a Universal Query Engine to directly draw insights from unstructured data. For aggregation queries, this paper designed an unbiased sampling algorithm to solve the problem that it’s hard to apply index on virtual columns. Inspired by the workflow of the C++ compiler, this paper also proposed a workflow of optimizing and generating prompt to achieve more efficient query.
strength_weakness: strength_weakness: 1. UQE seems to lack scalability on the Semantic Retrieval task. The latency reported in Table 5 is as high as 38.08 seconds. Also, the paper didn’t state that whether these numbers are the total latency or average latency per query. If they are latency per query, using UQE on the Semantic Retrieval task still has a far distance from practice.
2. The cost of embedding operation of Algorithm 1 and 2, and the cost of Algorithm 2 are not discussed in the paper. Although the authors wrote in the appendix that the g function in Algorithm 2 caused the high latency problem, no quantitative analysis was provided.
3. Lack of comparison with important baselines. I noticed that the references [9] and [33] mentioned in related work are not included in the baselines. Using LLM as UDF may significantly slow down the queries, however, they may have better accuracy on aggregation queries. Thus, I think it’s necessary to include them in the baselines to further evaluate the performance of the trade-off made by Algorithm 1 between accuracy and latency. 
4. Lack of evaluation of UQE’s time cost in the main text of the paper. I noticed those results are reported in the appendix. However, in the database area, the latency and cost are significant metrics that should be reported in the main text of the paper.
questions: questions: I noticed that lc-gpt-4-turbo also has quite a high latency on Semantic Retrieval tasks, is it possible to do more optimization on the model itself to improve the performance to make such a method practice?
limitations: limitations: Yes, the authors have mentioned the limitations of this work, but I suggest to provide more comprehensive evaluations, detailed discussions about UQE’s scalability and cost. After all, these are critical problems about the practicality.

Comments: title: last reminder on adding your thoughts on NeurIPS authors' feedback
thank you!
We'd love to hear your opinion on the rebuttal
We'd love to hear your opinion on the rebuttal
We'd love to hear your opinion on the rebuttal
comment: Thanks for your quick response! Most of my concerns are resolved now. I am increasing my score to 5.
Dear reviewers,

We sincerely appreciate your effort during the review process. As the deadline for reviewer-author discussion is approaching in several hours, we would like to send out a last reminder to see if you have any quick follow up questions to our rebuttal, so that we might still be able to respond.

Thank you so much for your understanding!

Best,
Authors
Thank you reviewer 6SvA for your kind reply and your recognition of the paper!
Dear reviewer uiEW,

We have provided additional experiments and explanations to address your valuable feedback.

Could you please kindly take a look, and let us know if you have further concerns so that we can provide further clarifications if needed?

Thank you so much!

Best,
Dear reviewer E5QU,

We have provided additional experiments and explanations to address your valuable feedback.

Could you please kindly take a look, and let us know if you have further concerns so that we can provide further clarifications if needed?

Thank you so much!

Best,
Dear reviewer rs84,

We have provided additional experiments and explanations to address your valuable feedback. 

Could you please kindly take a look, and let us know if you have further concerns so that we can provide further clarifications if needed? 

Thank you so much!

Best,
Thanks for the response. It cleared up my concerns.