Introducing "Forecast Utterance" for Conversational Data Science
Authors: Md. Mahadi Hassan
Alex Knipper
Shubhra Kanti Karmaker Santu
Venue: Accepted by TMLR ()

Abstract: Envision an intelligent agent capable of assisting users in conducting forecasting tasks through intuitive, natural conversations, without requiring in-depth knowledge of the underlying machine learning (ML) processes. A significant challenge for the agent in this endeavor is to accurately comprehend the user's prediction goals and, consequently, formulate precise ML tasks. In this paper, we take a pioneering step towards this ambitious goal by introducing a new concept called Forecast Utterance and then focus on the automatic and accurate interpretation of users' prediction goals from these utterances. Specifically, we frame the task as a slot-filling problem, where each slot corresponds to a specific aspect of the goal prediction task. We then employ two zero-shot methods for solving the slot-filling task, namely: 1) Entity Extraction (EE), and 2) Question-Answering (QA) techniques. Our experiments, evaluated with three meticulously crafted data sets, validate the viability of our ambitious goal and demonstrate the effectiveness of both EE and QA techniques in interpreting Forecast Utterances.

Published in Transactions on Machine Learning Research (03/2024)
Introducing "Forecast Utterance" for Conversational Data Science
Md Mahadi Hassan"
mzh0167@ auburn edu
Department of Computer Science and Software Engineering
Auburn University
Alex Knipper*
ruk0035 @ wwwurnncccmmmmom edu
Department of Computer Science and Software Engineering Auburn University
Shubhra Kanti Karmaker (Santu)
sks0086 wwwuurn.cccmmmm edu
Department of Computer Science and Software Engineering
Auburn University
Reviewed on OpenReview: htterr/pophaaeeewwww.r/e//tttt22
Abstract
Envision an intelligent agent capable of assisting users in conducting forecasting tasks through intuitive, natural conversations without requiring in-depth knowledge of the underlying machine learning (ML) processes. A significant challenge for the agent in this endeavor is accurately comprehend the user's prediction goals and, consequently, formulate precise ML tasks. In this paper, we take pioneering step towards this ambitious goal by introducing new concept called Forecast Utterance and then focus on the automatic and accurate interpretation of users' prediction goals from these utterances. Specifically, we frame the task as slot-filling problem, where each slot corresponds to a specific aspect of the goal prediction task. We then employ two methods based on self-supervision with synthetic examples for solving the slot-filling task, namely: 1) Entity Extraction (EE), and 2) Question-Answeriin (QA) techniques. Our experiments, evaluated with three meticulously crafted data sets, validate the viability of our ambitious goal and demonstrate the effectiveness of both EE and QA techniques in interpreting Forecast Utterances.
Introduction
Imagine a conversational AI agent designed to assist end-users, who are not experts in machine learning, with simple forecasting tasks such as estimating a publicly traded stock's future price or predicting the average temperature of geographic location in the upcoming week. One critical challenge for such agent is accurately understand the user's forecasting goals and formulate precise machine learning task accordingly. As these conversations are expected to happen in real-time, and each user may have unique data sets and data science needs (Parsons et al. (011), it is unrealistic to assume that any training data is available to pre-train these conversational agents, making the supervised learning paradigm impractical. As such, the central question we investigate in this paper is how to automatically and accurately understand users' forecasting goals from their utterances in an unsupervised fashion.
Motivation:
Time series forecasting an essential tool for informed decision decision-making and strategic planning across various organizations. It offers valuable insights into future business developments, including revenue, sales, and resource demand. However, small businesses and machine learning enthusiasts from diverse backgrounds often face challenges in harnessing the benefits of time series forecasting due to several factors: 1) Limited ML-related technical expertise among entrepreneurs and business owners; 2) A lack of highly competent data science teams small businesses; and 3) The high cost of hiring external consultants and data privacy concerns when involving third
'Md Mahadi Hassan and Alex Knipper contributed equally to this work.

---

Published in Transactions on Machine Learning Research (03/2024)
parties. In this context, conversational approach that simplifies the application of time series forecasting would be highly appealing. Furthermore, time series forecasting is primarily self- self-supervised in nature as it relies mostly on historical data without requiring additional human labels, making it an ideal candidate for formulating ML tasks in a self-supervised fashion. With the growing popularity of AI chatbots, the proposed "Conversational Data Science" approach can significantly broaden ML's reach and impact across industries.
The Conceptual Leap:
Existing MI. solutions demand clear understanding of MI. techniques on the user's end as well requires significant manual effort for executing the end-to-end Data Science Pipeline, making Data Science inaccessible to the general public Karmaker al. (2021). However, Conversational AI research holds the potential to democratize data science for a broader audience. This paper connects machine learning pipeline automation with Conversational AI, creating a novel solution for forecasting task formulation by integrating simple yet powerful methods from both disciplines.
In real-time conversations, end users will provide their dataset "on the fly" and try to formulate a forecasting task on the provided dataset. This makes it impossible to pre-train an ML task formulation model, as the attribute set is different each time. To address this challenge, we frame as an unsupervised slot-filling problem, where each aspect of the prediction goal represents slot. We then implement two approaches that leverage self-supervision with synthetic examples to populate the slots: 1) Entity Extraction (EE) and 2) Question-Answering (QA) techniques.
Challenges:
Contributions: This paper takes an initial step towards formulating prediction tasks through natural conversation by focusing a specific type of user utterance called a Forecast Utterance. We design and develop techniques for automated understanding of users' forecasting goals from these utterances in an unsupervised fashion. The main contributions of this paper include the following:
Introducing the Forecast Utterance, an expression of users' prediction needs and goals via natural language, and creating three benchmark datasets, through extensive manual effort, to promote future research this area.
Framing Forecast Utterance Understanding as an unsupervised slot-filling problem, where every prediction need represented as a distinct slot. To address this task of slot-filling, our approach involves the application of Entity Extraction (EE) and Question-Answering (QA) strategies. Both strategies utilize self- self-suuervii by generating synthetic examples.
Conducting case studies with three real-world datasets to demonstrate the feasibility and efficacy of the proposed ideas and techniques.
2 Related Works
Over the past decade, the machine learning community has made significant advancements in automating machine learning pipelines. Despite these developments, automating Prediction Task Formulation remains a challenge due to its human-centric nature (Karmaker et al., 2021 Sarkar al. (023). In parallel, NI.P researchers have made significant progress in domains such as Dialog Systems (Weizenbaum) 1966). Slot Filling (Lafferty et al. 220 Zero-Shot Learning (Palatucci et al. 2009; Sarkar et al. 2022), E-commerce search (Santu et al. 2017) and many more, to create more sophisticated conversational agents. For example, Dialog Systems research has evolved through Conversation Topic Prediction, Dialogue State Tracking, and open-domain dialogue system innovations, with Large Language Models (LLMs) like ChatGPT being among the latest developments. On the other hand, the Slot Filling problem has been addressed as sequence labeling task by leveraging CRFs (Lafferty et al., 20016), RNNs (Williams and Zipser 1989). and self-attention transformers (Vaswani et al., (017). while Self-Suuervised Learning (Akata et al. (2013) has primarily focused on recognizing unseen labels and has become a very popular paradigm in ML research recently. In this section, we will delve deeper into these areas and discuss their relevance our research.
Dialog Systems: In Dialog Systems research, significant progress has been achieved through advancements in Conversation Topic Prediction (Khatri et al.) 2018) and Dialogue State Tracking (DST) (Henderson et al.) 2014 4a b
DST improvements involve a range of approaches, including schema guidance for better structure (Chen et al. 2020
Zhu et al. 2020 Kapelonis et al. 2022), recursive inference for deeper understanding (Liao et al. 2021). generalization
and value normalization for more adaptability (Williams 2013; Wang et al., 2020). zero-shot transfer lcarning for
2

---

Published in Transactions on Machine Learning Research (03/2024)
data efficiency (Campagna et al. 2020; Rabinovich et al. 2022). , and attention modulation for improved focus during
inference (Veron et al. 2022). Open-domain dialogue systems have also seen significant advancements. GODEL's (Peng)
ct al. 2022) grounded pre-training adapts to diverse downstream tasks, FusedChat (Young et al. 2022) combines task-oriented and open-domain dialogue for natural conversations, & ChatGPT further enhances conversational agent performance across various applications such as data science (Hassan et al. 2023).
Slot Filling:
Slot Filling has been studied across applications like recommender systems and chathots, using approaches
such as RNNs (Kurata et al.) 2016 Mesnil et al. 2015), integrating CRFs with RNNs (Huang et al., 2015 Reimers
and Gurevych/ 2017 Jbene et al.
2022). and self-attention mechanisms for sequence labeling (Shen et 2018 Tan
et
2018 Zhao ct al. 2022). Joint learning of intent detection and slot-filling has been explored (Liu and Lane
2016 Goo al. 2018 Zhang al.
2019 Chen and Luo (023), incorporating few-shot learning and model-agnostic
meta- meta-learning (Bhathiya and Thayasivam, 2020 Krone et al. 2020). Transfer learning has led to zero-shot slot filling
approaches (Mehri and Eskenazi 2021 Wang et al. 2021 Larson and Leach 2022), enhancing knowledge transfer
between pre-trained models and target domains, improving performance on unseen slots and achieving state-of-the-art
results.
Sell-Supervised Learning with Synthetic Examples: Recent self-supervised learning research explores synthetic data to improve model robustness and adaptability. Leveraging synthetic datasets enhances contrastive learning, leading to more generalized feature representations (She and Xu (02T). Additionally, the critical role of advanced data augmentation in disentangling content from style and boosting out-of-domain robustness has been emphasized (von) Kügelgen et al. 2021 Ng et al., 2020). Notably, the potential of synthetic data to empower neural machine translation, particularly for resource-scare languages, highlights its growing influence (Ruiter al., 2021 These advancements solidify the pivotal role of synthetic and augmented data in propelling the frontiers of self-supervised learning. Again,
recent research showcases time series forecasting using meta-learning frameworks (Oreshkin et al.)
2021 Abdallah
al. 2022 Van Ness et al. 2023). feature construction for time series Bariables(Wang et al. 2017). layer ensemble
architecture Rahman et al. 2014). data augmentation & adversarial domain adaptation (Hu et al. 2020 Jin et al. 2022).
and ordinal regression recurrent neural networks (Orozco and Roberts 2020). Also researcher has tried to recommend prediciton task to users interactively (Xu al.) 2019). However, these approaches don't otn't address Forecast Utterance Understanding for unseen datasets in real-time, which is this paper's primary focus.
Difference from Previous Work:
Our work distinguishes itself from previous works by introducing novel concept called "Forecast Utterance" and demonstrating the feasibility of so-called "Conversational Data Science". In contrast to existing Dialog Systems and Slot-Filling research, primary focus is on understanding end-user's forecasting needs by framing it as slot-filling task where each slot represents a unique aspect of the goal prediction task. Additionally, we propose a novel synthetic data generation technique to fine-tune pre-trained transformer based models for Entity Extraction (BE) and Question Answering (QA) tasks to perform inference in real-time.
3 Problem Definition
We treat the task as a slot-filling problem, with each aspect of the user's prediction need as a slot. To achieve this, we require an expression language capable of translating abstract goals into a slot-value style format. We introduce the "Prediction Task Expression Dictionary" (PeTED), consisting of slot/value pairs define the forecasting task's objectives and constraints.
3.1 Prediction Task Expression Dictionary
Assuming user provides a relational database schema with tables and relations, can simplify this by treating all tables, whether they define an entity relation, as single joined "giant" table containing all entities and attributes. Figure demonstrates an example schema.
For the purposes of this work, we employ four slots in the PeTED expressions: "Target Attribute", "Aggregation Constraint", "Filter Atrrbute"", and "Filtering Constraint" which can be easily extended an arbitrary number of slots in the future. The values for "Target Attribute" and "Filter Attribute" can be any attribute from the schema, while "Filtering "onstraint" captures constraints such as equal or greater than. Meanwhile, "Aggregation Constraint"
3

---

Published in Transactions on Machine Learning Research (03/2024)
Timestamp
Entities
Variables / Observations
T
e 11
1k
O
0
T2
21 .... 22
O 21 0pp
Figure 1: Sample Database Schema with all entities and attributes joined together.
represents constraints like count, sum, average, etc (see appendix A.2 details). Consider the following prediction goal and the corresponding PeTED expression, for example:
For each airline, predict the maximum delay that any of its flights will suffer next week.
This prediction goal can be expressed by the following PeTED expression:
Target Attribute: SERTVAL_DRIAY
Pi Piltering Constrašnt: NONR
Aggregation Constraint: max_agg
3.2 Filling PeTED Slots
In this section, we discuss the filling process for the PeTED slot "Target Attribute". We omit details for other slots due to space constraints, as they follow a similar process. Each PeTED slot is modeled as a random variable, e.g., the Target Attribute slot represents probability distribution over all attributes given schema. These probabilities indicate the likelihood of an attribute being the desired Target Attribute given a Forecast Utterance.
We assume uniform prior over all attributes. meaning each is initially considered equally likely the target. Upon receiving Forecast Utterance, the agent extracts information and updates its belief about the Target Attribute slot by computing the posterior distribution for the corresponding random variable.
Formally, let the list of attributes in the given schema be A {ar a2,.... On } and represent the Forecast Ullerance. Our goal is to rank attributes A based on q. We assume that user utterance contains clues about the target attribute, so attributes with higher semantic similarity to q are more likely to the Targer attribute. Thus, attributes with higher similarity are ranked higher, as they are more likely the desired target attribute.
User utterances often uncertain and implicit, making crucial extract salient phrases for accurate inference. This can be achieved using Entity Extraction (EE) techniques (Nasar et al.) 2021) or Question-Answering (QA) techniques (Soares and Parreiras 2020), where targeted questions identify slots and answers extract salient phrases. Since one-to-one mapping from salient phrases to candidate attributes is unlikely, one needs to consider the following two probabilities jointly to make an accurate inference about the Target Auribute slot.
1. Given salient-phrass x extracted from forecast utterance q. what is the likelihood that a is indeed relevant the desired Target Attribute? To capture this, introduce binary random variable Rg, which is defined as the relevance of a salient-phrase with respect to the target attribute.
R. 1, if is relevant salient-phrea
(1)
otherwise
Here, x can be any salient-phrase extracted from user utterance, such that x Z(g), where Z(g) theoretically defines all possible n-grams with n (,.......) Mathematically, we need estimate P(R,, = 1(x).
2. Given a relevant salient-pi aalient-phrase x E Z(g) and an attribute a; from the data-base schema, what is the probability that ag indeed the target attribute? Mathematically, need estimation of P(as/2, R₂ = =1).
4

---

Published in Transactions on Machine Learning Research (03/2024)
Finally, all attributes in the data-base schema ranked according the following joint probability.
P(a.lg) max {P(R. = 1/2) x P(as R. 1)} xe
(2)
3.3 The self-supervision with synthetic examples Approach
Calculating probability distributions for PeTED representations is challenging due to the non-deterministic nature of user utterances and the varying number values for each slot in real-time, unseen datasets. Pre-training a model with variable slot options becomes difficult.
To address this, propose self-supervised learning approach utilizing synthetic examples for conversational forecasting task formulation. Upon receiving that simulate probable user utterances (explained in Section 4.2) These examples help the model learn the dataset's attributes, granularity, and types. Once trained, the model accurately interprets forecast utterances, as demonstrated by our experiments. With the dataset schema provided, users don't 't need manual data labeling or annotation, therehy facilitating self-super learning with synthetic examples.
dataset, generate unsupervised, artificial training examples
4 Estimation of PeTED Expressions
In this section, we first outline our assumptions for estimating PeTED expressions. Next, elaborate the joint probability estimation process from Equation Finally, we discuss the process of ranking candidate attributes using our probability estimates to fill the PeTED slots.
4.1 Assumptions
The assumptions for our case studies are as follows:
Our PeTED expression consists of four slots: Target Attribute, Aggregation Constraint, Filter Attrbbute and Filtering Constraint.
A Forecast Utterance may not contain all required information about each slot, i.e., users may provide partial/incom-
plete information.
Each slot can have one candidate value.
4.2 Estimation of P(Rx 1[x)
We present the estimation process for the "Target Attribute" slot, while noting that other slots follow similar approach. The probability P(R,- 1|z) signifies the likelihood that salient-phrase (x) extracted from the forecast utterance (q) is relevant to the desired target attribute. Here, Z(4). where Z(q) represents all possible n-grams in g. with n= 1, Since (q)) is computationally intractable by definition, we tackle this complexity by utilizing Entity Extraction (EE) and Question Answering (QA) techniques. These methods allow us to extract limited number of salient-phrases, along with confidence scores, from the forecast utterance and estimate probabilities accordingly.
While EE and QA techniques offer potential in computationally estimating P(Rg 1|x), directly incorporating pre-trained EE/QA model is unsuitable for real-time usev-provided database schema scenario. Given the lack of pre-existing training dataset tailored each unique schema/domain. fine-tuning pre-trained models is unattainable. With diverse user base expecting assistance in devising forecasting tasks for their distinct problem domains and database schemas, fine-tuning for every possibility is infeasible.
To address this limitation, we present a robust approach, comprising two methods for synthetic data generation. The first method, heuristic technique, involves constructing realistic template utterances with empty slots, subsequently populated with relevant attributes and their synonyms derived from the provided schema. This method generates context-specific, custem-crafted examples capturing the core of forecasting tasks. For instance, consider the following template:
Predict the average for each airline tomorrow.
5

---

Published in Transactions on Machine Learning Research (03/2024)
We can fill in the blank of this example template utterance using different attributes and their synonyms from given user schema.
The second method utilizes a T5 model finetuned on the CommonGen (Lin et al. 2020) task, which generates artificial user utterances via a keyword- neyword-to-sqqueeee se-sequence task, aiming to create more natural- like utterances containing specified slots that partially conform to our templates. We developed three versions of the T5 model: the first remains unaltered, the second fine-tuned with 1,000 templated samples integrate template essence subtly, and the third is fine-tuned with 10,000 examples enforce template structure more emphatically. The resulting T5-based synthetic dataset combines balanced mixture of utterances from each version, ensuring diversity, various degrees of template conformity, and natural language expression.
We utilize both synthetic datasets along with their corresponding slots to create training samples formatted according to CoNLL-2003 standards (Tjong Kim Sang and De Meulder 2003) for Entity Extraction (EE) and according to SQuAD standards for Question Answering (QA) (Rajpurkar al. 2016). This comprehensive foundation allows to effectively fine-tune pre-trained hansformer-basee models for EE and QA task. The fine-tuned models are capable of accurately identifying key phrases and generating confidence scores from previously unseen utterances. These confidence scores are instrumental in providing reliable estimation of the probability P(R- 12x). demonstrated in our experimental findings.
Algorithm 1: Forecasting Goal Extraction from User Utterances via Slot-Filling.
Algorithu TasiEo=mmllstinn/
Input: Attributes {a:). PeTED, utterance(g)
Output: PeTED with revised probability distribution
Initialization: pretrained Transformer-based Language Model, PeTED with uniform distribution
D +- TrainingSetGennerationn/
model + Fine-tune Transformer- LM on EE/QA task using
8
X, conn + Salient-Phrases and confidence scores extracted from g by applying model P(R, 1/2 E X) + Normalize o into probability distribution
for æ C X do for o; E schema do P(a,(2, Rs = 1) + sem__smmmee (em_ssim(a,.a) end
14
16
Normalize P(as|a, lx, Rs 1) over all o;
20
end
22
PeTED < Re-compute PeTED probabiliy distributions using equation do
24
Show top item from PcTED to user;
27
if If user agrees then return PeTED; else
28
Remove item from PeTED; Re-compate PeTED distributions;
while User has not agreed or list is not exhausted
4.3 Estimation of P(as|x, Rx =
Given relevant (R2 1) salient-phrase and candidate attribute a; from the database schema, P(a:/z, R2 1) represents the probability that is the desired target. We assume that attributes with high semantic similarity to relevant salient-phrases are more likely to be the target attribute, as users often mention it directly or through synonyms/salient- phrases. Consequently, we model P(a x, Rg = 1) as proportional to the semantic similarity between x and a; Mathematically, P(as|æ, R2 = 1) x Sem_similariiy((a....
4.4 Ranking Algorithm for Slot-Filling
Algorithm [ details the slot-filling process to generate PeTED expression from the user's database schema and forecasting goals. As stated in Section 3.2 the "Target Attribute" values are initialized using a uniform distribution across all schema attributes. Algorithm receives the database schema attributes, initial PeTED expression, and

---

Published in Transactions on Machine Learning Research (03/2024)
user utterance as imput. It then generates updated PeTED expression, incorporating posterior distribution for the target attribute based on the joint probability calculated from equation 2 The attributes are then ranked based on this probability, with higher values indicating a greater likelihood of being the desired target.
5 Case-Studies and Experimental Setup
Entity Extraction
Question Answering
Dataset
Model
Universal
Custom
Universal
Custom
Heuristic
T5
Houristic
T5
Heuristic
T5
Heuristic
T5
Bert
0.676
0.831
0.743
0.896
0.793
0.840
0,747
0.765
Flight Delay
RoBERTa
0.673
0.864
0.737
0.890
0.753
0.762
0,836
0.856
XLNet
0,750
0.871
0,808
0.894
0.619
0.756
0,666
0.741
Albert
0.657
0.862
0.817
0.889
0.675
0.822
0.752
0.817
Bert
0.625
0.615
0.614
0.539
0.801
0.754
0.640
0.739
Online Delivery
RoBERTa
0.637
0.609
0.619
0.582
0.729
0.735
0.760
0.741
XL.Net
0.655
0.650
0.624
0.5709
0.633
0.606
0.536
0.595
Albert
0.645
0.585
0.640
0.547
0.780
0.734
0.609
0.685
Bert
0.569
0.539
0.553
0.525
0.610
0.56
0.639
0,570
Student Performance
RoBERTa 0,584
0.568
0.528
0.565
0.609
0,555
0.628
0,531
XLNet 0,595
0.521
0.515
0,469
0.582
0,516
0.472
0.529
Albert
0,564
0.551
0.541
0,571
0.558
0.568
0.512
0.526
Table 1: Fine-tuning performance all four attributes extraction task using EE/QA. Fl score of models that trained with Heuristic-bai based and T5-based synthetic data are labelled with Heuristic and T5 respectively.
5.1 Data-sets and Evaluation Metric
Our methodology stands out by dynamically processing any dataset provided by the user, instantly generating synthetic examples tailored to that dataset. Subsequently, it trains a transformer- language model (LM),, specificall designed to extract information relevant to the dataset directly from user utterances. This streamlined process ensures efficient and accurate retrieval of speesct- insights, catering the immediate needs of users without the prerequisite of pre-established datasets.
In the course of our research, we put our system to the test using three publicly available Kaggle datasets: Flight Delay ( Online Food Delivery Preferences (OD) and Student Performance Detaills in Appendix A.3). These datasets were chosen to cover a range of topics and data types, showcasing the versatility and robustness of our approach. For evaluating how well our method works, we created test sets for the three datasets using human volunteers with data science expertise, who generated utterances expressing forecasting goals. Each instance consists of user utterance and associated ground truth slot-value labels. To minimize bias, three volunteers independently created and labeled datasets.
The final test sets contain 344, 170, and 209 sentences for the FD, OD, and SP datasets, respectively. This detailed process ensures that our test sets are both thorough and reliable. To measure how well our model works, we use the Fl score to see how accurately it can identify the correct information in sentence. We also use the mean reciprocal rank (MRR) to measure the efficacy of our ranking system. These measures help us understand the effectiveness of our model in real-world scenarios, proving the value of our innovative approach academic research.
5.2 Embeddings and Pre-Trained Transformer based Language Models
We employed three word embeddings techniques (Word2Ved Mikolov al. 2013)) Glo Ve Pennington et al. (2014) FastTex(Bojanowsse al. (2017) and alongside them we also include Universal Sentence Encoder (USE) Cer al. 2018)) to capture semantic similarity. The rationale for including sentence-level approach is that sentence embeddings have demonstrated strong performance in capturing semantic similarity across standard benchmarks Mahajan et al.
2023 ). We use four pre-trained transformer- based language models (Bert Devlin et al. (2018). RoBERTa Liu et al.
/https//wwwwkkggle.c http::/wwww.....ooo///////hhaaaa
7

---

Published in Transactions on Machine Learning Research (03/2024)
2019), XLNet Yang et al. (2019), and Albert Lan et al. (2020) to fine-tune for the EE/QA tasks in case studies. The fine-tuning process for the EE/QA tasks include two variations: 1) custom models, fine-tuned exclusively on artificially generated data from a single dataset, and 2) universal model, fine-tuned on artificially generated data for all three datasets. As discussed in Section 4.2 we employed two methods to generate artificial data: heuristic-basee and T5-based. In the case study, models fine-tuned using heuristic-generated artificial data are labeled with the keyword Heuristic, while those fine-tuned with T5-generated data are labeled with the keyword T5.
5.3 Hyper-parameter Tuning
Due the nature of our implementation, our experimental results heavily depend on how well the fine-tuned model (EE/QA) extracts salient-phrasee from the user utterances. This, in turn, depends the effectiveness of the fine-tuning process described in Section 4.2 Therefore,
performed exhaustive hyperparameter search using subset of both of our -eneerrtty data-set. We then took the highest-scoring set of hyperparameters and fine-tuned the extractive models one more time using the whole training data-set. See Appendix A.6 (Tables 12 15 for details.
6 Results
The performance of the fine-tuned extraction models (EE/QA) on the complete training data is displayed in Table [] By analyzing the results in Table [ we have identified the preferred model configurations for different settings, as presented in T We observed that Universal models yield reasonable F1 scores, making them suitable for users with strict memory constraints. Furthermore, T5 models do not outperform Heuristic models across all three test datasets, indicating that if user utterances follow template like Forecast Utterance, training with templated data can be effective.
EE
Custom
RoBERTa
Albert
Heuristic
Universal
Hert
XL.Net
Custom
RoBERTa
RoBERTa
TS
Universal
Bert
XLNet
Table 2: Best extractive models for different settings. Models under the label Heuristic and T5 are the best performing models trained on Heuristic- based and T5- ba-dd synthetic data respectively.
Aegregation Operation
Target Attribute
T5
T5
Heuristic
0.9
Heuristic
0.8
0.98
0.97
Universal-EE EE
Universat-AA
Custom-EE EE Gustom-QA
Universal-EE EE
Universal-04
Custom-ttt EE
Custam-GA
Universa--E Ef
Universal-AA QA Contom-EE EE
Custom-De -4
Universal-EE EE Untversal-O QA Custom-tt EE
Custam-Oa OA
Universal-ge EE
Universal-O Custom EE
Custom-Da Q4
Universal-Ee EE
Universal-o QA
Custom Ef
Custam-Cr Q4
FD
00
FD
00
Figure 2: MRR for Aggregation Constraint prediction.
Figure 3: MRR for Target Attribute prediction.
We present the case study results in Figures 2 using the recommended configurations from Table 2 The Mean Reciprocal Rank (MRR) for each slot, derived from the four embedding techniques detailed in Section 5.2 is averaged across all models in Table 2 These visualizations demonstrate MRR performance for each slot based on our case study assumptions. Notably, models perform better on datasets with fewer attributes, which is intuitive. Furthermore, the final MRR score is influenced by the model's extraction task performance. In Figure models excel due the ease of extracting aggregation constraints (count, sum, etc.), while Figure performance declines filtering constraints (high, less, etc.) are more abstractive.
8

---

Published in Transactions on Machine Learning Research (03/2024)
Filter Attribute
1
Filter Operation
0.9
Heuristic
T5
0.75
Heuristio
0.7
0.5
0.5
0.3
Universal-EE Universa--o O4 Custom-Et EE Custom-
Universat-EE
Univnnal-o Custom-da EE
Custom-
Universal-Et EE
Univerzal Q4
Custom- EE
Custom-D
Universal
Universal-O 04
Curonn-
Custom GA
Unbersal EE
Universal QA
Cuttom
Custom GA
Universal EE
Universal-g Q24
Custom- EE
Custom-D 04
0D
Figure MRR for Filter Attribute prediction.
Figure 5: MRR for Filtering Constraint prediction.
We performed the Wilcoxon Sign-Ranked Test on all model pairs (detailed in Appendix, Table (8) using flat vector of MRR scores from all datasets and slots. The results revealed statistically significant differences between the MRR scores of the Vanilla method and any other extractive models (EE/QA).
6.1 Failure Analysis
In our extractive model, we observe slower convergence rate when handling utterances with abstract or implicit slot expressions. This is particularly evident in the performance of our proposed T5 and Heuristic models, which struggle to extract filtering constraints (Figure 5) effectively as they do for other slots (Figures instances where the Heuristic Universal XI.Net model encounters extended convergence times. A closer analysis of these examples reveals that, when users mention slot values implicitly, the model tends to extract incorrect salient phrases, underscoring the difficulties in accurately discerning subtle or abstract expressions within the text.
Table showcases
User Utterance
Extracted
Count
predict the total order where the preferred medium of order is online within tomorrow
"within"
9
Predict the maximum final grade of students where student has extra educational support at school
7
Table 3:
This table displays the user utterance (first column), extracted salient phrase (second column), and iterations needed for correct slot-value (third column).
6.2
Open-Domain Dialog systems
User:
Predict the maximum first period grade students where student has partaken extra-currieular activities Is this something that can help you plan schedule for your college students?
GODEL:
User:
nope. just formulate a ML problem the utterance. Do you have any idea on the methods of determining the maximum amount of first period grade of students where student has partaken in estra-curriculat activitics?
GODEL:
User:
nope. that is your job
GODEL:
You should get me a job based on this information and that is why want to get a job.
User:
you are not helping with problem creation.
GODEL:
T aggree that you are not helping with the problem creation problem.
Table 4: User Interaction with GODEL. The model is hallucinating and no way of converging.
In this section, we present a qualitative study comparing two recent Open Domain Dialog Systems (ODDS): GODEL and ChatGPT. While a direct comparison between our proposed method and these ODDS is not feasible, we aim to discuss the adaptability of ODDS in the context of this highly ambitious task. Our study encompasses four distinct system setups to evaluate the performance of these models across diverse scenarios. Due to space constraints, we present only one such scenario (see Appendix A. for details). For both GODEL and ChatGPT, we provide the following instructions as the system's role: "You Machine Learning expert who will help a small business owner

---

Published in Transactions on Machine Learning Research (03/2024)
User:
Predict the maximum first period grade of students where student has partaken in extra-curricular activities
ChatGPT:
Can you please tell me the time period for which you want to make this prediction?
User:
I do not have time period available with me. Please the given information predict.
ChatGPT:
To do this, we can consider each school term as time period and use the following slot values: Aggregation operation: Maximum; Target attribute: First-period grade; Filter attribute: Participation in extracurricular activities; Filter operation: Equals to "Yes"
User:
please check the filter operation again.
ChatGPT:
Apologies for the mistake. Can you please provide the correct filter operation so that I can update the slot
values accordingly?
User:
I do not have the filter operation value. Please check the original input to find the value. Okay. Just to confirm, did you mean to say that you didn't 't mention any filter operation value in your original message?
ChatGPT:
User:
No. The filter operation value is mentioned in the original message. Please read the message and find the value.
ChatGPT:
The correct slot values are as follows: Aggregation operation: Maximum; Target attribute: First-perion
grade; Filter attribute: Participation in extracurricular activities; Filter operation: Equals to "Yes"
User:
The value filter operation value is incorrect.
ChatGPT:
apologize for making a mistake earlier. Can you please tell me the correct filter operation value so that can update the slot values accordingly?
User:
I dont know
ChatGPT:
Based on your initial request, we can create time series forecasting problem as follows: Aggregation
operation: Maximum; Target attribute: First-period grade; Filter attribute: Participation in extracurricular activities
User:
Filter operation value is missing in above response.
Table 5: User Interaction with ChatGPT. The model struggles with extracting correct filter operation.
by formulating an interesting problem through conversation. You should perform a slot-filling task based on the conversation and use those slot information to formulate a time series forecasting problem for for the user. The slots are aggregation operation, target attribute, filter attribute, and filter operation. You have to infer the slot values from the conversations". User interactions with GODEL are illustrated in Table while interactions with ChatGPT can be found in Table The study reveals that GODEL fails converge, while ChatGPT performs relatively well. Both ChatGPT and our proposed model struggle to extract the Filter operation constraint. Given the potential of ChatGPT, we plan to explore it as future direction for improving the extraction of such constraints and enhancing overall model performance. In summary, ChatGPT shows great promise in materializing the vision of "Conversational Data Science".
7 Conclusion
In this paper, we have introduced the concept of Forecast Utterance, where, user expresses their forecasting goals through natural language. We have primarily focused on how to automatically understand users' prediction goals from such utterances accurately in an unsupervised fashion. As first step towards the ambitious goal of "Conversationnl Data Science", we have framed this task as an unsupervised slot-fillin not-filiing problem and, accordingly, proposed self-supervised solutions with synthetic examples based on Entity task's 's user-centric and real-time nature, the lack of custom training data for fine-tuning EE/QA models in the ranking algorithm was a significant challenge, which we addressed by our proposed heuristic and T5 based synthetic data generation technique. Experimental results show that the proposed synthetic data generation technique significantly improved the accuracy of the overall ranker, and, therefore, demonstrates the feasibility of general "Conversational
Answering QA) techniques. Due to the
10

---

Published in Transactions on Machine Learning Research (03/2024)
Data Science" paradigm. This task is ambitious as well multidisciplinary in nature and requires attention from multiple research communities, including NI.,, AutoMI. and HCI.
8 Limitations
Our work is limited by the following assumptions made in the paper:
We assumed that PeTED expression consists of four slots: Target Attribute, Aggregation Constraint, Filter Attribute and Filtering Constraint, which may not hold in some real word scenarios.
In this work, we assumed that Forecast Ulterance may not contain all required information about each slot, i.e., users may provide partial/incompltt information. But, we have not studied how to prompt users to gather further information or ask clarification questions, which is an interesting future work.
We have assumed that each slot can have one value at maximum, which may not hold real word. We plan extend our approach multiple values scenario in the future.
Although our case study is limited in many ways, this limited scope should not undermine the potential impact of our proposed idea, i.e., Forecasting Task Formulation Via Natural Conversation. Imagine Alexa, Siri, Watson and other home assistants serving your personal data scientists. As the first work of its kind, we made choice regarding the inevitable trade-off between taking small step towards ambitious goal vs taking significant step towards solving well-defined problem. Thus, we had to leave more complex scenarios as future work. As a novel perspective, we believe our work will inspire the researchers to pursue this ambitious problem and make substantial impact through wider adoption of such "end-userrinn-the-loop"" conversational AutoMI. solutions.
9 Acknowledgements
This work has been partially supported by the National Science Foundation (NSF) Standard Grant Award #2302974 and Air Force Office of Scientific Research Grant/Cooperative Agreement Award 1-A9550-2331-04222
References
Mustafa Abdallah, Ryan Rossi, Kanak Mahadik, Sungchul Kim, Handong Zhao, and Saurabh Bagchi. 2022. AutoFore cast: Automatic time-series forecasting model selection, In Proceedings of the 31st ACM International Conference Information & Knowledge Management, CIKM '22, pages 5-14. Association for Computing Machinery.
Zeynep Akata, Florent Perronnin, Zaïd Harchaoui, and Cordelia Schmid. 2013. Label-embedding for attribute-based classification In 2013 IEEE Conference Computer Vision and Pattern Recognition, Portland, OR, USA, June 23-28, 2013, pages 819-826. IEEE Computer Society.
Hemanthage S. Bhathiya and Uthayasanker Thayasivam. 2020. Meta learning for few- joint intent detection and slot-filling In Proceedings of the 2020 5th International Conference on Machine Learning Technologies, ICMLT 2020, page 86-92, New York, NY, USA. Association for Computing Machinery.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146
Giovanni Campagna, Agata Foryciarz, Mehrad Moradshahi, and Monica S. Lam. 2020. Zero-shot transfer learning with synthesized data for multi-domain dialogue state tracking In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, ACL 2020, Online, July 5-10, 2020, pages 122-132. Association for Computational Linguistics.
Daniel Cer, Yinfci Yang. Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo- Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018 Universal sentence encoder for English In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Lu Chen, Boer Lv, Chi Wang, Su Zhu, Bowen Tan, and Kai Yu. 2020. (Schema-guuii guided multi-domain dialogue state tracking with graph attention neural networks) In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7521-7528, AAAI Press.
Yan Chen and Zhenghang Luo. 2023. Pre-trained joint model for intent classification and slot filling with semantic feature fusion Sensors, 23(5).
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. |ERT; pre-training of deep bidirectional transformers for language understanding CoRR, abs/1810.04550
Chih-Wen Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung- Ching-Chich Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018. [Slotggated modeling for joint slot filling and intent prediction) In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HILT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pages 753-757. Association for Computational Linguistics.
Md. Mahadi Hassan, R. Alexander Knipper, and Shubhra Kanti Karmaker Santu. 2023, Chatgpt as your personal data scientist CoRR, abs/2305.13657
Matthew Henderson, Blaise Thomson, and Jason D Williams. 2014a. The second dialog state tracking challenge. In Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL), pages 263-272.
Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014b. The third dialog state tracking challenge. 2014 IEEE Spoken Language Technology Workshop (SLT), pages 324-329.
Hailin Hu, MingJian Tang, and Chengcheng Bai. 2020. DATSING: data augmented time series forecasting with adversarial domain adaptation] In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 2061-2064. ACM.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging
CoRR,
Mourad Jbene, Smail Tigani, Rachid Saadane, and Abdellah Chehri. 2022. A robust slot filling model based on Istm and eri for iot voice interaction In 2022 IEEE Globecom Workshops (GC Wkshps), pages 922-920
Xiaoyong Jin, Youngsuk Park, Danielle Maddix, Hao Wang, and Yuyang Wang. 2022. Domain adaptation for time series forecasting via attention sharing In Proceedings of the 39th International Conference on Machine Learning, pages 10280-10297, PMLR. ISSN: 2640-3498.
Eleftherios Kapelonis, Efthymios Georgiou, and Alexandros Potamianos. 2022. A multi-task BERT model for schema- guided dialogue state tracking
Shubhra Kanti Karmaker, Md Mahadi Hassan, Micah Smith, Lei Xu, Chengxiang Zhai, and Kalyan Veeramachaneni. 2021. Automl to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 54(8):1-36.
Chandra Khatri, Rahul Goel, Behnam Hedayatnia, Angeliki Metanillou, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018 Contextual topic modeling for dialog systems In 2018 IEEE Spoken Language Technology Workshop (SLTT), pages 892-899.
Jason Krone, Yi Zhang, and Mona T. Diab. 2020. Learning classify intents and slot labels given handful of examples CoRR, abs/2004.10793.
Gakuto Kurata, Bing Xiang, Bowen Zhou, and Mo Yu. 2016. Leveraging sentence-level information with encoder LSTM for semantic slot filling In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2077-2083. The Association for Computational Linguistics.
12

---

Published in Transactions on Machine Learning Research (03/2024)
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001a. Conditional random fields: Probabilistic
models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28- - July 1, 2001, pages 282-289.
Morgan Kaufmann.
John D. Lafferty, Andrew McCallum, and Fernando N. Pereira. 2001b. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighleenth International Conference on Machine Learning (ICML 2001). Williams College, Williamstown, MA, USA, June 28- July 1, 2001, pages 282-289.
Morgan Kaufmann.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT A lite BERT for self-supervised learning of language representations In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net.
Stefan Larson and Kevin Leach. 2022. A survey of intent classification and slot- slot-filling datasets for lask-oriented dialog
Lizi Liao, Tongyao Zhu, Le Hong Long, and Tat-Seng Chua. 2021. Multi-domain dialogue state tracking with recursive inference| In WWW '1: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 2568-2577 ACM IW3C2.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning, In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823-1840 1840, Online. Association for Computational Linguistics.
Bing Liu and lan R. Lane. 2016. Attention-based recurrent neural network models for joint intent detection and slot filling In Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016, pages 685-689. ISCA.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692
Yash Mahajan, Naman Bansal, and Shubhra Kanti Karmaker. 2023. The daunting dilemma with sentence encoders; Success standard benchmarks, failure in capturing basic semantic properties) CORR, abs/2309.0374)
Shikib Mehri and Maxine Eskenazi. 2021. GenSF: Simultancous adaptation of generative pre-trained models and slot filling Version: 1.
Grégoire Mesnil, Yann N. Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Hakkani-Tür Xiaodong He, Larry P. Heck, Gökhan Tür, Dong Yu, and Geoffrey Zweig. 2015. Using recurrent neural networks for slot filling in spoken language understanding IEEE ACM Trans. Audio Speech Lang. Process., 23(3):530-555
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv: 1001.3011..111
Zara Nasar, Syed Waqar Jaffry, and Muhammad Kamran Malik. 2021. Named entity recognition and relation extraction: State-of-the-art, ACM Computing Surveys (CSUR), 54(1):1-39.
Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020. [SMBA:: self-supervised manifold based data augmen- Lation for improving out-of-domain robustness) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1268 1283. Association for Computational Linguistics.
Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2021. Meta-learning framework with
applications to zero-shot time-serics forecasting In Thirty- Fifth AAAI Conference on Artificial Intelligence, AAAI 1021, Thirty- Third Conference on Innovative Applications of Artificial Intelligence. IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages AAAI Press.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Bernardo Pérez Orozco and Stephen J. Roberts. 2020. Zero-shot and few-shot time series forecasting with ordinal regression recurrent neural networks) In 28th European Symposium on Artificial Neural Networks, Compulational Intelligence and Machine Learning, ESANN 2020, Bruges, Belgium, October 2-4, 2020, pages 503-508.
Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton, and Tom M. Mitchell. 2009. Zero-shot learning with semantic output codes In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Proceedings of a meeting held 7-10 December 2009, Vancouver, British Columbia, Canada, pages 1410-1418, Curran Associates, Inc.
Mark A. Parsons, Gystein Godøy, Ellsworth LeDrew, Taco F. de Bruin, Bruno Danis, Scott Tomlinson, and David Carlson. 2011. A conceptual framework for managing very diverse data for complex, interdisciplinary science J. Inf. Sci., 37(6))555-569)
Baolin Peng. Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. 2022. Godel: Large-scale pre-training for goal-directed dialog arXiv.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Ella Rabinovich, Matan Vetzler, David Boaz, Vineet Kumar, Gaurav Pandey, and Ateret Anaby- 2022. Gaining insights into unrecognized utterances in task-oriented dialog systems
Md. Mustafizur Rahman, Shubhra Kanti Karmaker Santu, Md. Monirul Islam, and Kazuyuki Murase. 2014, Forecasting time series A layered ensemble architecture In 2014 International Joint Conference on Neural Networks, IJCNN 2014, Beijing, China, July 6-11, 2014, pages 210-217. IEEE
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text, In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, EMNLP 2016, Austin Texas, USA, November 1-4, 2016, pages 2383-2392 The Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2017. Optimal hyperparameters for deep -etm-networks for sequence labeling tasks) CoRR,
Dana Ruiter, Dietrich Klakow, Josef van Genabith, and Cristina España-Bonet. 2021. Integrating unsupervised data
generation into self-supervised neural machine translation for low now-resource languages) In Proceedings of the 18th Biennial Machine Translation Summit - Volume 1: Research Track, MTSummii 2021 Virtual, August 16-20, 2021, pages 76-91. Association for Machine Translation in the Americas.
Shubhra Kanti Karmaker Santu, Parikshit Sondhi, and ChengXiang Zhai. 2017. On application of learning to rank for e-commerce search In Proceedings of the 40th International ACM SIGIR Conference on Research and Development Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 475-484 ACM.
Souvika Sarkar, Dongji Feng, and Shubhra Kanti Karmaker Santu. 2022. Exploring universal sentence encoders for zero-shot text classification) In Proceedings of the 2nd Conference of the Asia-Pucific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference Natural Language Processing, AACI/IJCNIP 2022 Volume 2: Short Papers, Online only, November 20-23, 2022, pages 135-147. Association for Computational Linguistics.
Souvika Sarkar, Dongji Feng, and Shubhra Kanti Karmaker Santu. 2023. Zero-shot multi-label topic inference with Sentence encoders and llms) In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 16218-16233 Association for Computational Linguistics.
Dongyu She and Kun Xu. 2021. Contrastive self sulf-supervised representation learning using synthetic data Int. Autom. Comput., 1864)5556-567.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self- attention network for run/cnn-free language understanding. In Proceedings of the Seiooy- AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5446-5455. AAAI Press.
Marco Antonio Calijorne Soares and Fernando Silva Parreiras. 2020. A literature review question answering techniques, paradigms and systems. Journal of King Saud Sitiversity-Compptte and Information Sciences, 32(6):635- 646.
Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention In Proceedings of the Thirty-Secone AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 4929-4936. AAAI Press.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2003, pages 142-147. Edmonton, Canada.
Mike Van Ness, Huibin Shen, Hao Wang, Xiaoyong Jin, Danielle C. Maddix, and Karthick Gopalswamy, 2023. Cross-frequency time series fota-toreaasting
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, In Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Mathilde Veron, Olivier Galibert, Guillaume Bernard, and Sophie Rosset. 2022. Attention modulation for zero-shot cross-domain dialogue state tracking In Proceedings of the 3rd Workshop on Computational Approaches to Discourse, pages 86-91. International Conference on Computational Linguistics.
Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. 2021. Self-supervised learning with data augmentations provably isolates content from style In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 16451-16467.
Liwen Wang. Xuefeng Li, Jiachi Liu, Keqing He, Yuanmeng Yan, and Weiran Xu. 2021. Bridge to target domain by prototypical contrastive learning and label confusion: Re-explore zero-shot learning for slot filling
Yexiang Wang, Yi Guo, and Siqi Zhu. 2020. Slot attention with value normalization for multi-doman dialogue state tracking] In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3019-3028. Association for Computational Linguistics.
Yiren Wang, Dominic Seyler, Shubhra Kanti Karmaker Santu, and ChengXiang Zhai. 2017. A study of feature construction for text-based forecasting of time series variables In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM 2017, Singapore, November 06 10, 2017, pages 2347-2350.
ACM.
Joseph Weizenbaum. 1966. ELIZA a computer program for the study of natural language communication between man and machine Commun. ACM, 9(1):36-45 45.
Jason Williams. 2013. Multi-domain learning and generalization in dialog state tracking In Proceedings of the SIGDIAL
2013 Conference, The 14th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 22-24 August 2013, SUPELEC, Metz, France, pages 433 441. The Association for Computer Linguistics.
Ronald J. Williams and David Zipser. 1989, A learning algorithm for continually running fully recurrent neural networks Neural Comput., 1(2):270-280.
Lei Xu, Shubhra Kanti Karmaker Santu, and Kalyan Veeramachaneni. 2019. MIfriend: Interactive prediction task recommendation for event-driven time-series data. arXiv preprint 1rXiv:1906. 12348.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. 2019. Xinet: Generalized autoregressive pretraining for language understanding) Cite 1rx1: Pretrained
models and code are available at https:/giilub..om/iilnggaaaiiiip
Tom Young, Frank Xing, Vlad Pandelea, Jinjie Ni, and Erik Cambria. 2022. Fusing task-oriented and open-domain
dialogues in conversational agents 36( 10): 1622- 1629. Number: 10.
Chenwei Zhang, Yaliang Li, Nan Du, Wei Fan, and Philip S. Yu. 2019. [Joint slot filling and intent detection via capsule neural networks In Proceedings of the 57th Conference of the Association for Computational Linguisties, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 5259-5261 Association for Computational Linguistics.
Junfeng Zhao, Shiqun Yin, and Wenqiang Xu. 2022. Attention- Attention-based iterated dilated convolutional neural networks for joint intent classification and slot filling. In Big Data, pages 150-162. Springer Singapore.
Su Zhu, Jieyu Li, Lu Chen, and Kai Yu. 2020. Efficient context and schema fusion networks for multi-domain) dialogue state tracking In Findings of the Association for Computational Linguisties: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 766-781. Association for Computational Linguistics.
A
Appendix
Algorithm 2: Artificial Training Data Generation through Heuristic for Fine-tuning EE/QA Models for "Target
Attribute" slot.
Algorithm Traini angaattt thweurihhhhriii st
Input: Schema S, Template utterance T, Attribute-phrases A
Output: Artificial Training Dataset
U Generate artificial utterances by filling every E with every a E A.
Unyo + Create additional artificial utterances from set U/ by replacing every E A with their synonyms.
U U
UL < annotate each E U with salient-phrases labeled as the target attribute.
return UL
Algorithm 3: Generating synthetic utterances using T5 model variants based attributes: The algorithm takes attributes, T5 model, templates, and K input, and generates synthetic dataset containing K balanced utterances from three T5 model variants, each with different levels of template conformity.
Input: Attributes, T5 model, templates, K
Output: Synthetic dataset
Load T5 model finetuned on CommonGen task Initialize synthetic dataset D - {}
for E (1,2,3} do
if = 1 then
5
M <- T5 model (unaltered)
else if i
then
7
+ T5 model fine-tuned with 1,000 templated samples
else
M + T5 model fine-tuned with 10,000 examples
end
Generate artificial user utterances U, using M and attributes for seyword-to-sequenee task
12
- DUU,
13 end
Mix utterances in D to create balanced blend of utterances from each model variant
15 return D)
16

---

Published in Transactions on Machine Learning Research (03/2024)
A.1 Artificial Data Generation
We present robust approach, comprising two methods for artificial data generation. As described in Algorithm the first method, a heuristic technique, involves constructing realistic template utterances with empty slots, subsequently populated with relevant attributes and their synonyms derived from the provided schema.
The second method as described in Algorithm 3 utilizes a T5 model fine-tuned on CommonGen (Lin et al. 2020) task. The CommonGen task, as detailed [Lin et al. 2020), designed enhance text generation models like 'T5 by requiring them create coherent sentences from a given words. This task evaluates the model's ability to weave these words into contextually meaningful sentences, testing its comprchension and creative synthesis skills. Fine-tuning T5 CommonGen specifically aims to improve its generative commonsense reasoning, making it adept producing human- like, contextually rich text. In our study, we used a T5 model that had been specially trained with the CommonGen task to generate synthetic user sentences. This approach involves transforming list of keywords into full sentences that sound natural and follow certain patterns we have set. This is aimed at creating sentences that include specific information (or 'slots') in way that makes sense and useful for our research. For example, when we input the keywords CAAATIIONNAARRRTT" "MAX", "DURATION OF SCHEDULED TRIP", "DESTINATION AIRPORT". "CANCELLATION REASON", "IS SSS_MOR__THHE THAN" into trained T5 model, it generated the following sentence:
"Find the longest scheduled trip each destination airport where the reason for cancellation was given more than one day earlier."
This example shows how the model can take a list of keywords and turn them into a clear, useful sentence. highlights the model's ability to understand and use language way that matches how people might ask for information, demonstrating value for creating better automated conversation systems.
A.1.1 Artificial Data Formats
After We generate the artificial dataset, we convert them into two format for training the Transformer Based Models. Each Format design is described below:
CoNLL-2003 standard: In this format we annotate each token of the generated data with appropriate labels. Our customized labels are: I-ATR (Target Attribute), I-AGG (Aggregation operator), I-FLT (Filtering Attribute), I-FLO (Filtering Operator). For example, If the generated sentence "Predict the average departure delay for each airline wherre elapsed time is more than five hours for tomorrow." Then the annotated data will be as following:
Predict I-NP the NN I-NP O average NN T-NP T-AGG departure NN I-NP 1 I-ATR delay NN I-NP I-ATR for NN I-NP each NN I-NP 0 airline NN I-NP where NN I-NP O
elapsed NN T-NP I-FLT time NN I-NP I-FLT
NN I-NP
more NN I-NP I-FLO than NN I-NP I-FLO five NN I-NP O
hours NN I-NP O
for NN I-NP tomorrow NN I-NP 0
Again if the artificial data is "Predict the average first period grade of students where school is Mousinho da Silveira" then the annotated data will look like the following:
Predict NN I-NP the NN I-NP average NN T-NP T-AGG
17

---

Published in Transactions on Machine Learning Research (03/2024)
first NN I-NP I-ATR period NN I-NP I-AIR
grade NN I-NP I-ATR
of NN T-NP O students NN I-NP I-ENT where NN I-NP O school NN I-NP 1-FLT is NN I-NP I-FLO Mousinho NN I-NP
da NN I-NP 0
Silveira NN O
SQuAD standards for Question Answering: For the SQuAD format, consider the artificial data the context for this task and try answer slot specific question from it. We the following questions:
Aggregator: what is the target aggregator?
Target Attribute: what is the target attribute?
Filter Attribute: what is the target filtering attribute?
Filter Operator: what is the target filtering operator?
Examples follows:
1:
"context"; "predict the average departure delay for each airlinc tomorrow", qas":
"question": :: "what is the target aggregator?".
"answers" ":
"text": "average", "answer_start"; 12, "answer_end"; 19
1. "id" :
"question"" ": "what is the target attribute?", "answers"""
"text": "departure delay",
"answer_start" 20, "answer_end"" 35
1.
"id ":
]
{
"context": "predict the average first period grade in the next fall semester among the students where school is mousinho da silveira".
"qss" ":
"question"; :: "what is the target aggregator?", "answers":
"text"; "average", "answer_start"; 12, "answer_end";; 19
l.
"id": "..."
18

---

Published in Transactions on Machine Learning Research (03/2024)
"question"; ": "what is the target attribute aanswers""
"text": "first period grade", "answer_start 20, "answer_end""; 38
id
"qusstion" :: "what is the target filtering attribute?". "answers":
"text": "school", "answer_start"" 90, "answer_endd; 96
l.
"id": "..."
"question"; : "what is the target filtering operator?", "answers"; ":ett": "is", "answer aaarte 97, annwer_eedd., 99
"
A.2 Filtering and Aggregation Operations
We have designed PeTED (Section 3.1 in such a way that it will provide constructs for variables, operators, and functions. Note that the type of variables is dependent on implementation, and the operators will depend on specific variable types. We have included some basic operators, which include filtering operators (equal to, greater than, etc.) and aggregation operators (count, sum, average, etc.) as part of this case study. Table/oshows list of sample operators for Forecast Utterance Understanding agent, along with their supported types.
A.3 Datasets
We experimented with three datasets: Flight Delay (FD), Online Food Delivery Preferences (OD), and Student Performance (SP) Dataset. Tables 7 the attributes and entities of those three dataset schemas. Using these datasets, we conducted an exploratory study to understand how our proposed slot-filling method performs in practical scenario. We use Algorithm to generate artificial data for each attribute the used datascts. We format the generated artificial training examples into the CoNLL-20033 format for the EE task and the SQuAD format for the QA task. We split those data into 8:2 ratio for training and testing split. We have created handcrafted validation sets for each of the three datasets by actively engaging human volunteers with decent data science expertise and asking them to create utterances expressing forecasting goals. Along with user utterances, each instance in the validation set contains ground truth labels correlated to it. We engaged three volunteers to create and label the datasets independently to avoid bias. Additionally, we cross-validater the dataset created by one volunteer by scrutinizing it with another volunteer, not involved in generating the particular dataset in the first phase. The handcrafted validation data-sets contain 344, 170 and, 209 utterances FD, and SP data-set, respectively.
A.3.1 Qualitative Examples
In this section, we showcase brief and precise explanations several utterances selected from our handcrafted validation dataset and describe the challenges to extract time series formulation related slot value from them.
19

---

Published in Transactions on Machine Learning Research (03/2024)
Utterance: "I want know the total departure delay for each airline that will be delayed more than five minutes by air system next week."
Domain: Flight Delay
This example represents a time series forecasting problem in airline operations. The goal is to predict the total departure delay for each airline with air system delays over five minutes next week. A model can be trained on historical data, capturing time-dependent patterns optimize resource planning and minimize passenger impact.
Description:
Challenges:
Implicit information:
Some critical information for problem formulation is not explicitly mentioned, such as the desired granularity of the forecast (e.g., daily, hourly) or the specific features to be used in the model.
Complex relationships: The utterance refers to multiple interrelated variables ("total departure delay" and "airlines with delays more than five minutes") that need to understood the context of the problem. This adds complexity to the task of extracting relevant slot-values.
Multiple slot-values: The utterance combines several aspects of the problem, which can make it challenging extract and separate the relevant slot-values accurately.
Domaan-specifie knowledge: Understanding the context of the utterance requires domain-specinf knowledge of the airline industry and the factors that contribute to flight delays. This knowledge is necessary to identify and extract the correct values.
Utterance: "Predict the average change in duration of flights for Qatar Airways which will have an elapsed time more than four hours and will start within the next week".
Domain: Flight Delay
This example represents time series forecasting problem in airline operations. The goal is to forecast the average change in flight duration for Qatar Airways for flights with over four hours of elapsed time starting in the next week. A model can be trained on historical data to capture time-dependent trends, helping airlines optimize scheduling and enhance customer experience.
Description:
Challenges:
The utterance contains terms like "average change in duration" that might be difficult to map directly to specific slot-values without clarification whether it refers to increase decrease in duration, both.
Ambiguity:
Complex relationships:
The utterance refers to multiple interrelated variables ("elapsed time more than four hours" and "flights starting within the next week"") that need to be understood in the context of the problem. This adds complexity to the task of extracting relevant slot-values.
Multiple slot-values: The utterance combines several aspects of the problem, making challenging to extract and separate the relevant slot-values accurately.
Doomain-ppecffte knowledge: Understanding the context of the utterance requires domain-specific knowledge of the airline industry and the factors that contribute to changes in flight duration. This knowledge is necessary to identify and extract the correct slot-values
Utterance: "Predict the maximum order placed by mistake where the age of a customer is less than 21 for the next month".
Domain: Online Delivery
This example represents time series forecasting problem in the online delivery domain. The objective is to forecast the maximum number of mistaken orders placed by customers under 21 years old for the upcoming month. By training model on historical data and capturing time-dependent trends, businesses can proactively identify and address potential issues, improving customer service and satisfaction.
Descríption:
Challenges:
The utterance refers to multiple interrelated variables ("maximum order placed by mistake" and "age of a customer less than 21") that need be understood in the context of the problem. This adds complexity to the task of extracting relevant slot-values.
Complex relationships:
20

---

Published in Transactions on Machine Learning Research (03/2024)
Multiple slot-values:
The utterance combines several aspects of the problem, making it challenging to extract and separate the relevant slot-values accurately.
-Dommin-speeiffe knowledge: Understanding the context of the utterance requires domain-specific knowledge the order placement process and factors that contribute to mistaken orders. This knowledge is necessary to identify and extract the correct values.lues.
Utterance: "Predict the total order where the preferred medium of order is online within tomorrow".
Domain: Online Delivery
Description: This example represents time series forecasting problem in the online delivery domain. The goal is to predict the total number of orders placed online for tomorrow. A model trained on historical data can capture time-dependent trends, helping businesses prepare for demand and optimize their operations.
Challenges:
The utterance refers to single variable ("total order where the preferred medium of order is online"), which needs to be understood in the context of the problem. This adds complexity to the task of extracting relevant slot-values.
Complex relationships:
Multiple slot-values: The utterance combines several aspects of the problem, making challenging to extract and separate the relevant slot-values accurately.
Domain-specifie knowledge: Understanding the context of the utterance requires domain-specitf knowledge of the order placement process and factors that contribute to online orders. This knowledge is necessary to identify and extract the correct slot-values.
Utterance: "Predict the maximum final grade of students where the student has extra educational support at school".
Domain: Student Performance
Deseription: This example represents time series forecasting problem in the student performance domain. The objective is to predict the maximum final grade for students who receive extra educational support at school. By training a model on historical data and capturing time-dependent trends, schools can better understand the impact of additional support and adapt their educational strategies accordingly.
Challenges:
Target variable: The utterance mentions "maximum final grade", which is single variable. However, it is not clear whether the goal is to predict the maximum final grade for an individual student or for group of students with extra educational support. This ambiguity adds complexity to the task of extracting relevant slot-values.
Data filters: The utterance introduces a filtering condition based on extra educational support. Extracting this filter accurately is crucial ensure that the problem is well-formulated and the predictions are relevant.
Domaann-peciffe knowledge: Understanding the context of the utterance requires domainn-secciff knowledge of the educational system and the factors that contribute to student performance. This knowledge is necessary to identify and extract the correct slot-values, such as the features to be used in the model.
The qualitative study examines various time series forecasting problem formulations across different domains, empha- sizing the unique challenges in extracting slot-values from user utterances.
Flight Delay Domain: In this domain, challenges include handling implicit information not explicitly mentioned in the utterance, addressing ambiguity in terminology, understanding complex relationships between variables, extracting multiple vllues, and utilizing nomain-sspecffe knowledge about the airline industry and factors that contribute to flight delays.
Online Delivery Domain: For this domain, challenges comprise understanding complex relationships between variables, such as order placement processes and factors contributing to mistaken orders, extracting multiple slot-values, and incorporating domain-specific knowledge about online delivery and e-commerce.
Student Performance Domain: In the context of student performance, challenges involve dealing with ambiguity in the target variable (e.g., individual group performance). extracting data filters accurately, such identifying specific conditions like extra educational support, and applying domain-spectie knowledge about the educational system and factors contributing to student performance.
21

---

Published in Transactions on Machine Learning Research (03/2024)
Addressing these challenges requires the development of advanced natural language processing techniques, context- aware algorithms, and incorporating domain knowledge to accurately extract slot-values and formulate time series forecasting problems. Each domain presents unique difficulties, making it essential to adapt models and techniques to handle specific domain-related complexities effectively.
Table 6: Operators
Operation
Supported Ops
Supported Types
Set
Filter O
all fil
None Numerical
greater_fit
sil eq fq_fil, neq_ fil
Categorical/Entity
Aggregation count_age __gg
None
Og g
sum agg,
Numerical
avg_agg,
min_agg,
Numerical
max agg
majority_agpp agg
Categorical/Entiyy
Table 7: Flight Delay Data-set details
Attribute Type
Attribute Name and Mcanings
Timestamp
Date, Day_of_weekk peeeduled_deparuure hour scheduled_tines
clapsddd clapsod_time
Entities
Airline, Flight_number, Tail_number
Origin airport, Destination airport
Categorical
Cancelled status, Cancellation reason
Numerical
Departure_dels d_ely_ Arrival_delay, delly Weather_delay
System_
Security_dehee
Late aircrafttdelll.
Table 8: Online Food Delivery Data-set details
Attribute Type
Attribute Name and Mcanings
Timestamp
Order Time
Entities
Food Order
Categorical
Gender, Marital Status, Occupation, Educational Qualifications, Medium, Meal, Preference, Time saving, Easy Payment option, More Offers and Discount, Good Tracking system, Self Cooking, Health Concern, Late Delivery, Poor Hygiene, Unavailability, Unaffordahle, Influence of time, Residence in busy location, Google Maps Accuracy, Good Road Condition, Low quantity low time, Delivery person ability, High Quality of package, Politeness, Freshness, Good Taste, Good Quantity,
Numerical
Monthly Income, Family size, latitude, longitude, Pin code, Easc and convenient, More restaurant choices, Good Food quality, Bad past experience, Long delivery time, Delay of delivery person getting assigned, Delay of delivery person picking up food, Wrong order delivered, Missing item, Order placed by mistake, Maximum wait time, Influence of rating, Less Delivery time, Number of calls, Temperature, Reviews
A.4 Transformer-basedd Language Models
We used existing Transforner-bast Language Models and fine-tuned them on Entity Extraction and Question Answer- ing tasks to extract salient phrases related to the target slots from user utterances to formulate the goal prediction task.
22

---

Published in Transactions on Machine Learning Research (03/2024)
Table 9: Student Performance Data-set details
Attribute Type
Attribute Name and Meanings
Binary
Student's school, Student's sex, Student's home address type, Family size, Parent's cohabitation status, Extra educational support, Wants take higher education, Internet access at home, With romantic relationship. Family educational support, Extra paid classes within the course subject, Extra-curricular activities, Attended nursery school,
Entity
Students
Nominal
Mother's job, Father's job, Student's guardian
Numerical
First period grade. Second period grade, Final grade, Free time after school, Going out with friends, Workday alcohol consumption, Weekend alcohol consumption, current health status, Number of school absences, Quality of family relationships. Number of past class failures, Weekly study time, Home to school travel time, Father's education, Mother's education, Student's 's age
We have used four popular Transformer-basee Language Models for our case study in addition to baseline method without using any Language Models.
Bert (Bidirectional Encoder Representations from Transformers) Devlin et al. (2018) has delivered a phenomenal impact in the Machine Learning community, achieving state-of-the-an results in a wide variety of NLP tasks, such as Question Answering (SQuAD v1.1), Natural Language Inference (MNLI), Named Entity Recognition, and so on. We use the 'bert-base-cased" variation of the model which contains 110 million parameters.
Bert:
RoBERTa: RoBERTa Liu al. 2019) is extension of BERT with several changes to the pre- pre-training procedure, including training for longer time periods, with larger batches, over more data, and without the next sentence prediction objective. We use 'roberta-base variation of the model which contains 123 million parameters.
XLNet: X et al. (2019) is an auto-regressive language model which outputs the joint probability of a sequence of tokens based on the transformer architecture with recurrence. XLNet is an extension of the Transformer-X model and has also performed well in EE and QA tasks. We use clnet-base-cased" variation of the model which contains 110M parameters.
Albert: Albert Lan et al. (2020) introduce A Lite Bert architecture that incorporates factorized embedding param- eterization and cross-layer parameter sharing parameter-reduction techniques. We use the base version of Albert ("alhert-hase-v22) which contains 31 million parameters.
A.5 Word-Embedding Techniques
We use phrase embeddings as part of our process to compute the semantic similarity (cosine and euclidean distance) between the user utterances and candidate target attributes. The list of attributes in the dataset are ranked according to these semantic similarity scores. In this case study, used three oftte-of-theaart word embeddings and one sentence enbedding.
Vord2Vec Mikolov et al. (0013) is a way to vectorize text. It's a two-layer neural net which vectorizes words. For input, it takes a text corpus and it outputs a set of vectors. These vectors are called feature vectors, which represent words in the corpus used. We used pre-trained version of Word2Vec provided by Google. This embedding is trained on the Google News dataset, and the model contains 300-dimensional vectors for million words and phrases.
Word2Vec:
FasfText: FastTex(Bojanwsski et al. (2017) is word embedding method where each word is represented as n-gram of characters. our used version of FastTTxt that contains million word vectors trained on the Wikipedia 2017, UMBC webbase corpus, and statmt.org news dataset (16B tokens)
GloVe: GloVe Penningtonn et al. (2014) is another popular embedding technique which is trained on aggregated global word-word co-occurrence statisties from corpus. We used version of GloVe embedding that contains 6B tokens, has vocab size of 400K, is uncased, and is 300-dimensional. is trained on the Wikipedia 2014 4 dump alongside the English Gigaword 5th edition dataset.
23

---

Published in Transactions on Machine Learning Research (03/2024)
Universal Sentence Encoder: The Universal Sentence Encoder encodes text into high dimensional vectors that can be used for text classification, semantic similarity, clustering, and other natural language tasks.
A.6 Hyper-Parameter Tuning
We performed an exhaustive hyperparameter search using subset of the artificially generated dataset. We present the search space for out hyperparameter tuning in Table [n Table [1 where we varied the learning rate and weight decay. We choose the values of learning rate and weight decay by manual tuning. We perform the trial of hyperparameter tuning once, and F1 scores of extractive models using the handcrafted validation dataset are monitored during the trial. We then took the highest-scoring set of hyperparameters and fine-tuned the extractive models one more time using the whole training dataset. In Table T2 and Table [3 we present the final set of hyperparameters selected for Entity extraction and Question Answering tasks respectively using the Heuristic based dataset. In Table T and Table [5 we present the final set of hyperparameters selected for Entity extraction and Question Answering tasks respectively using the T5 based dataset.
Hyperparameter number of epochs batch size learning rate Weight decay learning rate optimizer adam epsilon
Search space
64
[1e- - 6, le - 4]
[0.01, 0.3]
Adam
le-8
sequence length
64
Table 10: Hyperparameter search space for Entity Extraction models
Hyperparameter number of epochs batch size learning rate Weight decay learning rate optimizer adam epsilon
Search space
1
32
[1e- 6, le
4]
[0.01, 0.3]
Adam
le-8
max sequence length
128
max answer length
30
Table 11: Hyperparameter search space for Question Answering models
Flight Delay
Online Delivery
Student Performance
Universal
Methods
Weight
Learning
Weight Decay
Learning
Weight Decay
Learning
Weight
Learning
Decay
Rate
Rate
Decay
Rate
BERT
0.25
7.5c-5
0.01
7.5e-5
0.1
lc-5
0.15
5c-5
XL.Net
0.01
7.5e-5
0.15
le-4
0.3
lc-4
0.05
7.5e-5
RoBERTa 0.25
le-5
0.25
le-5
0.2
le-5
0.3
5e-5
Albert
0.01
5e-5
0.3
le-5
0.05
lc-4
0.15
Ic-5
Table 12: Final Model Hyper-parameters for Question Answering task using Heuristic based artificial data
A.7 Experimental Setups
In Table 16 we present the estimated time to fine-tune all combination of models we present in the case study. We have used one Nvidia Quadro RIX 5000 GPU and reported the time needed to fine-tune the models in our case study. As universal models use artificial data for all three dataset, we present them in first four rows Table [6] As the ensemble models uses fine-tuned EE and QA models' output and perform ensemble operation on them, do not specifically fine-tune those models.
24

---

Published in Transactions on Machine Learning Research (03/2024)
Flight Delay
Online Delivery
Student Performance
Universal
Methods
Weight
Learning
Weight Decay
Learning Rate
Weight Decay
Learning Rate
Weight Decay
Learning
Decay
Rate
Rate
BERT
0.25
7.5e-5
0.2
5e-4
0.3
5e-5
0.1
7.5e-5
XLNet
0.1
5e-5
0.01
7,5c-5
0.01
7,5c-5
0,3
5o-5
RoBERTa
0.25
5c-5
0.15
lc-4
0.15
lc-4
0.25
7.5c-5
Albert
0.1
5e-5
0.3
le-5
0.3
Se-5
0.3
le.4
Table 13: Final Model
for Entity Extraction task using Heuristic based artificial data
Flight Delay
Online Delivery
Student Performance
Universal
Methods
Weight Decay
Learning
Weight Decay
Learning Rate
Weight Decay
Learning
Weight Decay
Learning
Rate
Rate
BERT
0.3
7.5e-5
0.05
5e-5
0.2
0.0005
0.3
0.0005
XLNet
0.01
0.0001
0.25
5e-5
0.15
5e-5
0.3
7.5e-5
RoBERTa 0.25
5e-5
0.15
le-5
0.3
5c-5
0.01
0.0001
Albert
0.25
0.0001
0.3
le-5
0.3
7.5e-5
0.01
7.5e-5
Table 14: Final Model Hlype--paaameter for Question Answering task using TS generated artificial data
A.8 Reproducibility
For all reported experimental results:
Flight Delay
Online Delivery
Student Performance
Universal
Methods
Weight
Learning
Weight Decay
Learning
Weight
Learning
Weight Decay
Learning
Decay
Rate
Rate
Decay
Rate
BERT
0.05
7.5e-5
0.2
7.5e-5
0.01
0.0001
0.05
7.5e-5
XLNet 0.3
7.5c-5
0.25
7.5c-5
0.05
0.0001
0.1
7.5c-5
RoBERTa 0.15
0.0001
0.05
0.0001
0.1
7.5c-5
0.25
0.0001
Albert
0.05
7.5e-5
0.01
7.5e-5
0.15
0.0001
0.25
5e-5
Table 15: Final Model Syper-parameters for Entity Extraction task using T5 generated artificial data
Entity Extraction
Question Answering
Dataset
Model
Universal
Custom
Universal
Custom
Heuristic
T5
Heuristic
T5
Heuristic
T5
Heuristic
T5
Bert
105
120
100
122
605
780
570
792
FD
RoBERTa
90
123
105
124
600
820
540
804
XLNet
160
150
150
155
1860
2797
1800
2889
Albert
120
140
120
137
610
801
590
817
Bert
100
121
675
1248
OD
RoBERTa
104
700
1996
XLNet
157
154
2160
4082
Albert
121
135
690
1251
Bert
103
121
555
859
SP
RoBERTa
100
123
560
856
XLNet
160
159
1800
2781
Albert
125
585
837
Table 16: Time required (in seconds) for fine-tuning on all four attributes extraction task using EE/QA and variations of Ensemble methods.
25

---

Published in Transactions on Machine Learning Research (03/2024)
A clear description of the mathematical setting, algorithm, and/or model: Mathematical design and problem formulation is discussed in Section 3.2 We have discussed how we do the artificial dataset generation in Algorithm 2 and the process of slot filling with user in the end loop in Algorithm
Submission of a zip file containing source code, with specification of all dependencies, including external libraries, or link to such resources (while still anonymized) Description of computing infrastructure used: A zip file will be submitted in the submission panel along with the manual.
The average runtime for each model algorithm (e.g., training, inference, etc.), estimated energy cost: Average runtime is discussed in Appendix A.7
Number of parameters in each model: parameter count of each model we used is discussed in Appendix A.4
Corresponding validation performance for each reported test result: We artificially generate data to train out extractive model and test those model on handcrafted dataset described in Section [5.1 As creation of handcrafted dataset is very costly so, we have used all of human generated dataset as out test set hence, we do not have validation dataset for case study.
Explanation of evaluation metrics used, with links to code: Explanation of evaluation metric used is given in Section Link to code is not applicable as we implement it ourself. This code will be included inside the zip file that will submit in the paper submission panel.
For all experiments with hyperparameter search:
The exact number of training and evaluation runs: details in Appendix A.6
Bounds for each hyperparameter: details in Appendix A.6
Hyperparameter configurations for best-performing models: details in Appendix A.6
Number of hyperparameter search trials: details in Appendix A.6
The method of choosing hyperparameter values (e.g., uniform sampling, manual tuning, etc.) and the criterion used to select among them (e.g., accuracy): details in Appendix A.6
Summary statistics of the results (e.g., mean, variance, error bars, etc.): details in Appendix A.6
For all datasets used:
Relevant details such as languages, and number of examples and label distributions: details in Appendix A.3
Details of train/validation/test splits: details in Appendix A.3
Explanation of any data that excluded, and all pre-processing steps: details in Appendix A.3
A zip file containing data or link to downloadable version of the data: Not applicable, as we generate data artificially.
For new data collected, a complete description of the data collection process, such as instructions to annotators and methods for quality control: details in Appendix A.3
A.9 Statistical Significance Tests
We also report the average of MRR numbers across all embeddings and present them in Table 17, where we can clearly see that extractive models works better than vanilla model (without using any extraction) in almost all the cases. We further conducted the Wilcoxon Sign-Ran Ranked Test for all pairs of models (presented in appendix, Table [8). We took the MRR score for all the data-sets and slots and concatenated them into flat vector for each model conduct the statistical analysis. We found that the observed difference between the MRR scores of Vanilla method and any other extractive model (EE/QA) statistically significant.
26

---

Published in Transactions on Machine Learning Research (03/2024)
Aggregator
Target Attribute
Filter Operation
Filter Attrihute
Models
FD
OD
SP
FD
OD
SP
FD
OD
SP
FD
OD
SP
Vanilla
0,541
0.667
0.845
0.37
0.342
0.416
0,480
0.456
0.387
0.437
0.246
0,483
Net
0.988
0.979
0.986
0.890
0.85
0.797
0.906
0.559
0.293
0.933
0.754
0.783
0.849
0.799
0.775
0.473
0.470
Heuristic
RoBERTa
0,988
0,986
0.98
0.839
0.802
0.913
0.430
0.455
0.834
0.755
0.307
Albent
0,988
0,982
0,986
0.934
0,921
0.792
0.892
0.584
0.290
0.23
0.777
0.780
XI Net
0.987
0.980
0.987
0.973
0.901
0.804
0.839
0.556
0.391
0.921
0.882
0.649
Bert
0.992
0.982
0.987
0.984
0.759
0.624
0.900
0.459
0.312
0.914
0.718
0.599
T5
RoBERIa-EE
0.989
0.979
0.987
0.722
0.313
0.433
0.508
0.374
0.325
0.481
0.556
0.316
RoBERRiaQQ
0.987
0.977
0.787
0.669
0.922
0.509
0.504
0.677
Table 17: Average MRR of all model/embeddings.
Heuristic
TS RoBERTA.QO QA
vanilla
XLNet
RoBERTa
Alberi
RoBERTa
XLNeI
vanilla
1.0
XLNet
9.84c-9
1.0
Bert
3333--9
0.4003
1.0
Heuristic
RoBERTa
2.22077
0.1630
0.01383
1.0
Albert
130c-8
0,7661
0.5451
0.4207
1.0
RoBERTa-EE
0.00103
3.82c-6
4.50p-7
1.98c-5
1.94c66
1.0
RoBERTa-QA
4.73c-9
0.2636
0.329
0.4448
0.5182
1888e7
1.0
15
XLNNtt
9846-9
0.2955
0.04332
0,02603
0.3797
9.44c-7
0.2931
1.0
Bie
9.262.9
0.1119
0.2184
0.4417
0,1108
2.22=
0.1858
0,03285
1.0
Table 18: p-values from Wilcoxon Sign-Ranked Test
A.10
Qualitative Study of Bert-based Model
We include some examples of how our proposed system interact with user in Table 19 20 In those experiments, we use
Bert Transformer-base Language Model fine-tuned on Entity Extraction task (Table 19 and Question Answering task (Table 20
Turn
Conversation
System setup: Example No.
model: BERT, embedding: use, similarity function: cosine, schema: student performance
User:
Predict the total number of school absences where the student's family size is greater than 3 for
next year.
agent:
Thank you! It looks like the attribute of your goal prediction task is absences, is that correct? yes.
User:
agent:
Thanks for confirming! Let's focus on the aggregator of your goal prediction task. It looks like for the aggregator you want to use TOTAL, is that correct?
User:
yeah that correct.
agent:
Thanks for confirming!lee's focus on the filter of your goal prediction task. It looks like for the filter you want to use famsize, that correct?
User:
yes.
agent:
Thanks for confirming!I.co ..e''s focus on the filter operation of your goal prediction task. It looks like for the filter Operation you want use NONE, that correct? no.
User:
agent:
Okay, so then the filter operation of your goal prediction task should be NOT?
User:
agent:
Okay, so then is the filter operation of your goal prediction task should be IS LESSS THAN?
User:
no.
agent:
Okay, so then is the filter operation of your goal prediction task should be IS MORE THAN?
Continued on next page
27

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page Conversation
Turn
User:
yes.
agent:
Thanks for confirmig!Soo this is the problem you are trying to solve:
Predict the attribute: absences
aggregator: total
filter: famsize
filter operation: more than Is that correct?
User:
that seems all right.
Example No.
2
System setup:
model: BERT, embedding: use, similarity function: cosine, schema: online delivery
User:
predict the average number of calls made by delivery captain where the delivery persons ability is not good for the next month Thank you! It looks like the attribute of your goal prediction task is Number of calls, is that correct?
agent:
User:
yes
agent:
Thanks for confirming!Letts focus on the aggregator of your goal prediction task. It looks like for the aggregator you want to use AVERAGE, is that correct?
User:
yes
agent:
Thanks for confirming 'Lt's focus on the filter of your goal prediction task. It looks like for the filter you want to use Delivery person ability, is that correct? yes.
User:
agent:
Thanks for confirming!Ledt LLet's focus on the filter operation of your goal prediction task. It looks like for the filter operation you want to use NONE, is that correct?
User:
no.
agent:
Okay, so then is the filter operation of your goal prediction task should be IS BETWEEN?
User:
still incorrect. Please check it again.
agent:
Okay, so then the filter operation your goal prediction task should be IS LESS THAN?
User:
nope
agent:
Okay, so then is the filter operation of your goal prediction task should be NOT?
User:
wrong again. Okay, so then is the filter operation of your goal prediction task should be IS EQUAL TO?
agent:
User:
yes
agent:
Thanks for confirming!oo this is the problem you are trying to solve:
Predict the attribute: Number of calls
aggregator: AVERAGE
filter: Delivery person ability
filter operation: IS EQUAL TO
Example No.
3
System setup:
model: BERT, emhedding: use, similarity function: cosine, schema: online delivery
User:
predict the total delay of delivery person getting assigned where the customers location on google map is not accurate for the next month
agent:
Thank you! looks like the attribute of your goal prediction task is Delay of delivery person getting assigned, is that correct? yes
User:
agent:
Thanks for confirming! .le's focus the aggregator of your goal prediction task. It looks like for the aggregator you want to use TOTAL,, is that correct?
User:
you are correct
agent:
Thanks for confirming!Let's 's focus on the filter of your goal prediction task. It looks like for the filter you want to use Google Maps Accuracy, is that correct? yes that is.
User:
agent:
Thanks for confirming!l.ct focus on the filter of your goal prediction task. It looks like for the filter you want to use Google Maps Accuracy, is that correct?
User:
yeah.
agent:
Thanks for confirming!I.c .Lett' focus the filter operation of your goal prediction task. It looks
like for the filter operation you want to use NONE, is that correct? Continued on next page
28

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
User:
no.
agent:
Okay, so then the filter operation of your goal prediction task should be IS EQUAL TO?
User:
no.
agent:
Okay, so then the filter operation your goal prediction task should be HAS?
User:
agent:
Okay, so then is the filter operation of your goal prediction task should be IS LESS THAN?
User:
no.
agent:
Okay, so then is the filter operation of your goal prediction task should be IS GREATER THAN!
User:
no.
agent:
Okay, then the filter operation of your goal prediction task should be NOT? no.
User:
agent:
Okay, so then the filter operation your goal prediction task should be NOT? yes that's correct.
User:
agent:
Thanks for confirming !So this is the problem you are trying to solve: Predict the attribute: Delay of delivery person getting assigned
aggregator: TOTAL
filter: Google Maps Accuracy
filter operation: NOT
Example No.
4
System setup:
model: BERT, embedding: use, similarity function: cosine, schema: flight delay
User:
Predict total arrival delay for each destination airports whose flights start next Sunday
agent:
Thank you! It looks like the attribute of your goal prediction task is ARRIVAL DELAY, is that correct?
User:
that is correct
agent:
Thanks for confirming! Let's focus on the aggregator of your goal prediction task. It looks like for the aggregator you want use TOTAL, that correct? yes
User:
agent:
Thanks for confirming!Lc. focus on the filter of your goal prediction task. It looks like for the filter you want to use AIRLINE DELAY, is that correct?
User:
no i don't 't want any filter filter of your goal prediction task is NONE? yes
agent:
User:
agent:
Thanks for confirming!Lee Let's focus on the filter operation your goal prediction task. It looks like for the filter operation you want to use NOT, is that correct? no i don't It want any filter
User:
agent:
filter operation of your goal prediction task NONE? yes that's correct
User:
agent:
Thanks for confirming!S0 Soo this is the problem you are trying to solve:
Predict the attribute: ARRIVAL DELAY
aggregator: TOTAL
filter: NONE
filter operation: NONE
Table 19: Qualitative study of User Interaction with proposed agent BERT Transformer based Language Model which was fine-tuned on Entity extraction
task.
Turn
Conversation
System setup:
model: BERT, embedding: use, similarity function: cosine, schema: student performance
Example No.
User:
Predict the total number of school absences where student's family size greater than 3 for
next year.
agent:
Thank you! looks like the attribute of your goal prediction task is absences, correct? Continued next page
29

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page Conversation
Turn
User:
yes.
agent:
Thanks for confirming!Let'' focus the aggregator of your goal prediction task. It looks like for the aggregator you want to use TOTAL, is that correct?
User:
yeah that is correct.
agent:
Thanks for confirming! Let's focus the filter of your goal prediction task. It looks like for the filter you want to use schoolsup, is that correct?
User:
no.
agent:
filter of your goal prediction task is GI, am correct?
User:
its family size
agent:
filter of your goal prediction task the famsize?
User:
yes.
agent:
Thanks for confirming! LLet's focus the filter operation of your goal prediction task. It looks
like for the filter operation you want to use NOT, is that correct?
User:
no.
agent:
Okay, so then is the filter operation of your goal prediction task should be IS MORE THAN? yes.
User:
agent:
Thanks for confirmingSoo this is the problem you trying to solve:
Predict the attribute: absences
aggregator: total
filter: famsize
filter operation: more than Is that correct?
User:
that seems all right.
Example No.
2
System setup:
model: BERT, embedding: use, similarity function: cosine, schema: online delivery
User:
predict the average number of calls made by delivery captain where the delivery persons ability is not good next month Thank you! It looks like the attribute of your goal prediction task is Number of calls, is that correct?
agent:
User:
yes
agent:
Thanks for confirming!Lett focus on the aggregator of your goal prediction task. It looks like for the aggregator you want to use AVERAGE, is that correct?
User:
yes
agent:
Thanks for confirming Let's focus on the filter of your goal prediction task. It looks like for the filter you want Delivery person ability, is that correct? yes.
User:
agent:
Thanks for confirming!L.co focus the filter operation of your goal prediction task. It looks like for the filter operation you want to use NOT, is that correct?
User:
no its should he equal
agent:
filter operation of your goal prediction task the IS EQUAL TO? yeah there you go.
User:
agent:
Thanks for co this the problem you are trying to solve:
Predict the attribute: Number of calls
aggregator: AVERAGE
filter: Delivery person ability filter operation: IS EQUAL TO
Example No.
3
System sctup:
model: BERT, embedding: use, similarity function: cosine, schema: online delivery
User:
predict the total delay delivery person getting assigned where customers location on google map is not accurate for the next month Thank you! looks like the attribute of your goal prediction task is Delay of delivery person getting assigned, is that correct? yes
agent:
User:
agent:
Thanks for confirming! Let's focus the aggregator of your goal prediction task. It looks like for the aggregator you want use TOTAL, is that correct? Continued
User:
you are correct
page
30

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
agent:
Thanks for confirming Let's focus on the filter of your goal prediction task. It looks like for the filter you want Google Maps Accuracy, is that correct?
User:
yes that is.
agent:
Thanks for confirming!I.co !Ieti' focus on the filter operation your goal prediction task. It looks like for the filter operation you want to use NOT, is that correct?
User:
yes that is correct.
agent:
Thanks for confirming!So this is the problem you are trying solve:
Predict the attribute: Delay of delivery person getting assigned
aggregator: TOTAL
filter: Google Maps Accuracy
filter operation: NOT
Example No.
System setup:
model: BERT, embedding: use, similarity function: cosine, schema: flight delay
User:
Predict total arrival delay for each destination airports whose flights will start next Sunday
agent:
Thank you! It looks like the attribute of your goal prediction task is ARRIVAL DELAY, is that correct?
User:
that is correct
agent:
Thanks for confirming! Let's focus on the aggregator your goal prediction task. It looks like for the aggregator you want use TOTAL, is that correct?
User:
yes
agent:
Thanks for confirming!Les Let's focus on the filter of your goal prediction task. It looks like for the filter you want to use FLIGHT NUMBER, is that correct?
User:
don't 't want any filter filter of your goal prediction task is NONE? yes
agent:
User:
agent:
Thanks for confirming!Let Let's focus on the filter operation of your goal prediction task. It looks like for the filter operation you want to use NOT, is that correct?
User:
don't want any filter
agent:
filter operation of your goal prediction task is NOT, am I correct?
User:
no
agent:
Okay, then the filter operation of your goal prediction task should be HAS?
User:
agent:
filter operation of your goal prediction task the NONE?
User:
yes that's 's correct
agent:
Thanks for confirming So this is the problem you are trying to solve:
Predict the attribute: ARRIVAL DELAY
aggregator: TOTAL
filter: NONE
filter operation: NONE
Table 20: Qualitative study of User Interaction with proposed agent BERT as Transformer based Language Model which was fine-tuned on Question Answering
task.
A.11 Qualitative study of Open-Domain Dialog Systems
The full qualitative study of ChatGPT on this ambitious task is presented in Table 21 given different system instruction and see how the performance of ChatGPT varies with level of information provided to it before the prompt. From the examples we can see more informative system setup is effective.
24 In each of the table we have
31

---

Published in Transactions on Machine Learning Research (03/2024)
Turn System setup:
Conversation
You are a Machine Learning expert who will help a small business owner by formulating an interesting problem through conversation. You should perform a slot-filling task based on the conversation and use those slot information to formulate time series forecasting problem for the user. The slots are aggregation operation, target attribute. filter attribute, and filter operation. you have to infer the slot-values from the conversations.
Example No.
User:
Predict the total number of school absences where the student's family size is greater than 3 for next year.
chatGPT:
Great! To start, I need some additional information to formulate precise problem. Can you tell me more about the data you have regarding the school absence? I do not have the data. Please use the above information for prediction. Okay, I will need to identify the slots in the given information to formulate the problem precisely. The Aggregation operation: 'total number'
User:
chatGPT::
Target attribute: 'school absence'
Filter attribute: 'family size'
Filter operation: "greater than 3'
User:
Thank you.
chatGPT:
You're 'r're welcome! Feel free to let me know if you have any more questions or if there's anything else I can help you with.
Example No.
2
User:
predict the average number of calls made hy delivery captain where the delivery persons ability is not good for the next month Sure, I can help you with that. To clarify, you looking to predict the average number of calls made by delivery captains for the next month, but only when the delivery person's 's ability is poor? And could you please provide me with more information on how you define poor ability (i.e., what attributes are associated with poor ability)? I do not have any more information
chatGPT:
User:
chatGPT:
Okay, I can work with that. In that case, we can formulate the problem as time series forecasting task where the target attribute is the average number of calls made by delivery captains, and the filter attribute is the delivery person's 's ability. Specifically, we want to predict the average number of calls made by delivery captains for the next month, but only when the delivery person's 's ability is poor. To do so, will need to gather data on the average number of calls made by delivery captains for each month, as well as some measure of the delivery person's 's ability (i.c., numerical score rating). Once we have this data, can apply filter operation to only consider the data points where the delivery person's 's ability score is below a certain threshold (which will determine based on the data). Finally, we apply aggregation operation to compute the average number of calls made by delivery captains for each month, after filtering for poor delivery person ability, and use this historical data to train time series forecasting model that will predict the average number of calls for the next month, given the poor delivery person ability filter.
User:
can you list all the slot- values here
chatGPT:
Sure, here are the identified slot-values based on our conversation:
Aggregation operation: Average
Target attribute: Number of calls made by delivery captains
Filter attribute: Delivery person's 's ability
Filter operation: Poor ability
User:
filter operation is incorrect. Please check it.
chatGPT:
apologize for the mistake. Can you please correct me? What would be the correct filter operation for the problem we discussed?
User:
do not have the correct answer. Read the original prompt.
chatGPT:
You're right. 1 apologize for the confusion. Based on the prompt, the correct filter operation would be "delivery persons ability not good".
User:
still incorrect. Please check it again.
chatGPT:
I'm sorry for the mistake. Let me verify this again for you. Could you please clarify what the right filter operation is?
User:
Hint: filter operation is conditional check operation. Now, check for the correcting the prompt Continued next page
32

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn chatGPT:
Conversation
Thank you for the hint. Based on the prompt, the correct filter operation for the problem would be a conditional check operation, specifically "where the delivery persons ability is not good". This means that we would be filtering data to only consider cases where the delivery person's s ability is below a certain threshold, which we would need determine based on data.
Example No.
3
User:
predict the total delay of delivery person getting assigned where the customers location on google map is not accurate for the next month
chatGPT:
Sure, I can help you with that, Can you please tell me more about your business and what kind of industry you are in? Also, how do you currently manage your dcliveries? do not have any more information.
User:
chatGPT:
Oh, I sec. Let me clarify few things, when you say "total delay of delivery person getting assigned", do you mean
the time taken to assign a delivery person to a customer's order? And by "where the customer's location Google map not accurate", do you mean the location is not mapped correctly or that it is unclear where exactly the location is?
User:
can you list the slot-values here
chatGPT:
Sure, based on our conversation, I have identified following slots:
Aggregation operation: Total
Target attribute: delay of delivery person getting assigned
Filter attribute: customer's location Google map accuracy
Filter operation: inaccurate
So the complete problem statement would be:
"Forecast the total delay of delivery person getting assigned, where the customer's 's location on Google map is inaccurate for the next month."
User:
Thank you.
chatGPT:
You'le welcome! Feel free let me know you have any more questions or if there's anything else I can help you with.
Example No.
User:
Predict total arrival delay for each destination airports whose flights will start next Sunday Sure, we can work on that. Before we proceed, can you provide me some more information like which airline's flights are considering and which geographical region of the world? Also, can you tell me are we interested in predicting delays for a specific time period, such as a single day week, or longer period perhaps multiple weeks month? do not have any additional information
chatGPT:
User:
ChatGPT:
Alright, let me see understood correctly. We want to predict total arrival delay for each destination airport for flights that will start next Sunday. But we do not have any information about the airline or the geographical region. And we are interested in predicting delays for a specific day, Sunday. Is that correct? yes
User:
chatGPT:
Great. So based this slot-filling conversation, can formulate the following time series forecasting problem:
Problem Statement: Predict the total arrival delay for each destination airport for flights that will
start next Sunday. Slots:
Aggregation Operation: Total
Target Auribute: Arrival Delay
Filter Attribute: Destination Airport
Filter Operation: Next Sunday's 's flights
User:
filter attribute and filter operation is incorrect
chatGPT:
apologize for the mistake. Can you please tell me what is the correct filter attribute and filter operation? With that information, I will update the problem description for you. no. i didn't have any information
User:
chatGPT:
sec. In that case, can formulate the problem without the filter attribute and filter operation slots as follows:
Continued next page
33

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
Problem Statement: Predict the total arrival delay for each destination airport for flights that will
next Sunday, Slots:
Aggregation Operation: Total
Target Attribute: Arrival Delay
Time Period: A specific day, Sunday
User::
thank you very much
Table 21: Qualitative study of User Interaction with ChatGPT using the same
system setup described in the beginning.
34

---

Published in Transactions on Machine Learning Research (03/2024)
Turn
Conversation
System setup:
You are a personal data scientist who will help small business owner by formulating an interesting problem through conversation. You should perform a slot-filling task based on the conversation and use those slot information to formulate time series forecasting problem for the user. The slots are aggregation operation, target attribute. filter attribute, and filter operation. you have to infer the slot-values from the conversations.
Example No.
User:
Predict the total number of school absences where the student's family size is greater than 3 for next year.
chatGPT:
Thank you for providing the time series forecasting problem regarding school absence. Now, need some more information to infer the slot-values from the given conversation. Can you please provide me with the conversation between you and the small business owner that you had regarding this topic, so that can identify the slot-values and help you with the task? do not have any other information between me and small business owner. Alright, based on the forecasting problem that you have provided, I can infer the following slot-values:
User:
chatGPT:
Aggrogation operation: "sum" (since we want to predict the total number of school absences)
Target attribute: "school absence" (since this is what we want predict)
Filter attribute: "family size" (since we want to consider only those instances where the family size is greater than 3)
Filter operation: ">" (since we want to filter those instances where the family size is greater than 3)
Based the inferred slot-values, the time series forecasting problem would be:
Forecast the sum of school absences where the family size is greater than 3, for the next year.
User:
That is correct. Thank you. You't Te welcome! Feel free to let me know if you have any more questions.
chatGPT:
Example No.
User:
Predict the maximum first period grade of students where student has partaken in extra-curricular activities
chatGPT:
I'm sorry, but I cannot perform this task as you have provided time series forecasting problem without any conversation or slot information. To perform the task, please provide the necessary details.
User:
have no new information regarding the problem statement. Please perform the task using the provided information
chatGPT:
I'm sorry, but to perform a time series forecasting problem, we need time series dataset along with the slots that specify the aggregation operation, target attribute, filter attribute, and filter operation for data preprocessing. Without this information, it is difficult to formulate suitable time series forecasting model. Can you please provide more information about the problem statement?
User:
do not have more information. please use the original prompt to find the solution. apologize for the confusion, but the original prompt does not provide any conversation or slot information perform the slot filling task. The prompt only mentions time series forecasting problem without any context. Can you please provide more details or context about the problem statement so that I can assist you hetter? ok.
chatGPT:
User:
Example No.
User:
predict the average number of calls made by delivery captain where the delivery persons ability is not good for the next month.
chatGPT:
Sure, you provide some more details to help understand the context better? What kind of business are you running. and what is the role of the delivery captain and delivery persons? Also. can you provide the conversation that led this query help me fill in the slots?
User:
do not have any other information
chatGPT:
Alright, based on the limited information available, let me try to fill in the slots as follows:
Aggregation operation: "average"
Target attribute: "number of calls made by delivery captain"
Filter attribute: "delivery person's 's ability"
Filter operation: "not good"
Continued on next page
35

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn Example No.
Conversation
User:
predict total delay delivery person getting assigned where customers location on google
map is not accurate for the next month
chatGPT:
Let me clarify the slots hased on the statement you provided.
Aggregation operation: Summation
Target attribute: Delay of delivery person getting assigned
Filter attribute: Accuracy of customer location on Google Maps
Filter operation: Inaccurate
So, the slot values
follows:
Aggregation operation: Summation
Target attribute: delay of delivery person getting assigned
Filter attribute: accuracy of customer location on Google Maps
Filter operation: inaccurate To formulate time series forecasting problem, we need historical data about the delay of delivery person getting assigned inaccurate locations on Google Maps. Do you have any data that we can use for this purpose you got right
User:
Example No.
User:
Predict total arrival delay for each destination airports whose flights will start next Sunday I'm sorry, but think there's some misunderstanding. I'm supposed to perform a slot filling task based conversation with small business owner and formulate time series forecasting problem using those slot information. Could you please provide me with some information about the conversation?
chatGPT:
User:
do not have more information
chatGPT:
No problem, I'll give example conversation illustrate how slot filling be performed: Small Business Owner: Hi, can help me analyze my sales data for the past year?
Data Scientist: Sure, what specific aspect do you want me to focus on?
Small Business Owner: want to know how the sales have been for our top product.
Data Scientist: Alright, what is name of your top product?
Small Business Owner: It's called Widget A.
Data Scientist: Got it. Do you have any specific time period you want me to analyze?
Small Business Owner: Yes, want to the sales trend for Widget A for the past 12 months. From this conversation, we can infer the following slot information:
Aggregation operation: None (no aggregation requested)
Target attribute: Sales
Filter attribute: Product Name
Filter operation: Equal to Using this information, we can formulate time series forecasting problem as follows: "Forecast the sales trend for Widget A for the next 12 months, given the sales data from the previous 12 months." This problem involves time series forecasting, where we will use historical sales data to predict future sales for specific product.
User:
the above slot filling is incorrect. please redo the slot-filling task
chatGPT:
My apologies for the confusion earlier. Could you please provide with conversation between you and the small business owner so that can perform the slot filling task? Continued on next page
36

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
User:
I do not have any such information
chatGPT:
Not problem. I'll provide example conversation and perform the slot filling task:
Small Business Owner: Hi, can you help me with my inventory management?
Data Scientist: course, what specifically you need help with?
Small Business Owner: need to know which products are selling the most and which ones are not doing so well.
Data Scientist: Understood. For which time period do you want me to analyze the sales data? Small Business Owner: Please analyze data
past months.
From this conversation,
infer the following slot information:
Aggregation operation: Maximum count (to identify the top-selling products)
Target attribute: Sales
Filter attribute: Time
Less than equal to Using this information, we can formulate a time series forecasting problem as follows: "Forecast the sales trend for the top-selling products and the non-performing products for the next months, given the sales data for the past 6 months." This problem involves time series forecasting, where we will use historical sales data to predict the sales trend for the top-selling and por-perforning products.
Filter operation:
User:
the slot- tot-filling is incorrect.
the iteration continues until quits.
Table 22: Qualitative study of User Interaction with ChatGPT using the same system setup described in the beginning.
37

---

Published in Transactions on Machine Learning Research (03/2024)
Turn
Conversation
System setup:
You are a personal data scientist who will help small business owner by formulating an interesting problem through conversation. You should perform slot filling task based on the conversation and use those slot information to formulate time series forecasting problem for the user. The slots are aggregation operation, target attribute, filter attribute and filter operation. you have to infer the slot-values from the conversations. The value of the slots are: (predefined values for each possible slots are given here.)
Example No.
User:
Predict total arrival delay for cach destination airports whose flights will start next Sunday Alright. To clarify, you are looking for a time series forecasting problem to predict the total arrival delay for each destination airport for flights that will begin on next Sunday. Is that correct? yes.
chatGPT:
User:
chatGPP::
Great! So to formulate a time series forecasting problem, we'll Illl need to define the time interval we're looking Based on the given slots, we can use the date or day of the week the time variable.
For this problem, we'll elll use the date of flight departure as the time variable, and since we're only interested in flights departing next Sunday, we can filter the dataset by the "date" attribute using the ""quull" filter operation.
We'll also use the "arrival delay" attribute as the target variable, and the "tail numberr and "destination airport" attributes as our categorical variables. To aggregate the data, we'll use the "total" aggregation operation, since we're interested in the total arrival delay for each destination airport.
So our final problem statement is: Given flight data for all flights departing next Sunday, predict the total arrival delay for each destination airport. The categorical variables used for prediction are "tail number" and "destination airport". correct.
User:
Example No.
User:
predict the average air system delay for flights which will start from Atlanta International Airport which will have scheduled departure hour after 4 PM is central time and will start within next week
chatGPT:
To clarify, do you want the predicted average air system delay for all flights meeting the criteria you specified, or for specific flight or set of flights? Additionally, do you have access to historical data on flights from Atlanta International Airport to use for the time series forecasting?
User:
do not have access to historical data
chatGPT:
Without access to historical data, it would not be possible perform time series forecasting of the average air system delay for flights meeting the criteria you specified. However, based on the information provided, can still generate estimate of the average air system delay for those flights.
To do so, would need gather data recent flights departing from Atlanta International Airport during the specified time range and calculate the average air system delay for those flights. One way to gather this data would be to use an API such as the FlightStats API the Aviationstack API to retrieve real-time flight data for flights departing from Atlanta International Airport.
Once the data has been gathered, we would then need to filter the data based on the specified criteria (scheduled departure hour after PM in central time, departing within the next week). We could then calculate the average air system delay for the filtered data to obtain our estimate. The aggregation operation for this task would be 'average', : the target attribute would be 'air system delay', and the filter attributes would be 'scheduled departure hour" and "date'. The filter operation would be 'greater than' or 'after' for the 'scheduled departure hour' attribute and 'between' or 'about" for the 'date' attribute.
User:
correct. Thank you.
chatGPT:
You're welcome! you have any further questions or need additional assistance, feel free to ask.
Example No.
3
Continued on next page
38

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation Predict the total number of school absence where student's family size is greater than for next year
User:
chatGPT:
To formulate a time series forecasting problem, we would need historical data of the number school absences for past few years. Assuming we have that data, we can use the following slot information to filter the relevant data:
Aggregation operation:
Target attribute: absences
Filter attribute: famsize
Filter operation: greater than
Filter value: 3
filter the historical data to only include records where the student's 's family size is greater than and calculate the total number of school absences for each year. We then this data absences for the next year. The model can use various time series forecasting techniques such ARIMA exponential smoothing to make predictions based on the patterns observed in the historical data.
Using this information,
time series model forecast the total number of school
User:
there no filter value
chatGPT:
Oh, apologize for the oversight. To complete the slot filling task, we will need filter value for the "family size" attribute. Let's assume that the filter value is "4" indicating students with a family size greater than 3 (i.e., students who have 4 more family members in their household). So, the complete slot information would be:
Aggregation operation: total
Target attribute: absences
Filter attribute: famsize
Filter operation: greater than
Filter value: 3
We can use this slot information to filter the historical data and make predictions for the total number of school absences for students with family size greater than for the next year. did asked for filter value
User:
chatGPT:
apologize for the confusion. Given the updated slot information, the corresponding time series forecasting problem would be predict the total number of school absences for students with family size greater than 3 for the next year. To do this, we would extract the relevant information from the historical data, apply any necessary data cleaning and preprocessing steps, then use time series forecasting model to project the expected number of school absences for the next year. The model could built using techniques such as ARIMA, exponential smoothing or machine learning algorithms like LSTM or GRU, depending on the data and the specific requirements of the problem.
User:
can you show the complete slot information Sure, here the complete slot information:
chatGPT:
Aggregation operation: total
Target altribute: absences
Filter attribute: famsize
Filter operation: greater than
Filter value:
Using this slot information, we would extract the relevant data for students with family size greater than 3, apply any necessary preprocessing such as missing data imputation and feature scaling, and build time series forecasting model to predict the total number of school absences for the next year. The model would trained on the historical data evaluated based on performance holdout dataset. If the model's performance is satisfactory, it could be used to make predictions for future years as well.
Continued on next page
39

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
User:
can you remove filter value as i did not ask for it.
chatGPT:
Apologies for the confusion. Here's the updated slot information without the filter values:
Aggregation operation: total
Target attribute: absences
Filter attribute: famsize
Filter operation: greater than
Given this slot information, would filter the historical data to only include records where the student's 's family size is greater than without any specific value family size. We can then use the filtered historical data train time series forccasting model predict the total number of school absenees for next year for students with family size greater than The model be trained using various time series forecasting techniques such ARIMA exponential smoothing to make predictions based on the patterns observed in the historical data.
User:
Thank you.
chatGPT:
You're re welcome! If you have more questions or if there's anything else I can help with, feel free ask.
Example No.
4
User:
Predict the maximum first period grade of students where student has partaken in extra-curricular activities
chatGPT:
So. the aggregation operation is 'maximum', the target attribute is 'gl', the filter attribute is 'activities' and the filter operation 'has'. Therefore, we need collect data those students who have partaken in extra cura-currcclarr activities, and then predict the maximum first period grade among them using a time series forecasting model.
User:
That is correct. Thank you. You'te 're welcome! If you have any further questions or need any help in the future, feel free to ask.
chatGPT:
Example No.
4
User:
predict average numher calls made hy delivery captain where delivery persons ability is good for the next month To formulate a time series forecasting problem, we will need a time series data that has informa- tion about the average number of calls made by delivery captain and also if the delivery person's ability was not good for that particular order delivery. Assuming we have the required data, we can use the following slot information formulate the problem:
chatGPT:
Aggregation operation: average
Target attribute: number of calls
Filter attribute: delivery person ability
Filter operation: not good
Using this information, we can frame the problem statement as:
"Predict the average number of calls made by delivery captain, where the delivery person's 's ability was not good, for the next month."
The next step would be choose an appropriate time series forecasting model (such ARIMA, exponential smoothing, Prophet, etc.) and train it on the available historical data. Once the model is trained,
use to forecast the average number of calls for the next month. That is correct. Thank you.
User:
chatGPT:
You're welcome! you have any further questions or need any help the future, feel free to ask.
Example No.
Continued on next page
40

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
User:
predict the total delay of delivery person getting assigned where the customers location on google map accurate for the next month
chatGPT:
To formulate a time series forecasting problem for this scenario, we can define the following slots:
Aggregation operation: Total
Target attribute: Delay delivery person getting assigned
Filter attribute: Google maps accuracy
Filter operation: Not, Equal to
This means we want to predict the total delay of delivery person getting assigned for the next month, where the customer's s location on Google maps is not accurate.
To solve this problem need historical data on:
Delay of delivery person getting assigned
Gongle maps accuracy
Time
We also need external data (since we are predicting future delays): Forecasted customer demand (we can obtain an estimate of the number of orders expected for the next month)
Once we have this data, we can use various time-series models (such as ARIMA or SARIMA) to predict total delay of delivery person getting assigned for the next month based on historical data and forecasted customer demand.
User:
can you write the code for above prompt
chatGPT:
Unfortunately, cannot provide code implementation for the prompt you mentioned without additional information such as the specific dataset, programming language, and framework to be used.
However, can provide general outline of the steps involved in implementing the solution:
1. Collect historical data for the relevant attributes (Delay of delivery person getting assigned,
Google maps accuracy, and time).
2. Clean and preprocess the data (e.g.. remove duplicates and missing values, convert data types).
3. Use time-series model (such as ARIMA SARIMA) train on the historical data.
4. Use the trained model to generate forecast for the next month's total delay delivery person getting assigned based forecasted customer demand.
5. Evaluate the accuracy of the model using performance metrics such as the mean absolute error squared
6. Deploy the model to make real-time predictions for future delays based on new data.
Please that this is general outline, and the specific implementation may vary depending the data and the requirements of the project.
User:
Thank you for your help.
chatGPT:
You't re welcome! If you have any further questions or need any help in the future, feel free to ask.
Table 23: Qualitative study of User Interaction with ChatGPT using the same
system setup described in the beginning.
41

---

Published in Transactions on Machine Learning Research (03/2024)
Turn System setup:
Conversation
You are a Machine Learning expert who will help a small business owner by formulating an interesting problem through conversation. You should perform slot filling task based on the conversation and use those slot information to formulate time series forecasting problem for the user. The slots are aggregation operation, target attribute, filter attribute and filter operation. you have to infer the slot-values from the conversations. The value the slots are: (predefined values for each possible slots are given here.)
Example No.
User:
Predict the total number of school absences where student's family size greater than 3 for next year.
chatGPT:
formulate a time series forecasting problem based on the given information, we need to identify the relevant slots. The aggregation operation is 'total", the target attribute is 'school absence', and the filter attributes are 'family size' and 'year'. The filter operation is 'greater than'. However, don't have any historical data for 'school absence", so we cannot create time series forecasting problem.
Is there any other information regarding the data available?
User:
No. I do not have any additional information. In that case, we cannot formulate time series forecasting problem using the given information. We would need historical data on school absence and family size in order to create time series forecasting model. Without historical data, we cannot make predictions about the total number of school ahsences for next year.
chatGPT:
However, we could still use the given information to formulate other types predictive models. For example, could create classification model to predict whether student will be absent from school based on their family size and other relevant factors. Alternatively, we could create regression model to predict the number of school absences for particular student based their family size and other factors.
User:
please reinitiate the process from first message. Sure, let's start from the beginning. Hello! How can I assist you today?
chatGPT:
User:
Predict the total number of school absence where student's 's family size is greater than for next year
chatGPT:
To predict the total number of school absences where a student's 's family size is greater than 3 for year, need to formulate time series forecasting problem. In order do that, need to identify the relevant slots based on the given information.
The aggregation operation is 'total', the target attribute is 'school absence', and the filter attributes are "family size' and 'year'. The filter operation is 'greater than'. However, we don't 't have any historical data for 'school absence", so we cannot create a time series forecasting problem. Do you have any historical data for school absences and family size for previous years? If yes, please provide the data.
User:
what other data you required to create time series forecasting We would need historical data on school absences and family size in order to create a time series forecasting model. Here some additional pieces of information which would be useful: The time period for which the historical data is available (e.g. monthly, quarterly, annually)
chatGPT:
The range of dates/times covered by the data
The format in which the data is available (c.g. CSV, Excel file, database)
Any other relevant data that might have an impact on school absences, such as average income of families, student demographies, or school policies and programs related to attendance.
With this information, we can use time series forecasting methods such as ARIMA or exponential smoothing to create model that can predict future school absences based on historical trends and factors that impact attendance.
User:
chatGPT:
It seems like you didn't 't provide any prompt for me to assist you with. Is there anything related to machine learning or artificial intelligence that you would like to help you with?
Continued on next page
42

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
User:
we know that the prediction task is for next year and family size should be greater than 3. That the only data have
chatGPT:
In that case, without any historical data, it is not possible to formulate a reliable time series forecasting model to predict the total number of school absences for next year where student's s family size is greater than 3.
However, we could still use other types of predictive models to make some estimates based on the given information. For example, we might find data set from another source, such government agency or educational institution, that provides information on school absences for students with families of varying sizes. We could then that data train a classification or regression model to predict school absences for students with families of size greater than 3. Alternatiely, could survey families with children in the relevant school district gather information on family size and anticipated school absences for next year. We could use this survey data train predictive model such logistic regression decision regression make predictions based on family size and other relevant factors. okay. thank you You're welcome. If you have any other questions related to machine learning or artificial intelligence, don't 't't hesitate ask!
User:
chatGPT:
Example No.
2
User:
Predict the maximum first period grade of students where student has partaken in extra-curricular activities
chatGPT:
I'm sorry, but the given problem statement and list of slots are related to online food delivery services. Can you please provide me with more information about the business you are referring to and the data available for analysis?
User:
The problem is to predict the first period grade students who has taken part in extra-curricular activities.
chatGPT:
Great! To help me formulate the problem, could you please provide me with the following details through slot filling-
1. What is the aggregation operation you are interested in? Do you want to predict the average, total, mean, max or minimum of the first period grade of students who have taken part extra-curricular activities?
2. What is the target autribute you want to predict? it the first period grade?
3. Do you want to apply any filter on the data? If yes, what is the filter attribute and operation you want to use? For example, do you want to predict the first period grade for a specific age group of students or for students who have taken part in specific extra-curricular activities?
4. Do you have any additional information which can help in formulating the problem such as the time period the data collected from any other attributes about the students? Quits.
User:
Example No.
User:
predict the average number of calls made by delivery captain where the delivery persons ability is not good for the next month Sure, can help you with that. To clarify, the problem you are trying solve is forecasting the average numher of calls made by a delivery captain in month when the delivery person's ability is not good. Can you please provide me with the filter attribute and filter operation to narrow down the scope of the problem?
chatGPT:
Continued on next page
43

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
User:
do not have more data
chatGPT:
Alright. In that case, let's formulate the problem like this: Forecast the average number of calls made by delivery captains in month when the delivery person's is ability not good.
Problem:
To do this, we will take time series approach and use the historical data of number of calls made by delivery captains the past months as our time series data. We will focus on the following slots:
Aggregation operation: average
Target attribute: number of calls made by delivery captain
Filter attribute: delivery person ability
Filter operation: not good
We will gather data from the past 12 months, and use it to build a time series forecasting model that predicts the future average numher of calls in month where the delivery person's ability is not good.
User:
okay. thank you You're welcome. If you have any other questions related to machine learning or artificial intelligence, don't hesitate ask!
chatGPT:
Example No.
User:
predict the total delay of delivery person getting assigned where the customers location on google map not accurate for the next month
chatGPT:
Sure, can you provide me with some information to help me understand the problem better?
What is the industry or sector of the business?
Also, how does the business currently track the delay of delivery person getting assigned? And do they have historical data on the delay delivery person getting assigned situations where a customer's 's location google map is not accurate?
User:
do not any other information
chatGPT:
Alright, based the given slots, can formulate the problem follows:
Problem Statement: Forecast the total delay of delivery person getting assigned in minutes for the next month, given that customer's location on google map not accurate.
Aggregation Operation: Total
Target Attribute: Delay of Delivery Person Getting Assigned
Filter Attribute: Google Maps Accuracy
Filter Operation: Not
Time Series: Monthly
Based on the available information, we have filtered out the cases where the customer's location google map not accurate. We will use historical data the delay of delivery person getting assigned while considering only the cases where the customer's 's location on google map is not accurate. We will then this information forecast the total delay of delivery person getting assigned for the next month.
User:
remove time series slot from above prompt. did not asked for that. Sure, here is the updated prompt without the time series slot: Problem Statement: Forecast the total delay delivery person getting assigned minutes, given that the customer's location on google map is not accurate.
chatGPT:
Aggregation Operation: Total
Target Attribute: Delay of Delivery Person Getting Assigned
Filter Attribute: Google Maps Accuracy
Filter Operation: Not
Using historical data of delay of delivery person getting assigned in cases where customer's location on google map was not accurate, we can predict the total delay of delivery person getting assigned in future.
User:
that is correct. Thanks!
chatGPT:
You're 're welcome. If you have any other questions related to machine learning or artificial intelligence, don't 't hesitate to ask!
Example No.
Continued next page
44

---

Published in Transactions on Machine Learning Research (03/2024)
Table continued from previous page
Turn
Conversation
User:
Predict total arrival delay for each destination airports whose flights will start next Sunday Great! help me understand your problem better, can we first down the scope by fixing the aggregation operation, filter attribute and filter operation? will be casier to formulate the problem.
chatGPT:
User:
can you fill the slots based on the prompt provided
chatGPT:
Sure! So, based on the prompt, the "aggregation operation' 'total', the 'filter attribute" is 'day of week', and the 'filter operation' is "equal'. Since you want predict total arrival delay for each destination airport whose flights will start next Sunday, then the 'target attribute' should 'arrival delay' and the 'filter value' should be 'Sunday'. Is that correct?
User:
yes.
chatGPT:
Great! Now we have all the necessary slots filled, we can formulate the problem as follows: Problem Statement: Forecast the total arrival delay of flights for each destination airport, whose flights are scheduled for next Sunday.
Aggregation Operation: Total
Target Attribute: Arrival Delay
Filter Attribute: Day of week
Filter Value: Sunday
Does this make sense?
User:
do not need filter value, instead, need filter operation. Sorry the confusion. per the prompt, we can 'lilter operation to 'equal' since we want to predict arrival delay at each destination airport only for flights scheduled on Sunday. So, the complete problem statement is:
chatGPTT:
Problem Statement: Forecast the total arrival delay of flights for each destination airport, whose flights are scheduled for next Sunday.
Aggregation Operation: Total
Target Attribute: Arrival Delay
Filter Attribute: Day of Week
Filter Operation: Equal
User:
correct.
Table 24: Qualitative study of User Interaction with ChatGPT using the same system setup described in the beginning.
45


Reviews:
[{"review_id": "l6eAqayrPN", "reviewer": "Reviewer_rYGP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper tackles an interesting problem domain of democratizing datascience tools by leveraging advances in ML, LLMs and NLP.\n* The paper prudently notes that the current generation of decoder-only models like ChatGPT might be quite performant in this domain, and identify this as an avenue for future work.\n\n**Weaknesses**\n* The paper claims to introduce the concept of forecast utterance, but it doesn’t seem well-explained or novel. It isn't different from a regular input to an NLP task. So there's no added relevance for introducing it.\n* The paper claims a zero-shot learning approach. But it uses a heuristic created dataset to train the model. I'm not sure if this qualifies it as being Zero-shot which requires there are no examples used to train, and inference is run directly. And this wouldn't work for unseen datasets. It requires constructing those templates that seem specific to the datasets used. For e.g. in section 4.2, the template is about airlines, specific to that dataset. How can that be used for a different dataset? If custom hand-crafting of those templates was done for each dataset, then it cannot be used for any unseen dataset - and hence not zero shot.\n* The claim about use of NER techniques is not proven clearly. The abstract states that 2 techniques are used. Appendix A.4 says 3 popular NER techniques are used. But none of their details or citations are mentioned. IIUC a BERT model is fine-tuned to output salient phrases from the utterance, and then selecting the attribute in that phrase. Appendix A.6 again talks about EE and QA tasks, but there are no examples of this in the Appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a technique to do slot-filling for the task of ML problem formulation in the conversational data science domain. Specifically, the paper focuses on what kind of forecasting (i.e. regression but for values outside of the dataset's ranges), the task of identifying the relevant target variables, any filter operations needed on the user-provided data. The paper does this by first synthesizing a dataset of possible forecasting queries that can be made against the user-provided data leveraging it's schema, by using hand-crafted heuristic rules as well as employing a T5-based model. This synthetic data is then used to fine-tune several BERT-based model variants for the task of finding relevant segments within a query that can point to the target variable. The paper evaluates this technique on 3 Kaggle datasets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Intro states “...each user may have unique data sets and datascience needs, it is unrealistic to assume that any training data is available to pre-train these conversational agents, making the supervised learning paradigm impractical.” Are there any stats or references that show whether user data science needs are diverse or not? (In other words, are we sure the most of the user needs are not satisfied by a small subset of techniques?)\n* Because we're talking about forecasting and the mention of time-series forecasting, have you explored other conventional techniques like ARIMA commonly used in time-series forecasting?\n* PeTEL seems like a schema of key-value pairs. Is there a fixed vocabulary or grammar syntax rules for this if calling it a language? (e.g. for POS tagging? What makes it a language vs a schema of key-values?\n* Section 4.2 mentions “Given the lack of a pre-existing training dataset tailored to each unique schema/domain, fine-tuning pre-trained models is unattainable.” But the methodology presented in the paper involves fine-tuning. So isn't this contradictory?\n* Were there any ablation studies performed on the effect of the 3 versions of t5 model data mixtures used? What is the relevance/necessity of adding those 3 mixtures (ie. plain T5, T5+1k examples, T5+10k examples)? And isn't there a bias introduced in the template structure by adding in that mixture for fine-tuning the T5?\n* Algorithm 3 in appendix does not mention how attributes are used to create the utterance in the keyword-to-user utterance task. Is this done for each attribute (i.e. 1 utterance per attribute?) or a subset of attributes (ie. 1 utterance for a randomly selected subset of attributes?). This is a crucial detail that's missing.\n* Algorithm 1 states the sorted list of attributes is shown to the user and then based on if the user agrees or not, proceed. But isn’t the whole point that the user does not know about data science (and so they don’t know about the target attribute)? So why have them evaluate?\n* Section 6.1 is confusing. What does the count have to do here? Each inference is independent of the other, there’s no history being tracked, so what is meant by convergence? Is there some sort of majority voting done from multiple runs to get the final answer? If so, is this doesn’t seem to be explained anywhere.\n* There are too many examples of ChatGPT’s output to qualitatively analyze the performance, but this isn’t of much relevance to the paper. Include more of the examples of the BERT Based models used. What do the generated/synthetic utterances look like? What does the CoNNL-like annotation dataset look like? What do the responses look like?\n* Almost all of A.1 is in Section 4.2. Please dedupe.\n* The main point in the paper is using this approach works for unseen datasets. In that case, shouldn’t the evaluations be done on datasets that are completely unseen during training? I.e. no fine-tuning done on them using custom-crafted heuristic datasets? Maybe use 2 of those datasets in training and validation, but keep one dataset completely untouched and only use it for evaluation.\n* I'm a bit confused: Why to even extract the phrase? If the utterances are being generated based on a template that itself just adds in attributes, why can’t each example in the dataset be annotated with the attributes directly and predict that directly instead of needing the phrase? There aren't any implicit utterances in the dataset anyway.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eeGfVsjixu", "reviewer": "Reviewer_8Xry", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper addresses a significant problem of making time series forecasting accessible to non-experts, which can have a substantial impact on various industries and businesses.\n- The introduction of Forecast Utterance is a promising step towards making conversational data science more intuitive and user-friendly.\n- The unsupervised slot-filling problem formulation and the two zero-shot approaches (EE and QA) are novel and well-suited to the presented problem.\n- The creation of three benchmark datasets demonstrates a commitment to promoting future research in this area. And the case studies provide evidence of the feasibility and effectiveness of the proposed methods on real-world datasets.\n\nWeaknesses:\n- The paper could benefit from a more in-depth comparison of the two proposed zero-shot methods (EE and QA), discussing their relative strengths and weaknesses and the scenarios in which one might be preferred over the other.\n- The unsupervised nature of the proposed methods may lead to potential issues in terms of accuracy and robustness. The paper should discuss possible limitations and provide suggestions for future work to address them.\n- A more comprehensive evaluation, including quantitative metrics, can help assess the performance of the proposed methods and compare them with existing approaches.\n- The paper does not address the potential challenges in handling ambiguous or complex user utterances that may require additional clarification or follow-up questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a novel approach to facilitate time series forecasting for non-experts through natural language conversations with an AI agent. The authors introduce the concept of Forecast Utterance, which captures users' prediction goals in natural language. They frame the task of understanding these utterances as an unsupervised slot-filling problem and propose two zero-shot methods, Entity Extraction (EE) and Question-Answering (QA), for solving it. The paper also presents three benchmark datasets for future research in this area and provides case studies to demonstrate the feasibility and effectiveness of the proposed techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to my concerns above.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wgLpQY7QyN", "reviewer": "Reviewer_7NQw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The process is extensive, providing the full context for the problem, definition, data, evaluation, training data augmentation and a first baseline.\n\n# Weaknesses\nThe intro and motivation are very well, but the actual proposal could be a bit more concrete at the beginning. The first figure doesn't really help, as it stays in the level of arbitrary letters and is not intuitive. It would help readers that skim or do not reach all the way in depth, and even just those that might want easier life while going through the rest of the paper if you provide more intuitive explanation of what you do (with short examples) as opposed to the overall idea of forecasting. Also a scheme of the process (user-> slot-> answer) might help. For example while reading I was waiting for the part where you explain how a language model predicts arbitrary sequences with you method, which is related but not exactly what you do.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present the problem of forecasting by LLMs. They define a forecast by a few slots that one may fill (e.g., target attribute).\nThey provide a method to fill each of the slots given the human's input, those are based on Question answering or entity extraction and on simpler heuristics. The paper also annotates and validates about a thousand examples of sentences and filled slots for evaluation. The paper also creates augmentation data", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# minor\nUnnecessary \", through extensive manual effort\"\nUse citep note \\citet when the names are not part of the sentence. \"the CoNLL-2003 format Tjong Kim Sang & De Meulder (2003) (for EE) and SQuAD format Rajpurkar\net al. (2016) (for QA).\"", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors should make efforts to address the reviewers' comments in the revision. Particularly, Reviewer rYGP pointed out three weaknesses. It is important for the authors to clarify the nature of the techniques. Are they really zero-shot learning? Is hand-crafted dataset needed for each specific domain? If so, how difficult is it to construct them --- presenting them differently can improve clarity of the paper?
Decision: Accept (poster)