TSTR: Target Similarity Tuning Meets the Real World
Authors: Anirudh Khatry
Sumit Gulwani
Priyanshu Gupta
Vu Le
Mukul Singh
Ananya Singha
Gust Verbruggen
Venue: EMNLP 2023 Findings (2023)
Keywords: prompt engineering
code generation
target similarity tuning
example selection
TL;DR: Improving target similarity tuning (TST) as a smart few-shot selection strategy for code generation with LLMs by transforming frozen embeddings from larger models instead of fine-tuning a sentence embedding model.

Abstract: Target similarity tuning (TST) is a method of selecting relevant examples in natural language (NL) to code generation through large language models (LLMs) to improve performance. Its goal is to adapt a sentence embedding model to have the similarity between two NL inputs match the similarity between their associated code outputs. In this paper, we propose different methods to apply and improve TST in the real world. First, we replace the sentence transformer with embeddings from a larger model, which reduces sensitivity to the language distribution and thus provides more flexibility in synthetic generation of examples, and we train a tiny model that transforms these embeddings to a space where embedding similarity matches code similarity, which allows the model to remain a black box and only requires a few matrix multiplications at inference time. Second, we how to efficiently select a smaller number of training examples to train the TST model. Third, we introduce a ranking-based evaluation for TST that does not require end-to-end code generation experiments, which can be expensive to perform.

TSTR: Target Similarity Tuning Meets the Real World
Anirudh Khatry*
Sumit Gulwani
Mukul Singh
Gust Verbruggen Microsoft Keerbergen, Belgium
Priyanshu Gupta
Vu Le
Microsoft Delhi, India
Ananya Singha
Microsoft Redmond, US
Microsoft
Bangalore, India
Abstract
Split the value Column into using delimiier
Target
Table, www.spliicluumc..... Splitter. Tpeertyy w (iiterr(...))
Target similarity tuning (TST) is method of selecting relevant examples in natural language (NL) to code generation through large language models (LLMs) to improve performance. Its goal is to adapt a sentence embedding model have the similarity between
Split the column "output" in Tablei into columns, using and treating as CSV
Table te Lca
sper spl l (nitter...)))
Split
column
two rows.
NI. inputs match the similarity hetween their associated code outputs. In this paper, we propose differ- ent methods apply and improve TST in the real world. First, we replace the sentence trans- former with embeddings from a larger model, which reduces sensitivity the language dis- tribution and thus provides more flexibility in synthetic generation of examples, and we train a tiny model that transforms these embeddings to a space where embedding similarity matches code similarity, which allows the model re- main a black box and only requires a few ma- trix multiplications inference time. Second, we how to efficiently select smaller number of training examples to train the TST model. Third, we introduce a ranking-basse based evaluation for TST that does not require end-to-end code generation experiments, which can be expen- sive to perform.
TabTe.Frooeonn Rows (ss.st.pttst....)
Example of target utterance and associated code, the examples selected by TST and by relying on default embeddings.
Figure I:
TST improves the aligmment between sentence em- beddings of utterances by fine-tuning their cosine similarities to match the similarity of their asso- ciated code snippets. Sentence-BERT (Reimers and Gurevych, 2019) is used as embedding model. Especially for rare programming languages, the model benefits from sceing relevant code (+3% for SQL versus +16% for SMCalFlow using GPT-3).
Example. Figure I shows an example of how TST improves over default embeddings by teaching the model which parts of the utterance are important. The embeddings focus heavily on the "into rows" part of the utterance (where the user might have made a mistake). With TST, we can teach it to focus the "delimiter" part by focusing on the similarity between code snippets.
Introduction
Code generation from natural language utterances is an important and useful ability of large language models (LLMs). Experienced developers can save time, and less experienced users can use natural language to perform data transformation tasks that they would otherwise have carry out manually (Liu et al., 2023). Improving the code generation capabilities of L.LMs a popular research area (Wang et al., 2021).
This paper addresses four limitations of TST when applying it in the real world: (1) sensitivity to language, (2) inference with transformer models, (3) dataset curation and (4) evaluation.
Limitations (1) and (2) are due to the sentence embedding model. Training utterances and real ut- terances often come from diffcrrnnt distributions, for example, from users with different skill levels, and smaller models might be unable capture this vari- ation. When hosted, third-party LLMSs are used for code generation, performing inference with trans- former models might not be possible, making TST
Target similarity tuning (TST) was proposed a method for selecting relevant examples to exploit the in-context learning ability of LLMs and improve performance (Poesia et al., 2022).
First author

---

hard to in production.
CodeBERT (Feng al., 2020) was trained from BERT, CodeT5 (Wang al., 2021) was trained from and Codex (Chen et al., 2021) was trained from GPT-3.
We address both these challenges by replacing the sentence transformer with embeddings from a hosted, third-party model and training fully con- nected neural network (FCNN) to transform the embeddings to capture the code similarity. The large model provides stronger embeddings and the FCNN only requires (few) matrix-vector multi- plication during inference.
A powerful method to generate code from natu- ral language is prompting large language models (Chen al., 2021). As opposed to fine-tuning, this does not require large training datasets and expen- sive compute. Few-shot prompting consistently improves the performance of LLMs across vari- cty of tasks (Brown et al., 2020). Besides helping the model pick the correct programming language (Athiwaratkun al., 2023) the provided few-shots can teach the model about specific functions or parameters and contextualization.
Dataset curation (3) important, as given n (utterance, code) pairs, we can sample nnn - 1)/2 pairs to create training examples for TST, most of consist of irrelevant code pairs. Since we care about distinguishing the best examples, these irrelevant pairs are not desired. We address this challenge by selecting positive and negative examples close the important decision boundary and create smaller dataset that yields better performance and is much faster to train on.
One way of selecting relevant examples is by ing sentence embeddings (Liu et al., 2021). In some cases, however, similar natural language does not correspond to similar code, and vice versa. Syn- chromesh (Poesia et al., 2022) introduced target similarity tuning (TST) to address this challenge and fine-tunes the sentence embedding similarity to match the associated code similarity.
Our method proposed method of training the FCNN on top of frozen embeddings on a relevant set of examples is called TSTR (lastier), where R stands for real world.
Finally, optimizing hyperparameters of the TST model is expensive to evaluate when LLM calls are required. We show that an evaluation based on ranking of examples close to the decision boundary matches the end-to-end performance, providing a cheap way to evaluate TST models. We make the following contributions:
This work builds on the concept of TST and improves on important implementation details for training (selecting examples and allowing synthetic data generation), evaluating (cheap evaluation with proxy metric) and deploying (API call and small FCNN do not have hardware requirements) TST in practice.
(1 + 2) We show that training small model on top of frozen embeddings makes TSTR easier to train and use, and less sensitive to variations in language.
3 TSTR
We briefly recap TST, how that transfers to TST and how examples closer to the relevant decision boundary are selected to improve training
(3) We show that selecting examples close to the important decision boundary allows to train TST model with much fewer examples.
3.1 TST
Given two (utterance, code) pairs (u,, ei) and (u2, 02), vanilla TST fine-tunes semantic textual similarity (STS) model Sm to minimize
(4) We show that ranking (train, test) utter- ance pairs correlates to the performance of end-to-end code generation, providing cheap way evaluate TST models.
||mm (u1, u2) - Se(c1, 02)
(1)
2
Related Work
with S. similarity between code pairs. The STS model is SBERT, which pools BERT tokens and fine-tunes pooled embeddings to capture similarity between pairs of sentences. TST then transforms the embedding to capture properties of utterances that make their associated code similar.
Code generation from natural language a popular area of research (Le al., 2022; Li et al., 2022), Instead of starting from scratch, fine-tuning a pre- trained natural language understanding model to generate code a popular approach, for example,

---

3.2 TST with Embeddings
by experts. The training includes 8000 snip- pets annotated via an LLM (text-davincii-002 We report execution match and sketch match, two standard metrics for code generation (Poe- sia et al., 2022; Singh et al., 2022). Execution match (boolcan) is determined by executing both the ground truth and generated code snippets on table and checking for equality. Sketch match (C [0, 1J) is computed as the normalized (Leven- shtein) edit similarity between the ground truth and generated code snippets after masking con- stants (strings and numbers) and identifiers (col- umn names). We use sketch match as S..
We decouple the embedding from the similarity and compute
(u1, u2) (mm(t(()(()) to (m(())))
(2)
with cos the cosine similarity between vectors, m d-dimensional embedding model (such SBERT) and to IRd RRd trainable transformation, The parameters of frozen. To keep læ simple, we use a fully connected network with tanh activation.
3.3 Training
The training data consists of (un, U2, Se(ci, e2)) triplets. Instead of randomly sampling (u1, 01) and
4.1.2 SMCalFlow
(u2, (2) pairs, we aim to find examples close to the relevant decision boundary. That is,
SMCalFlow is task-oriented dialogue dataset of user-agent conversations, where each query is annotated with program in domain-specific language that facilitates dialogue over dataflow graph (Andreas et al., 2020). In line with previous work (Poesia et al., 2022), we select the first turn from each dialogue to define an NL-to-code task and sample 2000 training examples (out of 40K). The test set consists of 2673 examples. We follow Poesia et al. (2022) and use the normalized edit similarity for both evaluation and S..
about examples for which: (1) Se (2) is high, (2) (c1, (2) is low but cos(m(()), m(u))) is thah are similar the original embedding space but have dissimilar code. In other words, we care about examples with properties that we need to learn and that we need to unlearn. For each pair (M,, G:) we therefore rank all other (Ug:9y) by So(ei, (j) and select the top-Ax best ones. We then skip Ag examples to ensure that code similarity is not too high, rank the remaining examples by Se(m(u:) m(uj)) and again select the top-A, best ones in this new ranking. 0 and A are hyper- parameters.
4.1.3 Bash
The nl2bash dataset consists of bash code snip- pets, each with an expert-curated natural language description (Lin et al., 2018). The train and test sets contain 8090 and 606 examples, respectively. We use the template match metric proposed in the original paper for both evaluation and S..
4 Evaluation Setup
This section deseribes the datasets, metrics, models and hyperparameters used in our experiments.
4.2 Models and Hyperparameters
4.1 Datasets
We use text-embedding--aaa-000 (ada) and text-davinci-003 (GPT-3) (both from OpenAI) as the embedding and code generation models. TSTR, we use two fully connected layers with 512 parameters (see Section 5.2). To prevent overfitting, we apply dropout (0.3) between the embedding and the fully connected layer. A and X are set to 4.
We evaluate TST on three NL-to-code datasets across different low-resource languages. For each language, also report the evaluation metric and code similarity S..
4.1.1 Power Query M
The Power Query M language (or M) is used for transforming data in Power Query. We use the data from Khatry et al. (2023) consisting of code snippets sourced from StackOverflow (test) and the Power Query Community Forum? (train and test).
4.3 Baselines
Across experiments, we use the following baselines for example selections with embeddings. For each baseline, we select eight examples.
The testing set contains 500 snippets annotated
Vanilla ada and SentenceBERT embeddings.
"nttps://ggituub..om/nirrrssssss rress--bbeccaakkkssreee//iiii/rr
TST (Poesia et al., 2022) trained on examples selected according to our selection strategy
fabric nicrosoft.comm com/t5/ bip-eer-----ssiiieee services
Snmer-Query/bddp/aeee powerr---pppppppppp

---

Table I: TST performance across languages on end- to-end code generation task. We find that TST easily overfits on the language distribution of the training data, but TSTR does not.
how the training dataset is created (o). We count proportion of correct pairwise decisions.
We compare the execution match (end-to-end) and pairwise ranking evaluation for different embedding-baad based example selection strategies in Figure 2. Besides baselines, also consider the theoretical maximum for a given similarity using the
Technique 0.20
M
SMCF
Bash
Static
0.43
0.56
BentenceBERT
0.53
0.87
0.65
similarity.
ada
0.54
0.89
0.67
Our ranking evaluation captures the alignment of the TST model with relevant examples, the ones that are the most similar to a target code snippet, ob- served by the distinet relation between the trained TST models (+) and the theoretical maximum, This relation does hold when considering randomly sampled negative examples (+). Embeddings rank poorly for relevant examples, but very good for random examples. These observations highlight the need to select relevant examples for standalone ranking: some nuances of similarities in natural language should be unlearned.
TST
0.52
0.89
0.63
TSTR
0.55
0.90
0.68
(Section 3.3) for one epoch. Using random examples performed worse.
A hybrid approach with frozen SentenceBERT embeddings instead of ada (called TST/),
5 Evaluation
We perform experiments to compare TST against baseline embedding retrieval methods (5.1), we show that ranking relevant examples serves proxy metric to optimize hyperparameters without LLMs (5.2), we evaluate how embeddings from large models (ada) are more robust with respect to variations language (5.3), and we show the effect selecting relevant examples to train TSTR (5.4).
Figure shows end-to-end and ranking results for different configurations of TSTR on the benchmark. A model with too few parameters does not learn enough, and too large models (likely) overfit. More interesting is the relation between ranking performance and execution match, where we can use the former as a proxy determine the number of parameters of TSTR.
5.1 Performance
Table 1 shows results of T and baselines, as well a static prompt with eight randomly se- lected examples. TSTR consistently performs bet- ter (+1%) over vanilla embeddings.
Theoretical Maxima
SBERT
0.9
0.8
ada
TST
Surprisingly, the original TST approach to fine tunc Bent hurts performance on M (- 1%) and Bash (-2%). This may be attributed to overfitting on the language of the training set, and not being able to relate new variations in language to the code similarity (sce Section 5.3).
Ranking
0.7
TST
TSTT
TST
TSTF
0.4
0.3
8 ada SBERT
0.45
0.50
0.55
0.60
5.2 Standalooe Evaluation
Execution Match
We create a pairwise ranking dataset to evaluate TSTR without performing end-to-end code genera- tion and evaluate this approach on M.
Relation between execution match (top-1) and our pairwise ranking evaluation with random (x) and relevant (+) negative examples on the M dataset. Ranking with relevant examples shows a relation with code generation performance.
Figure 2:
Each test point consists of triplet (u,, Upp uh)) where u, is a reference utterance from the testing dataset, up and un are candidate utterances from the training dataset, and Se(Cp, 4) > Se(cn, c.). We consider two ways of sampling Up and u, and thus create two testing datasets: at random (*) or close to the relevant decision boundary, similar to
5.3 Variation Language
We show how TST handles variations in language by creating three different testing datasets, with u,

---

0.680
TST [512,512]
relevant decision boundary, shows the model what to forget and improves training of TST.
0.675
[S112,5125522
Table Influence of sampling up and un for training.
Ranking
11004,1104]
0.670
sampling
M
SMCF
Bash
random
0.46
0.23
0.15
0.665
random
0.49
0.22
0.17
positive only
0.61
0.35
0.20
0.660
[256,256]
TSTR
0.68
0.67
0.42
0.40
0.55
0.60
Execution Match
Figure 3: Relation between execution match (top-1) 1) and our pairwise ranking (+) for different fully connected layer configurations of TSTR on the M dataset. There is a clear relation between both metrics.
6 Conclusion
We introduce TSTR as a practical improvement of TST for selecting relevant examples in code generation from natural language. TSTR replaces fine-tuned SentenceBERT model with small, trainable transformation on top of frozen embed- ding model, and provides strategy for selecting better training examples. Additionally, we show that TST can be evaluated on pairs of utterances from the training set that are ranked with respect to a reference utterance from the testing set, which does not require end-to-end code generation. Our experiments show that TST outperforms classical TST when the language distribution of the example bank does not match that of the tests, that selecting examples closer to the relevant decision boundary improves performance, and that pair- wise ranking evaluation correlates to end-to-end code generation performance.
and (up, un) coming from train-train, test-train and test-test.
Results on are shown in Table 2. We see that TSTR; is significantly less sensitive to language that it has not during training (test- train). TST per- forms best when the samples are from the same dis- tribution as the training corpus (train-train). This shows that the model overfits on the training distri- bution, as performance does not carry over to other language distributions.
Table 2: Evaluating influence of variations in language. TST uses FCL on top of frozen sentence embeddings.
fest-train
train-train
test-test
R 0.68
0.72
0.58
7 Limitations
TST
0.57
0.90
0.57
TST assumes that similar code makes for good examples, and this assumption directly transfers to TSTR, When the code similar overall, but specific details are omitted, this can still result in suboptimal examples.
TST
0.49
0.66
0.51
5.4 Relevant Examples
We show how selecting examples closer to the rele- vant decision boundary improves training of T
An additional call to embedding model endpoint is required to select relevant examples. Whereas embedding calls are generally cheap", the network overhead can cause lower latency than inference with a small transformer.
As baselines, we select (1) random training pairs.
(2) ten times as many random training pairs, and (3) the best positive examples (highest code simi- larity) and negative examples at random.
Table 3 highlights the benefit of selecting the right training set configuration. Even with many more training pairs, random sampling performs poorly. Selecting positive samples based code similarity improves performance- tte system sees more desired examples during training. Selecting relevant negative examples, which are close to the
References
Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Domer, Jason Eisner, Hao
'OpenAl reports price of $0.0001/1K tokens for its most powerful text-ent ambedding-ada-aa model (June 2023).

---

Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho. Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang. Christopher H. Lin, Nlya Lintsbakh, Andy Mc- Govern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy. Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su. Zachary Tellman, Sam Thomson, An- drei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020. Task-oriented dialogue as dataflow synthesis. Trans- actions of the Association for Computational Linguis- tics, 8:556-571.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. 2022. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314-2153
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schritwieser, Rémi Leblond, Tom Becles, James Keeling, Felix Gimeno, Agustin Dal Lago. et al. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092-1199))
Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. N12bash: corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation LREC 2018, Miyazaki (Japan), 7-12 May, 2018.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang. Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shigi Wang. Qing Sun, Mingyue Shang. Su- jan Kumar Gonugondla, Hantian Ding, Varun Ku- mar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ra- manathan, Ramesh Nallapati, Baishakhi Ray, Par- minder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. 2023. Multi-lingual evaluation of code generation models. In International Conference on Learning Representations.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint 2rXiv:210100600
Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn Jack Williams, Neil Toronto, and Andrew D Gordon. 2023. "what it wants me to say": Bridging the abstraction gap between end-user pro- grammers and code- gentraennn large language mod- els. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-31.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse. Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Gabriel Poesia, Oleksandr Polozov, Vu Le, Ashish Ti- wari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code gen- eration from pre-trained language models. arXiv preprimt arXiv:2201.11227.
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert nert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint 1rXiv.2107.03374.
Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Carina Negreanu, Mohammad Raza, and Gust Ver- bruggen. 2022. Cornet: A neurosymbolic approach to learning conditional table formatting rules by ex- ample. arXiv preprint arXiv:2208..m 06032.
Zhangyin Feng. Daya Guo, Duyu Tang, Nan Duan, Xi- aocheng Feng, Ming Gong, Linjun Shou, Bing Qin. Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code- BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536-1547, Online. Association for Computational Linguistics.
Yue Wang. Weishi Wang. Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre- trained encoder-decoder models for code understand- ing and generation. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pages 8696-8708.
Anirudh Khatry, Joyce Cahoon, Jordan Henkel, Shaleen Deep, Venkatesh Emani, Avrilia Floratou, Sumit Gul- wani, Vu Le, Mohammad Raza, Sherry Shi, et al. 2023. From words to code: Hamesssing data for pro- gram synthesis from natural language. arXiv preprint arXiv:2305.01596


Reviews:
[{"review_id": "717u8PurIG", "reviewer": "Reviewer_cbui", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the reasons for rejection and questions by the reviewer below:\n\n### Question-1: Lack of comparison with other similar methods.\n\nThis work focuses specifically on the in-context (few-shot) learning problem. In this setting, common methods in practice are to select random examples [1], use embeddings directly [4] or fine-tune a separate transformer model [2,3] to select the few-shot examples. We compare to these approaches. Recent work rewrites the input to the embedding model to obtain embeddings that more closely align to a given task [5]. We will compare to this method as well.\nFurthermore, we will improve our evaluation by adding results on public NL to Code datasets: [nl2bash](https://github.com/TellinaTool/nl2bash/tree/master/data) and [SMCalFlow](https://microsoft.github.io/task_oriented_dialogue_as_dataflow_synthesis). SMCalFlow was used to evaluate TST [3].\n\n### Question-2: The metric of evaluation needs a clearer explanation for realistic meaning.\nWe use the execution match metric to report results for the most important question: end-to-end code-generation accuracy (Figure 2). This metric is popular in text-to-code settings [1,2,3,6,7] as a problem can be solved by multiple programs.  Because end-to-end testing with few-shot prompts is expensive, we use our proposed pairwise  ranking as a cheaper proxy metric for other questions (language variation and boundary example selection). Our experiments show that this proxy metric is correlated well with end-to-end results.  We can report end-to-end results for these questions on all datasets (see Question-1) in our revision.  \n\n### Question-3: Lack of description for the training set, which is one of the issues to be solved in this paper.\n\nThanks for your feedback. We collected the training set by scraping M code from the community channel set up by Power Query [8]. The natural language utterances for the M code were generated by using an LLM. This process used a few-shot prompt using the power query documentation as reference [9]. We will extend our discussion in Section 4 (under Data and Model) to include more details.\n\n### Data and Code\n\nFor review purposes, we privately share data and code [here](https://figshare.com/s/a8c3dc246a3d7d95a9e5). It is not meant to be shared publicly. All contained resources are proprietary.\n\n### References\n\n[1] Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I. Wang, Xi Victoria Lin. “LEVER: Learning to Verify Language-to-Code Generation with Execution.” ICML 2023.\n\n[2] Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. “Jigsaw: Large language models meet program synthesis.” ICSE 2022.\n\n[3] Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. “Synchromesh: Reliable Code Generation from Pre-trained Language Models.” ICLR 2021.\n\n[4] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. “What Makes Good In-Context Examples for GPT-3?” DeeLIO 2022 (ACL Workshop).\n\n[5] Shengnan An, Bo Zhou, Zeqi Lin, Qiang Fu, Bei Chen, Nanning Zheng, Weizhu Chen, and Jian-Guang Lou. “Skill-Based Few-Shot Selection for In-Context Learning.” arXiv preprint arXiv:2305.14210 (2023).\n\n[6] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen et al. “CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X.” KDD 2023.\n\n[7] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles et al. “Competition-level code generation with alphacode.” Science 2022.\n\n[8] Power Query Community Forum. https://community.fabric.microsoft.com/t5/Power-Query/bd-p/power-bi-services\n\n[9] Power Query Documentation. https://learn.microsoft.com/en-us/powerquery-m/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This proposed method could enhance the application of TST in the real world, by reducing the sensitivity of language and enabling the inference with transformer models.\nBesides, the proposed method could reduce the requirement for the scale of the training set.\nFurther, the authors also propose an evaluation method for the TST task.", "reasons_to_reject": "1. Lack of comparison with other similar methods.\n2. The metric of evaluation needs a clearer explanation for realistic meaning.\n3.  Lack of description for the training set, which is one of the issues to be solved in this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors try to improve the Target Similarity Tuning (TST) application on the Code Generation task.\nThey improve the similarity of the sentence embedding to match the code similarity and train the TST model with a small train set.\nThey also introduce a ranking-based evaluation for TST, without the requirement of expensive end-to-end code generation experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5LraCRfKD3", "reviewer": "Reviewer_TieY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the reasons for rejection and questions by the reviewer below:\n\n### Question-1: This work is very incremental\n\nThis work builds on the concept of TST and **improves** on important implementational details for **training** (selecting examples and allowing synthetic data generation), **evaluating** (cheap evaluation with a proxy metric) and **deploying** (API call and small FCNN do not have hardware requirements) TST in practice.\n\nWe will report results on two additional (public) datasets to further demonstrate these contributions ([nl2bash](https://github.com/TellinaTool/nl2bash/tree/master/data) and [SMCalFlow](https://microsoft.github.io/task_oriented_dialogue_as_dataflow_synthesis)).\n\n### Data and Code\n\nFor review purposes, we privately share data and code [here](https://figshare.com/s/a8c3dc246a3d7d95a9e5). It is not meant to be shared publicly. All contained resources are proprietary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed modifications lead to gains, especially when the language distribution of the examples is different than the test set one.\n- The dataset filtering improves results while making the train faster.\n- The proposed evaluation method is simpler and correlates well with the end to end evaluation.\n", "reasons_to_reject": "- This work is very incremental.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper the authors propose several modifications to Target Similarity Tuning (TST). \nTST consists of fine-tuning a sentence embedding model (SBert) such that the similarity between utterance (commands) embeddings are close to the similarity between the associated code examples.\n\nThe authors propose fine-tuning only a fully-connected network that receives as input the sentence embeddings instead of fine-tuning the embedding model. This allows the use of larger and third-party LLMs. \n\nThe authors also propose to filter the dataset such that it contains more relevant examples (examples for which the utterance similarity is high).\n\nLastly, the authors proposed a simpler metric that does not require the code snippets to be run. Instead, they construct a dataset using a utterance from the test set and two target utterances sampled from the training set. And see the proportion of correct decisions.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iFIOqeUcIq", "reviewer": "Reviewer_pWu8", "summary": "", "questions": "What is the R stands for in TSTR?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the reasons for rejection and questions by the reviewer below:\n\n### Question-1: Training and Evaluation data is not public\n\nOur work uses the data from a prior work [1]. We hope that the authors will release their data in the future.\nTo further address this issue, we will add the results on two public datasets: [nl2bash](https://github.com/TellinaTool/nl2bash/tree/master/data) and [SMCalFlow](https://microsoft.github.io/task_oriented_dialogue_as_dataflow_synthesis). Preliminary results on nl2bash show that their command match score improves by 50% between random few-shot and theoretical TST maximum. Additionally, see below for a private release of the data and code.\n\n###  Question-2: The presentation is not clear and really needs a lot of work\n\nThank you for your feedback. We will (1) clarify details on training set collection, (2)      clarify the contributions, (3) improve justification of the proxy metric, (4) improve the flow of the technical section, and (5) add results from two new domains.\n\n###  Question-3: What is the R stands for in TSTR?\n\nThe R stands for **r**eal-world, relating to the practical improvements we make to train, evaluate and deploy TST in real scenarios.\n\n### Data and Code\n\nFor review purposes, we privately share data and code [here](https://figshare.com/s/a8c3dc246a3d7d95a9e5). It is not meant to be shared publicly. All contained resources are proprietary.\n\n\n### References\n\n[1] Anirudh Khatry, Joyce Cahoon, Jordan Henkel, Shaleen Deep, Venkatesh Emani, Avrilia Floratou, Sumit Gulwani et al. “From Words to Code: Harnessing Data for Program Synthesis from Natural Language.” arXiv preprint arXiv:2305.01598 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) practical real-world problem with simple techniques to improve performance and present an indirect evaluation protocol for fast development", "reasons_to_reject": "(1) Training and Evaluation data is not public but I think the main idea is clear and can be extended to other public datasets. Also, the dataset is referenced in another paper so its not this paper's responsibility to release the data.\n(2) The presentation is not clear and really needs a lot of work.", "questions_to_authors": "What is the R stands for in TSTR?", "paper_topic_and_main_contributions": "This paper studies a practical real-world problem of deploying TST model in production.\nIt aims to solve 4 practical problems sensitivity and inference cost of transformer models, data set curation for training and final evaluation without code generation.\nThe paper proposes to train a neural network on top of embeddings to capture code similarity with the TST objective. It then presents a way to select pos/neg examples to yield better results and smaller data sizes for faster training. Finally, they claim the evaluation based on the ranking of examples matches the end-to-end code generation performance.\nOverall, I think the contribution is useful but the presentation of this paper needs a lot of work..", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L020: we how to \nL87: our method proposed method\nL137: on \"Synchromesh\" to further\nI think the writing needs to be improved."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main concern raised by the reviewer are that the code and data are not publicly available. The authors did make the code and data privately available for review but the reviewers felt that is not enough (and I agree). Apart from this the paper needs a few changes:

1. Need to report results on public datasets [the authors do promise that they will report results on 2 public datasets but the results on only 1 were provided during the response period)
2. More details on the manner in which the training dataset was curated.
3. Comparison with more recent related work (such as [5] as suggested by the authors)
4. The presentation could be better.

I request the authors to address the above concerns in a subsequent revision.
Decision: Accept-Findings