Can LLMs Implicitly Learn Numeric Parameter Constraints in Data Science APIs?
Authors: Yinlin Deng
Chunqiu Steven Xia
Zhezhen Cao
Meiziniu Li
LINGMING ZHANG
Venue: NeurIPS 2024 poster (2024)
Keywords: Program Synthesis
Evaluation
Dataset
Data Science
LLM4Code
Primary area: evaluation
TL;DR: A comprehensive evaluation on the proficiency of LLMs to handle numerical constraints in data science libraries.

Abstract: Data science (DS) programs, typically built on popular DS libraries (such as PyTorch and NumPy) with thousands of APIs, serve as the cornerstone for various mission-critical domains such as financial systems, autonomous driving software, and coding assistants. Recently, large language models (LLMs) have been widely applied to generate DS programs across diverse scenarios, such as assisting users for DS programming or detecting critical vulnerabilities in DS frameworks. Such applications have all operated under the assumption, that LLMs can implicitly model the numerical parameter constraints in DS library APIs and produce valid code. However, this assumption has not been rigorously studied in the literature. In this paper, we empirically investigate the proficiency of LLMs to handle these implicit numerical constraints when generating DS programs. We studied 28 widely used APIs from PyTorch and NumPy, and scrutinized the LLMs’ generation performance in different levels of granularity: full programs, all parameters, and individual parameters of a single API. We evaluated both state-of-the-art open-source and closed-source models. The results show that LLMs are great at generating simple DS programs, particularly those that follow common patterns seen in training data. However, as we increase the difficulty by providing more complex/unusual inputs, the performance of LLMs drops significantly. We also observe that GPT-4-Turbo can sustain much higher performance overall, but still cannot handle arithmetic API constraints well. In summary, while LLMs exhibit the ability to memorize common patterns of popular DS API usage through massive training, they overall lack genuine comprehension of the underlying numerical constraints.

Can LLMs Implicitly Learn Numeric Parameter Constraints in Data Science APIs?
Yinlin Deng Chunqiu Steven Xia Zhezhen Cao
Meiziniu Li
Lingming Zhang 1
I University of Illinois Urbana- Chhama-Chammpain Southern University of Science and Technology 6 The Hong Kong University of Science and Technology
(JnliioZ,chunqiud..iinnnng) @illinois.edu, 121 12110529 @
mlick @cse.ust.hk
Abstract
Data science (DS) programs, typically built popular DS libraries (such Py- Torch and NumPy) with thousands of APIs, serve as the cornerstone for various mission-critical domains such financial systems, autonomous driving software, and coding assistants. Recently, large language models (LLMs) have been widely applied to generate DS programs across diverse scenarios, such assisting users for DS programming or detecting critical vulnerabilities in DS frameworks. Such appli- cations have all operated under the assumption, that LLMs implicitly model the numerical parameter constraints in DS library APIs and produce valid code. How- ever, this assumption has not been rigorously studied in the literature. In this paper, we empirically investigate the proficiency of LLMs to handle these implicit numer- ical constraints when generating DS programs. We studied 28 widely used APIs from Py PyTorch and NumPy, and scrutinized the LLMs' generation performance in different levels of granularity: full programs, all parameters, and individual parameters of single API. We evaluated both state-of- state-of-the-art open-source and closed-source models. The results show that LLMs are great at generating simple DS programs, particularly those that follow common patterns seen in training data. However, as we increase the difficulty by providing more complex/unusual inputs, the performance of LLMs drops significantly. We also observe that GPT-4-Turbo can sustain much higher performance overall, but still cannot handle arithmetic API constraints well. In summary, while LLMs exhibit the ability to memorize common patterns of popular DS API usage through massive training, they overall lack genuine comprehension of the underlying numerical constraints.
Introduction
Data science (DS) is an emerging and important area that combines classic fields like statistics, databases, data mining, and machine learning (ML) to gain insights via complex operations on the abundance of available data [49]. DS libraries (such as PyTorch [41] and NumPy [38]) contain thousands of APIs used by developers and data scientists to process/analyse data. These DS APIs serve as the fundamental building blocks for almost important ML/DS pipelines, and have penetrated into almost every corner of modern society, including financial systems [18, 4], autonomous driving software 19, 27, 46), coding assistants [45, 37], etc. Due to their high importance and wide usage, automatically synthesizing valid DS programs has been critical research area [29, 21, 47].
One key challenge of DS code generation is to satisfy the complex constraints within each DS library API. DS library APIs perform transformations (e.g., matrix multiplication) on inputs (i.c., arrays or aike objects) with numeric constraints on API parameters and inputs. Figure I shows example
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

of a typical DS program where the DS library (i.e., PyTorch) is first imported, followed by creating some input_data, and then performing the data manipulation operation on the input_data using a groups) must satisfy the We refer to
DS API (torch. nn. Conv2d). The parameters of the API (e.g., kernel_siaee corresponding constraints between API parameters and the properties of the input, API constraints as the set of relationships between properties of input_date and API parameters that, if and only if when satisfied, leads to a valid DS API invocation. As seen in Figure I, not only are there constraints between the properties of the input_ _att and API parameters (e.g,, hernel_size I H 2+padding), but there are also constraints within API parameters (e.g, out_ channel % groups 0). These constraints are defined by developers according to the functionality of each DS API, and are usually specified in natural language within the API documentation. Such complex constraints are critical for DS applications, and DS users even DS experts may unintentionally violate such constraints (29, 26].
Large language models (LLMS) have achieved tremendous success in processing code [10, 2]. Due to their powerful code understanding and
shuchenn groupa
DS Library: PyTarch
wr.chamees groups
generation ability. LLMs have been applied to various coding tasks [34], such as code comple- tion [20, 6]. program repair [19, 54], and test generation [16, 17, 48]. For DS libraries, LLMs have been applied to solve practical user queries on StackOverflow [29] and even generate test programs to detect bugs in modern MI. frame-
katret_ g 2rpanding
Import tarch
torch. candn 16, 1000)
torch.m. am. (1o, ML, simne, pateinge2, groupse1) V m(x)
Figure I: Example DS program with constraints
works [16]. Prior work assumes LLMs, through massive training, can already implicitly model constraints in DS APIs by learning from numerous correct DS API uses [47, 21, 16). However, this assumption has not been systematically verified. Furthermore, popular DS-specific benchmarks like DS-1000 [29] do not specially test the LLM's ability to satisfy implicit constraints and instead focus on how to apply DS APls to solve data analysis tasks. These gaps in prior research raise a critical question: Can LLMs implicitly learn the numeric constraints in data science APIs?
Our work. To answer the question, we conduct a rigorous study on the performance of LLMs in generating valid DS programs satisfying diverse numerical API constraints. We collected a set of 28 representative DS library APIs across two widely-used Python DS libraries (PyTorcl and NumPy), each with their unique constraints/setup. Additionally, we categorize each API's constraints into different categories (e.g.. equality and arithmetic) and perform in-depth experiments on each constraint type. To support our analysis, we systematically created generation settings: full program, all parameters, and individual parameters, designed to test the LLMs under different evaluation scenarios. Additionally, we vary the difficulty level by adjusting the inputs to explore L.M behaviours when asked to solve more complex API constraints given more unnatural inputs.
Interestingly, contrary to the popular assumption in prior work, while LLMs can easily satisfy constraints when the inputs are simple, we observe that the performance drops drastically as we increase the difficulty or provide more unusual inputs. We found that LLMs tend to generate simple and common inputs seen during training, highlighting that LLMs are often memorizing patterns instead of truly understanding the actual DS API constraints. For example, for the widely used Conv2d API shown in Figure 1, when max siin_chnnels,,..eeeeaeaaae_seeee outtotttuaaaaa cutt_channels) is set to [128, 256), even GPr-4-Turbo [1] can only predict the correct value of groups ~44 of the time, while the other models are below 14%. Furthermore, based on our experimental findings, we constructed DSEVAL, the first benchmark for systematically evaluating LLMs' capabilities in understanding the important numerical API constraints for popular DS libraries. DSEVAL contains 19,600 different problems across 12 representative APIS to extensively compare and contrast the performance of different LLMs. DSEVAL supports lightweight and fast evaluation by extracting LLM generated parameters and quickly verifying the correctness using statc-of-the art SMT solvers (such as Z3 [13]) to avoid time-consuming execution-based evaluations. Our evaluation eight of-te-ofthheaard open-soure and closed-source models shows that while all studied models struggle with more difficult problems, GPT-4-Turbo consistently achieves the highest accuracy across all difficulty levels. For example, GPT-4-Turro achieves an average accuracy of 57.5% for hard constraints of PyTorch APIs, while the best open-source model can only achieve 39.2%, demonstrating the huge gap between large proprictary models and other open-source LLMs. Our design of DSEVAL is general and can be easily extended to additional libraries and APIs for the DS domain and even beyond.

---

Table I: Categorization of constraint types with exemplar API names, description, and examples.
Category
API names
Description
Example
CBatchNorn2d, CLinear
Copying specilic dimension
nfeat input__saaee [pappt_shapes[]] [1]
Equality
@aquaaze, it
Indexing the correct dimension
Inoquality
G8oftXax, Cnean Oeun, Uuxx
Single value related to rank
-rank < dim rank
Multiple values related to rank
rank S dim rank for dim in dine
MMaxPoo1dd, OdvgPoo12d OConv2d, (Convid
Multiplies constani number
kernel _ize S H padding
Arithmetic
Divides parameter
in ch.nnenllls X groupa
Oreahape, @reshape
Product of parameters
[npput_shappes shapes [[target_aaag
(Fold, OConvid
Complex arithmetic
1-11 0. (x)ppaddd (11 (iil 1:1+1]
maa Cluu
Uniqueness
Kdins)) Idimsl
Set-related
Stranspose
Completeness
shauttsshapes> [axes]
2
Study Approach
2.1 Scope of study
Instead of considering all possible DS programs and APIs, we focus simple DS programs with only a single API call. This allows us to isolate the evaluation to individual APIs or even individual API parameters, facilitating tine-griine analysis and a detailed examination of the LLMs' limitations with respect to various types of numerical constraint.
We specifically target the core APIs commonly used by users that perform operations on the input_ Additionally, we also only consider numeric API constraints: constraints with only numeric parameters such as integers. We ignore any other types of parameters (e.g., string) since they do not affect the validity of numeric constraints. As such, any non-numeric parameters produced by the model will be discarded during constraint validation.
Table I shows the types of constraints we considered in the study with the corresponding categories. We group the constraints into i) Equality: constraints where the values have to match exactly. We see that equality constraints are related to selecting or generating the right shape in the input_data. ti) Inequality: constraints where values have to be greater or less than. Inequality constraints include mainly rank related operations to stay within the valid rank range. iii Arithmetic: constraints involving arithmetic operations such as division, modulus products. There are also more complex API constraints that includes combination of many arithmetic operations. iv) Set-related: constraints where the satisfaction criteria depend on different set-based properties. For example, there are constraints that require parameters to unique or complete with respect to input_ shappt.
program Irput
paramatera rost
Stop
Inport torch
Japort tarch terch, Fundi 19, Pesl SeteFiti
Stop
Senerate tensor with torch.randn sutose the
LLM
saport torch
kernel sizee-3, stride-(3,
Stop Generate ose inpet tersor with torch.randn
tttct...rraa randa(3, 4, 5)
sxport nappy as (and(1),
Step cutout tsrch..snls, sunis, dtvv))
torch.ssn
24 csccaaing Cs, G, SPELL (13)
oalpu
5, 38, 4, 22
Figure 2: Example problem input and LLM output for each evaluation setting
2.2
Evaluation settings
Next, we describe our settings to evaluate the performance of LLM on handling the numeric con- straints. In total, we have settings: i) full program, ii) all parameters, and iii individual parameter.
Full program. For the full program setting, we want the LLM to synthesize a complete DS program using a specific API from scratch. To do this, we provide a 3-step instruction and the basic starting code of importing the DS library. Figure 2a shows an example of the full program input for the API torch. sum well example LLM output. We note that in this setting, the LLM has full
3

---

freedom to generate any type or size for the saput_data As such, the LLM may choose very simple input_data and API parameter values that can easily satisfy the constraint.
All parameters. In the all parameters setting, we directly provide the input_ for the API. Figure 2b also shows an example of the input for the API torch. mn. MaxPool2d where the LLM just needs to output the API parameters. This setting evaluates if/how LLMs can accurately solve the constraints as we vary the input_ data with more difficult uncommon cases. Still the LLM has full freedom to piek the full combination of parameters to satisfy the required constraint.
Individual parameter (main setting). To perform a finer-grained evaluation, we introduce the individual parameter setting where we ask the LLM to generate single parameter of the API. Figure 2c additionally demonstrates an example for np. reshese where we only allow the LLM to fill in a single parameter value of newshape. Furthermore, we can also add an additional constraint by directly providing the first value of newshape (2 in the example). This makes the problem even more challenging where instead of being able to simply copy the input_shappl to reason with the partial shape given and compute the final correct shape to satisfy the constraint. Compared to the prior two settings, the choices here are much limited. This makes the task harder to fully evaluate how LLMs solve complex API constraints, and serves as our main setting.
the LLM now has
2.3 Input creation and output validation
Creation. To produce the inputs for each of the 3 settings, we use fixed set of templates for each API. For the full program setting, we produce one input per API, changing only the API name in the input instruction. For the all parameters setting, we vary the input_data given to the API. In particular, we focus on two properties of the input_ 1) rank of the input_ data and 2) each dimension value. We create randomized inputs and increase the difficulty by either increasing the rank or the dimension values to measure the LLM performance. Note that input rank or dimensionality can affect different APIs depending API like torch. nn. MMfx that has a constraint of -rank S dim < rank will have its difficulty influenced by the actual rank of the input tensor. On the other hand, API like torch. Conv2d has a constraint of in_ channels % groups = o. which depends on the actual dimension value of the input (i.e., in_ channels). As the dimension value of in channels increases. it will be more difficult to select the groups parameter that can divide it evenly. Therefore, we increase the difficulty of different APIs based whether the constraint depends on the rank, dimension, or both. Similarly, for the individual parameter setting, we also randomize the input_ __ttttaaaaaaaa based on the previous two properties. Additionally, we pick the parameters with interesting constraints for the L.MM predict in order to be representative and cover the major constraint types. Furthermore, since we only ask the LLM to produce a single parameter value, we also vary the other parameter values in the API add additional constraints (details discussed in Section 4.3).
the specific numeric constraints (Table 1). For example, an
To ensure the input is valid, we leverage satisfi- ability modulo theory (SMT) solvers as shown in Figure 3a. SMT solvers, such Z3 [13], are
266
tools which can be used to solve an SMT prob- lem of determining whether mathematical first-order logic formula is satisfiable [5]. We first
DUT Cha So1 sioe
out charneta grouge 0 AUTio-
Satisfiall?
encode the API constraints into an SMT formula. We then randomly generate concrete values for the input_ shautsssspsssss and leave the other parameters that we want the LLM to generate as symbolic variables. Next, we use an SMT solver to check if the constraints are satisfiable (i.e.. there exists set of values for each symbolic variable that can satisfy the constraint). If it is satisfiable, the input we provide to the LLM is valid, otherwise we restart the process by randomly selecting the concrete values. In our study, we reuse the encoded API constraints provided by NNSmith [32] (a popular tool for testing ML. libraries via formal constraint solving) and add additional ones when needed.
chamat groupa
253
Concrnaa Crocttirrgetiii
Valld?
chasneia grosps kermit It seeetin
wi pmriprrc audiitepe"
SNT fomm
OO
S
Figure 3: Example usage of constraint solvers to generate inputs and validate outputs.
Evaluation. To evaluate the validity of the DS programs generated by the LLMs, we first parse the output to extract the input_ data and API parameters. We then check if the LLM predicted

---

values are valid. This is also done via SMT solving as demonstrated in Figure 3b where we use the SMT formulas and, this time, check all the concrete values generated are valid according to the constraints. Note that such light-wert weight constraint solving can support much faster validation than actually executing the generated DS programs, while still providing the same guarantee.
3 Experimental Setup
3.1 Subjects
We construct a dataset with 28 representative APIs in total from two popular DS libraries: PyTorch (18) and NumPy (10). For our API selection process, we begin by referencing prior work NNSmith [32] and examined all 73 core operators it supports. From these, we select 22 core APIs that have numeric parameter constraints and add additional 6 APIs to obtain the 28 APIs used in our study for both the full program prediction setting (Section 4.1) and the full API parameter prediction setting (Section 4.2). For a more detailed analysis, we select 12 APIS to cover the representative types of numeric constraint for examination in the single API parameter prediction setting (Section 4.3) and in our DSEVAL benchmark (Section 4.4). We use "representative" to mean representative with respect to the numeric parameter constraints in DS library APIs. Table shows the cafegorization of the different types of numeric constraints that exist in DS libraries. Our selection criteria aim to select list of APIs that have interesting numeric parameter constraints that can cover all the major constraint categories. A complete list of the 12 APIs and their corresponding constraints is provided in Table 3 in the Appendix.
We focus on the 3 settings described previously to analyse the performance of LLMs. For the full generate single input prompt per each studied API and ask the LLMs to
program setting, synthesize the complete DS program by varying the sampling temperature. For the all parameters setting, we have 14 difficulty settings, each with 200 different inputs per API, and use greedy decoding to obtain the LLM solutions. The difficulty setting is controlled by increasing the rank of input_ _data (from 2 to 8 in intervals of 1) with default dimension value as 16], and increasing the dimension value (i.c., (1,4), [4, ..... [128, 256)) with default rank as 3, separately. Finally, in the single parameter setting, we select one parameter for each API for the LLM to generate. For any parameters irrelevant to the constraint, we use the default value if it is an optional parameter, and randomly choose from a reasonable value range if a required parameter (Appendix C). We adopt the same difficulty setup and greedy decoding strategy as the all parameter setting.
3.2 Metrics
Validity. To measure validity, we directly extract the L.LM output predictions and evaluate according to the process described in Section 2.3. We define accuracy as the percentage of valid programs produced by the L.LMs in each difficulty setting.
Diversity. To measure diversity, we compute the unique valid rute: the percentage of unique valid programs generated via sampling. Note that we deduplicate by extracting the input shapes and numeric parameters, ignoring the irrelevant parameters and irrelevant code suffix.
3.3 Studied models.
We evaluate popular state-of-the-art LLMs, including both closed-soures and open-source models (detailed list shown in Table 2). For both the full program and all parameter settings, we only present the results for DeepSeek Coder-33b [22], state-of-the-art open-source model, due the space limit (other models follow similar trends). For the individual parameter setting (the main setting). we focus on the DeepSeek Coder family models (33b, 6.7b, and 1.3b) as well GPT-4-Turbo (2024-04-09 covering both state-of -ote----rttt-tttte open- source and close-source models, as well as models with different sizes. Apart from the full program setting, where the LLM generates a complete program, we perform infilling using the studied L.MSs' model-specific infilling format. To perform infilling using GPT-4- Turbo, we design a specialized prompt (see Appendix H). Unless otherwise stated, we use greedy decoding (i.e., temperature 0) and temperature of I when sampling for diversity evaluation.
5

---

Evaluation
4.1 Full program prediction
1.0
0.6
0.8
0.5
9.6
0.44
tomperatune
tomperature
Figure 4: Full program prediction result all 28 APIs (O PyTorch and @ NumPy).
To start with, we ask the LLM (DeepSeek Coder-33b) to predict the entire DS program from scratch given just simple instructions. Figure 4a shows the overall accuracy of the 18 APIs in PyTorch and 10 APIs in NumPy. We see that with low temperature the model has near perfect accuracy on almost all the APIS and temperature slowly increases, the accuracy tends drop (ending with around 0.5~008 with temperature=1), Surprisingly, we found that for torch. un Fold, which contains the most complex constraint, the LLM failed produce any valid DS programs. This demonstrates that LLMs may still struggle with satisfying the extremely difficult constraints even when given the full freedom of generating any input values. Furthermore, in Figure 4b, plot the proportion of unique valid programs generated by the model as we vary temperature. Of course when sampling at low temperatures, many of the inputs will be repeated, leading to low number of unique programs in general. In particular, the input shapes are often from widely-used computer vision datasets like 3*224*224 from ImageNet [15]. This indicates the LLMs tend to memorize some common patterns from either documentation user programs. However, we see that even though the unique valid rate increases with high temperatures to give more diverse and creative outputs, the percentage of unique valid programs can still be mostly below 50%. This demonstrates that while models are successful in generating high percentage of valid programs, lot of generated programs are repeated.
4.2
Full API parameter prediction
0.8
accuraty
0.4
Nachol
argenclapl
reshape(pt)
reshape(op)
fotcherradd (pt)
split(ep)
reshspe(ep)
seseece (op)
splitinp)
argnin(spl
rots(pt)
8.8
ethern(27)
5
11,4)
[4,8)
(6,16) (16,32) (32,64) (64,128)(228255) difficulty (irpet dix)
difficuley trank)
Figure 5: Full API parameter prediction result on all 28 APIS (O PyTorch and @ NumPy). The LLM has near 100% accuracy on some APIs, which are collectively referred others (x), where is the number of grouped APIs.
Figure shows setting where we randomly provide an input_ data and ask DeepSeek Coder-33b to complete the valid parameters of the API. We vary the difficulty by changing either the rank or the dimension value ranges of the input_ data to produce more complex and unnatural inputs. We use greedy decoding (temperature 0) to generate solution per problem, and compute the

---

average valid rate across the randomly created problems to compute accuracy for each difficulty level. Compared to Section 4.1 where LLMs achieve near pear-perfect accuracy for almost all APIs with low temperature like 0.2, we observe that the accuracy quickly drops when simply randomizing the input shape, especially for APIs with more complex constraints. This indicates that the learned patterns cannot easily generalize to less common input shapes. We further performed an interesting case study on the PyTorch API Linear, and found this phenomenon holds true across different models (Appendix D). However, we see that the majority of APIs maintain high accuracy even difficulty increases (others (x) in Figure 5). This because these APIs have relatively easy constraints. For example, APIs like to operate on, and the LLMs learn to predict dim=1 or just rely on the default parameter values the API which are always valid.
argmax only require predicting single integer representing the dimension
4.3 Single API parameter prediction
We now focus on the main finer- finer-grained evaluation setting where we ask LLMs to predict a single parameter value and discuss the input setup, results, and findings for each API separately. Here, we only discuss representative API constraints from each category and full results are in Appendix F.
1.0
3.000
85-4.30
6PT4
8.575
06-33
51
SETOL rara))
0.00
6.500
36- (onaot
6.40
6.475
6.000
6.30
11.41
(4.3)
14:4 18530 13:50
(mgos)
6.30
(a) O Py Torch catoh22dd
(b) PyTorch max w/ temp=(0,1 (0,1)
(c) O PyTorch max diversity
Ko
65-4-1
1.0
Crt6-Tarto
0.44
9.7
0.2
9.0
8.0
9.0
10.10
(32,4) J8N 10.
13.40
(4.51
18.30 DR s0 13BEEE
(d) O PyTorch reshape
Oy)o Conv2d
(l) O PyTorh Fold
gita Tume
1.2
1.4
9.15
1.4
1.2
3.1
[C41
10.01
18,00
DECRRY TREMY 10
(g) NumPy transpose
(h) O Conv2d on models
(i) Ó PyTorch Fold on IT models
Figure 6: Single API parameter result. Solid lines (except Fig. 6c) show the accuracy of using greedy decoding (temp=0). In Fig. 6b, dashed lines show the pass@1 accuracy in sampling experiments with temp-1. In Fig. 6d, dotted lines show the accuracy after excluding trivial solutions. In Fig. 6h and 6i, we use *-Inst. to distinguish between the generation settings: infilling (GPT4-Turby and free-form generation GGPT4-Turbo-Inss More details are provided in Appendix H and I.
Equality. BatchNorm2d in PyTorch applies batch normalization [25] on a 4D input tensor, with the second dimension as the number of features. We select the parameter num features for the models to predict, with the equality constraint that num features = imput_shapedt []. Figure shows the results increase the difficulty by changing the maximum possible value for each input
7

---

dimension. We observe that the DeepSeek Coder models drop from around 0.7~~088 to less than 0.5, while GPT-4's performance stays around 0.9 throughout different difficulty levels.
Finding: Overall, we found that smaller L.LSs even struggles with even the simple constraint of copying an existing value, while large state-of-the-at LLMs can maintain its high performance.
Inequality, max in PyTorch computes the maximum value along a dimension. The parameter we target is dim with the valid range being [-rank, rank). In Figure 6b, when using greedy decoding, all 4 I.LMs achieve close to perfect accuracy. Therefore, we also conduct sampling experiments and present the pass @ I accuracy and diversity in Figure 6b and 6c. For max we compute the diversity differently from Section 3.2 (sce Appendix G), since the number of possible unique valid outputs is very small. Interestingly, the smaller DeepSeek Coder-1.3b model achieves highest sampling accuracy for rank=8, but has the lowest diversity. This is because the smaller model often predicts common values like 1, whereas the larger model (33b) can explore various correct answers like -1,2.
Findings:
We found that larger models are indeed better at capturing the simple inequality constraints and modeling the true probability of various possible values, while smaller models tend to memorize common patterns, leading to less diverse predictions.
Arithmetic. reshape in both PyTorrch and NumPy attempts to rearrange the dimensions in the
_aput_data,, with the constraint being II, ixput_shapeee [i) == II new_shape[j]- Since we found that it is common for the LLMs to simply predict the same shape or a permutation of the original, we add an additional constraint: we specify the first dimension of the new_ sa__hape to be different from any dimensions input_shapes Figure 6d shows the results as we vary the ranks of the input_ for Py Torch (similar trend in NumPy). We observe that most LLMs in the beginning perform well; however, as the difficulty increases, their performance drastically lowers. Meanwhile, GPT-4-Turbo performance does not drop even with more difficult inputs. We found the reason is that GPT-4-Turro tends to always predict the special -1 value for reshape where the new _aaape will be automatically inferred by the library. Figure 6d showcases this exact phenomenon in PyTorch (similar trend as NumPy) where dotted lines present the accuracy of any outputs without -1. We see that now even GPT-4- struggles in generating valid parameters without using the - crutch for the constraint.
Conv2d in PyTorch applies a 2D convolution over a 4D input tensor. The LLMs are asked to predict the parameter groups, where they have to divide both in_ channelll and default value for groups the trivial 1 (and therefore always valid). To ensure that there least one non-trivial value for groups, we randomly sample in, channels and out_ aat_nnlneel within the value range such that their greatest common divisor is greater than 1. Figure 6e shows that the accuracy steadily drops as we increase the magnitude of values: even GPT-4-Turbo can only solve 24% of the hardest subset of problems, which other models drop below 14% for the same problems.
evenly. The
Fold in PyTorch aims to combine an array of sliding local blocks into a large containing tensor. The constraint required for fold is the most complex out of all studied APIs where the LLM tries to generate a k_size tuple, and the product of the tuple must divide the 2nd index of the input_shapes evenly. Furthermore, also needs to satisfy a complex equation over multiple parameters as shown in Figure 6f. We use the default values for all parameters other than out out_size and ask LLMs to produce the correct k_ size. Shown in Figure 6f, due to the complexity of the constraint, even on the lowest difficulty with small values, L.MSs achieve relatively poor accuracy compared to other APIs. As we increase the values, the accuracy drops to nearly 0%. This highlights the high degree of difficulty in many DS APIs which current LLMs cannot reliably solve.
Findings: Arithmetic parameter constraints in DS APIs are extremely challenging for all LLMs. Our results show that current state-of-the-ad LLMs cannot effectively solve such complex construints with their performance drops drastically and even sometimes drops to zero increase the difficulty.
Set-related. transpose in NumPy attempts to rearrange/transpose the input_ _atua according to the given new new_dim. In transpose, the constraint is that the model-predicted new_dim must be a permutation of the original dimensions in input_data. We found that the LLMs tend to predict very simple permutations; as such, similar to reshape, we directly provide the first dimension of new_de _dim to increase the difficulty. We see that in Figure 6g, LLMs generally perform well solving this constraint, and their performance improves with larger model sizes. Interestingly, the lowest difficulty of rank 2 has drop in performance. We theorize that this is because when the rank is 2, it is
8

---

Table 2: DSEVAL benchmark result. Each column shows both the accuracy/diversity and ranking (2)
O Py Torch
@ NumPy
Size
Easy Acc (2)
Medium
Hard
Div (2)
Easy Acc (2)
Modium
Hard
Div (2)
Acc (2)
Acc (2)
Acc (2)
Acc (2)
GPT-4-Tubbo
77.2
66.2 (1)
(1)
(-)
85.1 (1)
71)
-()
33b
64.7 (5)
41.5 (4)
28.2 (5)
25.8 (6)
78.5 (3)
57.0 (2)
48.8 (3)
20.9 (1)
or DeepSeck
6.7b
66.2
39.8 (5)
33.4 (4)
38.8 (4)
73.3
45.8 (8)
35.6
(7)
1.3h
59.0 (8)
34.4 (6)
26.8 (6)
(6)
(8)
46.3 (7)
30.5
(7.8
00 CodeLlama
13b
64.7 (6)
44.6 (3)
(4)
29.2
74.4 (4)
48.5 (6)
36.8 (6)
(3.
62.6 (7)
32.7 (8)
13.8 (8)
21.2 (7)
67.1 (7)
53.2 (5)
45.4 (5)
18.7 (4)
B StarCoder
15b
65.6 (4)
46.3 (2)
39.2 (2)
39.9 (2)
70.8 (6)
56.7 (3)
51.5 (2)
18.3 (5)
@ CodeQuen1.5
7b
67.5 (2)
33.2
25.2
53.2 (1)
80.0 (2)
54.7 (4)
47.1 (4)
19.3
more common to directly call transpose() without any additional arguments. Therefore, the LLMs struggle bit when given this unnatural task when asked to predict new_dim in low ranks.
Findings: We found that LLMs generally perform well across the set-related constraints, and their performance scales with increasing model sizes. However, they still struggle with unconmon or unnatural inputs that are no commonly seen during training.
Instruction-tuned models. We additionally investigate the performance of instruction-tuna (IT) LLMs [59] with bhain-of-thouge (CoT) prompting [51]. Due to computational limitations, we selected constraints from PyTorch on which GPT-4-Turbo (without CoT) performs poorly for this experiment and analysis. The detailed experimental setup is described in Appendix I. Recall that for Conv2d, the task essentially to predict groups such that it is common divisor of two integers. As observe that some models tend to predict trivial answer 1, specifically mention "Don't set groups=1" in the prompt and consider such answer as invalid in evaluation. From Figure 6h, we observe that GPT-4-Turbo with CoT performs well at this non-trivial task, maintaining over 85% accuracy even with values up to 255, By contrast, the best open-source model can only solve 22%! This shows that although models like CodeQwen achieves close performance to GPT-4-Turbo on existing popular benchmarks like HUMANEVAL [10], there is still a huge gap in terms of coding and math reasoning ability between 4 and other open-source models. Mcanwhile, when we use the same setup on the extremely difficult constraint in Fold, we see that even GPT-4-Turbo fails to perform well (less than 5% accuracy in later difficulty settings). This demonstrates that while CoT prompting may elicit better performance in constraints like in Conv2d, it still cannot effectively handle other more complex arithmetic constraints. In addition to CoT, we also test ReAct [57], another prompting strategy to elicit more reasoning process from LLMs. We observe that while ReAct can perform better than CoT, it still fails to solve more complex arithmetic constraints (detailed in Appendix J). Additionally, we attempt to include API documentation in prompts, but found that this does not always improve performance on our tasks (detailed in Appendix K).
4.4 DSEVAL: A public benchmark for numerical DS API constraints
Based on the above findings, we further construct a public benchmark DSEVAL with the same individual parameter prediction setting and the same representative set of APIs as studied in the Section 4.3. For each API the benchmark, there are 7 different difficulty settings (grouped as easy, 3 medium, and 2 hard ones) and each with 200 randomly created problems. In total, this gives us 19,600 problems in DSEVAL to extensively evaluate the performance of different LLMs.
Table 2 shows the accuracy and diversity of all 8 models. First, we observe that the LLMs' accuracy drops when increasing the difficulty levels on the benchmark problems. This is also reflected by prior results where LLMs across the board struggle with more difficult problems. Next, we see that GPT-4- Turbo consistently achieves the highest accuracy across all difficulty levels, showing the gap between state-of ohethe-rrt proprietary models and other open-source LLMs. Furthermore, we observe some interesting ranking changes across difficulty levels. For example, while CodeQwen1.5 [3] achieves the second-best performance in the lowest difficulty level, its performance drops substantially on the medium and hard problems (second worst on PyTorch medium and hard). Other models like StarCoder [31] improve their relative performance and achieve higher ranking more difficult

---

problems, showing that different LLMs can perform differently depending on the input and constraint required to satisfy.
We also study the diversity (see Appendix G for more details) of the LLM outputs, except we do not study GPT-4-Turro due to its cost. Interestingly, LLMs which achieve high ranking in accuracy do not necessarily perform well in generating diverse correct solutions. This indicates that certain LLMs generate similar solutions satisfy the constraint, without paying attention the specific context. Therefore, they are not suitable for tasks like fuzz testing [16] which requires efficiently exploring a large solution space, or for tasks involving uncommon API usage. We further categorize some common mistakes made by LLMs on DSEVAL and provide additional insights in Appendix E. Overall, DSEVAL serves as the first benchmark to systematically evaluate the performance of LLMs on satisfying complex numeric API constraints for popular DS libraries and can be extended to support additional APIs and DS libraries.
Related work
LLMs for code. LLMs have made remarkable advancements in a wide range of coding tasks,
including code synthesis [60, 10, 2], debugging [11, 8), repair [53, 54, 7], and analysis [36, 56, 55].
Notably, recent works [29, 16] also demonstrated LLMs' effectiveness in synthesizing DS code, which requires programming proficiency in DS APIs from specialized libraries such as NumPy [38] and PyTorch| [41]. Trained on billions of code including such DS code, LLMs, such as StarCoder [31] and DeepSeek Coder [22]. have been extensively evaluated on DS code synthesis tasks. However, no prior study has systematically examined whether LLMs can indeed understand numerical API constraints of these scientific libraries instead of just memorizing the trained data [14].
Coding benchmarks for LLMs. Most code generation benchmarks [10, 33, 2, 22] are formulated with natural language description and tests to verify the functional correctness of I.L.-genertedd code. However, these benchmarks mostly target general-purpose code. access L.IM code generation for DS tasks, DS- (100 [29] is created by collecting real DS problems from StackOverflow, and ARCADE [58] evaluates LLMs' ability to solve multiple interrelated problems within DS notebooks. Compared existing DS benchmarks, our study explores different granularity levels systematically evaluate to what extent LLMs can implicitly learn DS APIs' numeric parameter constraints.
Math reasoning of LLMs. To evaluate LLMs' arithmetic reasoning performance, GSM8K and other benchmarks [12, 42, 35, 24, 28] construct math problems in natural language requiring mathematical computations to solve. Compared to these cxisting benchmarks, problems designed in our study implicitly encode the arithmetic logic inside the DS library API, and thus can evaluate the LLMs' capability in understanding and solving numerical API constraints in the important DS libraries.
Conclusion
In this paper, we present the first systematic study on how LLMs understand the numerical API constraints for important DS libraries. Our study results show that current LLMs often memoize common patterns rather than truly understanding the actual numerical API constraints. Moreover, GPT-4-Turbo largely outperforms other open-source source models and can well understand some simple arithmetic constraints using COT. Based on our finding results, we also constructed DSEVAL, the first benchmark (with 19,000 problems) for systematically evaluating LLMS' capabilities in understanding the important numerical API constraints for popular DS libraries (such PyTorch and NumPy).
10

---

Acknowledgments and Disclosure of Funding
This work was partially supported by NSF grant CCF-2131943 and Kwai Inc. This project is supported, in part, by funding from Two Sigma Investments, LP. Any opinions, findings, and conclusions or recommendations expressed in this material those of the authors and do not necessarily reflect the views of Two Sigma Investments, LP.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1ige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[2] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoe Le, and Charles Sutton. Program synthesis with large language models, 2021.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[4] Silvio Barra, Salvatore M. Carta, Andrea Corriga, Alessandro Sebastian Podda, and Diego Re- forgiato Recupero. Deep learning and time series-to-image encoding for financial fore- casting, IEEE/CAA Journal of Automatica Sinica, 7:683-692, 2020. URL https: //ttss://ap...
semanticscholar. org/ org/CorpusID:: : 218468218.
[5] Clark Barrett and Cesare Tinelli. Satisfiability modulo theories. Handbook of model checking, pp. 305-343, 2018.
[6] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint 2rXiv:2007.12255, 2022.
[7] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. Repairagent: An autonomous, llm-based agent for program repair. arXiv preprint drXiv:2403.17134 2024.
[8] Nghi Bui, Yue Wang, and Steven C.H. Hoi. Detect-locaaiie-repair A unified framework for learning to debug with CodeT5. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 812-823, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- .rg.llogg..rg/22...
tics. doi: 10. / findings-emnlp.57. URL findings- emmlp 57.
[9] Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2722-2730, 2015, doi: 1..100//CCV22015.1122
[10] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint Xiiv:2107.03377 2021.
[11] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug, ArXiv, abs/2304.05128,, 2023. URL https://p/pp http:p:://ppp..mmmtiicccho..... semanticschooar. org/Coo rrggCorrpusI:: 258059885.
[12] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthiss Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
11

---

[13] Leonardo de Moura and Nikolaj Bjørner. Z3: An efficient smt solver. In C. R. Ramakrishnan and Jakob Rehof (eds.), Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-78800
[14] Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Benchmark probing: Investigating data leakage in large language models. In NeurIPS 2023 Workshop Backdoors in Deep Learning- The Good, the Bad, and the Ugly, 2023.
[15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 10099/CVVR2299990068888
[16] Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2023, pp. 423 423-433 435, New York, NY. USA, 2023. Association for Computing Machinery. ISBN 9798400702211. I. doi: 10. 10.114553599226.359888 URL
bettps:/ddo..rrgge
/114/359//99666. 3598067.
[17] Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang. Large language models are edge-case generators: Crafting unusual programs for fuzzing deep learning libraries. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE '24, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400702174. doi: 10. 10.11455357755033623333 URL https:///oi. org/10.........
[18] Yue Deng, Feng Bao, Youyong Kong. Zhiquan Ren, and Qionghai Dai. Deep direct reinforce- ment learning for financial signal representation and trading. IEEE Transactions on Neural Networks and Learning Systems, 14653-664, 2017. URL https: piteps::/apl/ htpp//ap..... http://aap..mmaaiiisssssso semanticscoola. org/Corph ttg/CorpusID. 9398338.
[19] Zhiyu Fan, Xiang Gao, Martin Mirchev, Abhik Roychoudhury, and Shin Hwei Tan. Automated repair of programs from large language models. In Proceedings of the 45th International Conference on Software Engineering, ICSE '23, pp. 1469-1481. IEEE Press, 2023. ISBN 978 9781665457019. doi: 10. 1111109/CE444619 2023000009222202222222 URL https://doi. org/10. 101.110.//1 ICSE48619, 2023.00128.
[20] Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. Codebert: A pre-trained model for programming and natural languages, 2020.
[21] Ken Gu, Madeleine Hounde-MeLaughlee Andrew McNutt, Jeffrey Heer, and Tim Althoff. How do data analysts respond ai assistance? wizard-of-oz study. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-22, 2024.
[22] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. Deepseek-coder. When the large language model meets programming-ther rise of code intelligence. arXiv preprint arXiv:2401.141990 2024.
[23] Qianyu Guo, Xiaofei Xie, Yi Li, Xiaoyu Zhang, Yang Liu, Xiaohong Li, and Chao Shen. Audee: automated testing for deep learning frameworks. In Proceedings of the 35th JEEE/ACM International Conference on Automated Software Engineering, ASE '20, pp. 586-499 New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450367684, doi: 3416571.
10.1 N0.1145/33248843441667 URL https: http:s//:///do http://doo...gge.. 0rg/10.
[24] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[25] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
12

---

[26] Md Johirul Islam, Giang Nguyen, Rangeet Pan, and Hridesh Rajan. A comprehensive study on deep learning bug characteristics. In Proceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium the foundations of software engineering, pp- 510-520, 2019.
[27] Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(())4909-499) 4926, 2022. doi: 10.
1 1109/TTTS.202.3005552
[28] Rik Kedzioik. Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow (eds.), Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1152- 1-52-1157, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10. 10118866/1/111111
URL StRL:tht::// acc wwweclannhlooyyooro.ooo wwggN16--oa 1136.
[29] Yuhang Lai, Chengxi Li, Yiming Wang. Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319-18345. PMLR, 2023.
[30] Meiziniu Li, Jialun Cao, Yongqiang Tian, Tsz On Li, Ming Wen, and Shing-Chi Cheung. Comet: Coverage-guize
model generation for deep learning library testing. ACM Trans. Softw. Eng. Methodol., 32(5), jul 2023. ISSN 1049-331X. doi: 10.1145/3583566 URL org/10. 10. 1145/3583566.
https://doi: http://oo...g//1/0//
[31] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff. Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki. Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.0611 2023.
[32] Jiawei Liu, Jinkun Lin, Fabian Ruffy, Cheng Tan, Jinyang Li, Aurojit Panda, and Lingming Zhang. Nnsmith: Generating diverse and valid test cases for deep learning compilers. In Proceedings of the 28th ACM International Conference on Architectural Support for Pro- gramming Languages and Operating Systems, Volume 2, ASPLOS 2023, pp. 530-543, New York, NY. USA. 2023. Association for Computing Machinery. ISBN 9781450399166 doi: (41144/3575593.35557700 URL https: //o.i wtt.:oo........./5..e1111111111. /1t..11/////1.1515111111
[33] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation. In seirth- Conference on Neural Information Processing Systems, 2023. URL https:
/ //openreview wwt mett/forumidd-qqxx1100
[34] Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.0228 02977, 2024.
[35] Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023.
[36] Mohammad Mahdi Mohajer, Reem Aleithan, Nima Shiri Harzevili, Moshi Wei, Alvine Boaye Belle, Hung Viet Pham, and Song Wang. Skipanalyzer: A tool for static code analysis with large language models, 2023.
[37] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang. Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations, 2022. URL https://apl semanticscholall org/ Perg/CorpuIID. 252668917.
[38] Numpy. The fundamental package for scientific computing with python. Eestps:/nnumne org, Accessed: May, 2024.
13

---

[39] Numpy. Numpy documentation. httpss//oumpy.org//b/////cee Accessed: May, 2024.
[40] Numpy. Numpy unit tests. https://github. http://atttub..cmmmmuneeae
com http://wwuy//uuggttttr///tt//uaa
tests, Accessed: May. 2024.
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https::/prroccoddingg meer neurips cc/paper_ files/papert
pdf.
[42] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems?. 2021.
[43] PyTorch. Pytorch documentation. http:://prrorrd..ooooooo///a//aa cessed: May, 2024.
Ac-
[44] PyTorch. Pytorch unit tests. http::///wtt.....oo.o//c//////// Accessed: May, 2024.
[45] Steven Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D. Weisz. The programmer's s assistant: Conversational interaction with a large language model for software development. In Proceedings the 28th International Conference
Intelligent User Interfaces, IUI '23, pp. 491-514, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701061. doi: 10.1 10.114575881644335444377 URL https::/ oooei.orrg/1..111 3581641.m 3581641.3584037.
[46] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving, 2016.
[47] Yiyin Shen, Xinyi Ai, Adalbert Gerald Soosai Raj, Rogers Jeffrey Leo John, and Meenakshi Syamkumar. Implications of chatgpt for data science education. In Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. I, pp. 1230-1236, 2024.
[48] Mohammed Latif Siddiq. Joanna C. S. Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinicius Carvalho Lopes. Using large language models to generate junit tests: An empirical study, 2024.
[49] Wil Van Der Aalst and Wil van der Aalst. Data science in action. Springer, 2016.
[50]
Anjiang Wei, Yinlin Deng, Chenyuan Yang, and Lingming Zhang. Free lunch for testing: fuzzing deep-learning libraries from open source. In Proceedings of the 44th International Conference Software Engineering, ICSE '22, pp. 995-1007, New York. NY. USA, 2022. Association for Computing Machinery. ISBN 9781450392211. URL https://doi. org/ 10. 1144/3551000 .01.110000.333101...
doi: 10.1145251003.3510041
[51] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. ofain-of-thhoghh prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[52] Wikipedia contributors. Plagiarism Wikipedia, the free encyclopedia, 2024. URL https: wen.ieiiieii wtt.:/ippeeaa..oooiii////eeeeeee wwtg/wiki//ellinggrrrr distance. [Online; accessed 20-May-2024J.
[53] Chunqiu Steven Xia and Lingming Zhang. Less training, more repairing please: revisiting automated program repair via zero-shot learning. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 959-971, 2022. 2022.
14

---

[54] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang, Automated program repair in the era of large pre-trained language models. In Proceedings of the 45th International Conference on Software Engineering. ICSE '23, pp. 1482-1494. IEFE Press, 2023. ISBN 978 665457019. doi: (0.100//CSS4441192200200 200.:199..22.0111111 29.... URL. httpht:ss:/dde https:///ooo../////// 10. 1ww.111191//// ICSE48619 2023. 00129.
[55] Chenyuan Yang, Zijie Zhao, and Lingming Zhang. Kerellpt: Enhanced kernel fuzzing via large language models. arXiv preprint arXiv:2401.00566 2023.
[56] Chenyuan Yang, Yinlin Deng, Runyu Lu, Jiayi Yao, Jiawei Liu, Reyhaneh Jabbarvand, and Lingming Zhang. Whitefox: White-box compiler fuzzing empowered large language models. Proceedings of the ACM on Programming Languages, 8(00PSLA2):709-7355 2024.
[57] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. The Eleventh International Conference on Learning Representations.
[58] Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. Natural language to code generation in interactive data science notebooks. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 126-173, 2023.
[59] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang. Fei Wu, al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308. 10792, 2023.
[60] Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. Planning with large language models for code generation. The Eleventh International Conference on Learning Representations, 2023. URL httpss//openreweeee hwt.:/iwwwwww....... net/forum? 1d=Lr8cD0tybfL.
15

---

A Problem statement
We first begin by describing the type of programs we are targeting well as any terminology definitions. In our study, we aim to verify the ability of LLMs on satisfying the numerical constraints for DS library APIs. Figure shows an example of typical DS program consisting of different DS APIS created by following these steps: i) importing the DS library (e.g., PyTorch) to be used in the program: ii) obtaining or generating input data; iii) performing data manipulation operation using the input data to produce outputs. In order for a DS program to be valid, it needs to satisfy the constraints required in the DS library APIs used. Next, we will describe each component of DS program in more detail.
DS APIs. After importing the DS library. we start by obtaining or creating some input data to be used in the program. The data creation process is done using a specific type of DS APIs, refer to by us as generation APIs. The output of generation APIs is a specific data structure (e.g., tensors, arrays, dataframes) used by each DS library defined by the parameters of the APIs. In the example from Figure 1, the generation API is torch. randn to produce the input data of x. The DS library specific input_data commonly has the following properties: i) shape: the size, rank or dimensions of the data structure (e.g.. [20, 16, 50, 1000]) ii) dtype: the type of primitive data in the input_ __datadtaaaa (e.g., float) tii) value: the exact data values in the input_datte While most of the constraints in DS library APIs focus relationship of the shape, both dtype as well as value can be important to satisfy additional constraints. Furthermore, more complex generation APIs can create heterogeneous input_daaa _ata that contains different dtypes or even nested structures.
Using the iaput_data created by the generation API, DS programs then use manipulation APIs to perform additional operations. Different from generation APIs where the output produced depends only on the provided API parameters, manipulation APIs create the output based on both the API parameters and the input data. We refer to API parameters the options used to initialize the behaviour of the manipulation API. In the example, the manipulation API is Conv2d and is first defined with a set of parameters (e.g., kernel_size) and then in the next line applied the output y. For the program to be valid, the parameters of the manipulation API must satisfy the corresponding constraints between API parameters and the properties of the input _ata. Note that we consider both model manipulation APIs (like Conv2d in the example) as well as sequential manipulation APIs where the input_ data is directly provided as a parameter of the API.
create
API constraints. We refer to API constraints the set of relationships between properties of input_date and API parameters that, if and only if when satisfied, leads to a valid DS API invocation. Figure provides some of the example API constraints for Conv2d. We observe that not only are there constraints between the properties of the input_ _ata and API parameters (e.g.. in channel % groups o, kernel_size size <
padding), but there are also constraints within API parameters (e.g.. out channel % groups = o). Failure to satisfy any one of those constraints will lead to an invalid DS API innovation where, when executed by the DS program, will lead runtime error.
B Benchmark details
Table 3 lists all the 12 APIs we studied in Section 4.3 and Section 4.4, where we ask LLMs to predict a single API parameter. In Column "Constraint", the underlined parameter is the one that the models need to predict, and we also list all the parameter constraints related to it. Column "Category" presents the categorization following Table 1, highlighting that our selected APIs and API parameters can cover all the categories and are representative of their group.
Difficulty settings. As discussed in Section 1.1, we have two different difficulty settings depending on the specific APIs, namely rank and rank. Below are the detailed setups for each API.
For torch. max, np- squeeze, np- argmax, are more related to tensor dimensions, we design the difficulty level by increasing the rank of input data (i.e., how many dimensions it has).
np. transpose, mp.max whose constraints that
For torch. nn. BatchNorm2d, torch. nn. Fold, torch. nn. Conv2d, torch. nn. Maxxoo112, np. split whose constraints that are more related to the actual values (either API parameter or the dimensions of input data), we design the difficulty level by increasing the range of relevant values.
16

---

For torch. reshape, up reshape, since their constraints are closely related to both rank and value, we study both settings for each of them.
Table 3: List of APIs and corresponding constraints used in DSEVAL.
Library
API full name
Constraint
Category
O PyTorch
torch. nn. Batchhorm2d
features input, shape
Equality
O PyTorch
torch.max
dim
Inequality
O PyTorch
Norch.nn.Foo
Arithmetic
% k_size
O PyTorch
torch. ferch.nn.cc wn..nn..oo Conv2d
groupe
Arithmetic
out_ channels X groups
O Py'Torch
torch. merch.nn.he MaxPoo12d
kermel_eize R padding
Arithmetic
PyTorch
torch torch.reshape
[inpat_ [ Saargtt_sae
Arithmetic
NumPy
np. squeeze
soput__ [axis]
Equality
NumPy
argarx
-rank S axia
rank
Inequality
NumPy
np. np.resh
saapat__shape I ahape
Arithmetic
NumPy
mp. split
(at___chae(axii X section-O
Arithmetic
C NumPy
np. transpose
-rank (input_ehaped) (azee)
for dim A
Inequality
Sct-related
-rank S din rank for dim in axie A
Inequality
NumPy
mp.max
{axis)
|axia|
Set-related
C
Common parameter value ranges
To create a set of diverse problems for LLMs to predict a single API parameter, we randomize the context, namely the input data shape and other API parameters. Since the goal of study is to focus on numeric API constraints, we want to control the complexity or naturalness of the constraint- related variables, and ensure that the other unrelated parameters always within reasonable and common range. More specifically, during problem creation, the irrelevant API parameter is an optional parameter, we just use its default value. If the irrelevant API parameter is required API parameter, then we randomly choose a value according to the common value range listed in Table 4. For example, for torch. www.ann.CCnnn..mmmmmmm (in_channels, out_ nut_channels, targeted parameter is groups, and sernel_size is a irrelevent but required parameter. Therefore, across all difficulty levels, we pick kernel_size randomly from [1,10].
pernel_size, groups-1), the
To obtain Table 4, we perform an extensive study and refer to developer- written unit tests [44, 40], API documentations [43, 39), and existing DS library fuzzing literature [50, 23, 32, 30] gather the common value range for each API parameter.
D Case study of the Linear API
When evaluated using the full API parameter setting (Section 4.2), observe much more significant accuracy drop for Linear compared to the other APIs (Figure 5). To gain deeper insights, we conduct an in-depth case study of this API.
For torch. nn. Linear fiin_featuree out out_features), the only constraint that the in fnauuuue should match the last dimension of the input tensor. However, the DeepSeek Coder-33b model tends to copy the wrong dimension of the input tensor, likely because it has not seen lots high-rank tensors (rank 4) in the pre- pre-training data.
17

---

Table 4: List of APIs and corresponding common input range used in DSEVAL.
API full name
Parameter name
Range
in_
[0, 128]
cat_nells
[0, 128]
kernel_size
[1, 10]
Conv [1 d (O)
stride
[1, 5]
padding
[0, 9]
dilation
[1,5]
kernel_size
[1, 10]
MaxPool2d (0)
stride
[0, 5]
padding
[0, 9]
BatchNorm2d (0)
num num_ features
[0, 256]
expand (6)
last_dim
[-10, 10]
[argmin argmax] (0,0)
keepdims
[True, False]
reshape (0,0)
out_shape
[0, 256]
in_features
256]
Linear (0)
out_features
[1, 256]
bias
(rree, False]
output_size
[2, 10]
kernel_size
[1, 10]
Fold (0)
stride
[],
padding
[0, 9]
dilation
[1,
Japut_shape
[0, 256]
APIs input_datt (0,@)
input_rank
[2, 8]
1.8
1.4
25-576
25-577
25-35)
25-359
48T6-Tarto
0.8
IPTe-Tarto
(L-76
(L-75
(Laias
(UPESS
Marcaaer Codeture-1.3
accurnty
Starcater
tedeburn-1.5
0.0
diffsnity (963)
diffioity (mwe)
(a) Ó PyTorch Linear Full API parameter (b) Ó PyTorc Linear Single API parameter
Figure 7: Result on torch. nn Linear using different LLMs
We further evaluate this phenomenon across different LLMs. Figure 7b shows the results for both the full API parameter and single API parameter setting for torch. nn. .n. Linear as we increase the difficulty (rank of the input data) across 8 LLMs. Similar to DeepSeek Coder-33b, the performance of other LLMs also drops significantly when rank reaches Afterwards, the performance stabilizes for higher difficulties (i.e., rank 4) especially for open-source LLMs. This is true for both the full API parameter and single API parameter setting.
Surprisingly, we found that even the state-of-the-arr GPT-4-Turbo drops in performance when the rank reaches 4. However, we see that GPT-4-Turee was able improve its performance in higher difficulties (i.c., rank > 6). After looking the results, we found that for lower ranks, GPT-4-Turbo
18

---

tends to use other APIs as "short-cuts" and forgo the analysis on torch nn. Linear directly as shown in Figure 8.
import torch
torch.randn(1. 10, torch.nn. Lww.h/mniiieerr81100000 m(x.iew((1.1))
Figure 8: Example GPT-4-Turbo's incorrect response. this example, instead of operating on the original input tensor and set in_feature to its last input dimension (10), GPT-4-Turbo multiplies all the dimensions together and performs a flattening operation (x. view(1, -1) before invoking Linear). This violates the instruction to the model, which prohibits modifying the API invocation code. As such, this model response evaluated as incorrect.
E Common mistakes made by LLMs
In this section, categorize common mistakes made by LLMS during our experiments and offer some additional insights and explanations.
LLMs struggle with uncommon input tensors: We found that across many APIs and constraints, LLMs struggle when provided with uncommon input tensor ranks (i.e., rank > 4) or uncommon shapes (e.g., = torch. (and 30, 23, 4)). The reason is that LLMs are mostly trained with data that contains very common shapes or ranks. As such, LLMs can easily make mistakes on uncommon inputs.
LLMs tend to predict common parameter values blindly: We also observe that LLMs tend to generate common parameter values (e-g., 0, 1, powers 2) which often turn out to be incorrect. This is again because LLMs are trained with pre-training code that frequently contains such parameter patterns and thus are likely to predict them even given a different input context.
LLMs pay attention to the wrong tokens/irrelevant parameters: LLMs can learn spurious correlations and pay attention to the wrong context tokens. For example, open- source LLMs struggle with the simple equality constraint in fe..fuurus=inputtt shape [-1] in torch. nn Linear because the attention weights are focused on the irrelevant parameters.
F Additional individual API parameter results
In this section, we provide the additional results for the individual API parameter setting.
Reshape. reshape in NumPy contains the same functionality and constraint as the Py Torch. Figure 9a shows the result across difficulty dimensions. Similar to the result discussed in Section 4.3, also observe the same trend in NumPy where GPT-4 is able to achieve superior performance.
MaxPool2d. MaxPool2d in PyTorch applies a 2D max pooling over a 4D input tensor. We fo- cus on predicting the API parameter padding, where needs satisfy the following constraint: kernel_size < H 2+padding (the problem is already simplified by setting stride to I and setting W-H). Figure 9b shows that even GPT-4 is incapable of getting this type of non-trivial linear constraints right when the value range increases to [128, 256), and we observe that it tends to predict O which the default value for padding and therefore fails to satisfy the second constraint. Meanwhile, DeepSeek Coder models, especially the 6.7B variant, do surprisingly well in the highest difficulty level we studied. After inspecting the outputs, we find that the DeepSeck Coder-6.78 model is able to predict an expression Kernel_sized /2 instead of constant number for padding, and since the expression correctly characterizes the constraints it always leads to a valid solution.
Squeeze. squeeze in NumPy aims to remove any dimension of length one from the input_ data. The constraint is for the LLM to predict a dimension dim where input_ caatt_shapes_diii = 1. We add an additional constraint when generating the input_dats such that there only one dimension with length one (only one correct dimension). Figure 9c shows the result as increase the rank
19

---

1.8
1.0
25-333
0.5
GPT4-Turbo
0.3
6PT4-Turbo
Bisput, shapes Enov. shopes
0.6
Input, shapes partiss shape
X rancoe randd,, 15, ID) ro.reehppex, nMrsttppes)
0.2
0.2
0.0
9.9
difficulity (irput_dis)
aifficulty
(a) PyTorch Reshape
1.0
05-1.36
1.0
05-1:3b
05-6.7b
05-6.7b
05-330
0.0
Irpat shopes
0.6
occuracy
9.6
irpat couptsccount(1
sp-randen. randly, exis-al.)
0.4
0.2
0.2
0.0
(4,8)
(0,18) [18,32] (52,64) (1)2,2)
difficulty linput die)
diffiealty (rank)
(b) O PyTorc MaxPoo12d
(c) @ NumPy squeeze
05-1.36
86-6.70
0.6
Input__happee (tapse/dta//ttddee
0.5
9.4
difficutty (rank)
(d) NumPy split
Figure 9: Single API parameter result (cont).
in input_data. We observe that while in smaller ranks (2-3), the LLMs achieve close to perfect accuracy, the performance quickly drops off when we increase the rank to be >4, where all 3 LLM sizes perform similarly. We found that this because LLMs tends to generate dimensions of 0 or (commonly seen in example code and pre-training data). As such, when given high ranked tensor where the correct squeeze dimension can be much higher than 0 or the LLM struggles to satisfy the simple constraint. This again demonstrate the memorization issue where LLMs tend to predict commonly seen parameters during training instead of reasoning over the actual API constraint.
Split. For split in NumPy, the goal is predict a dimension dim which can divide the index parameter index evenly. Just like squeeze, we also add an additional constraint when generating the input_data such that there is only one dimension that is divisible by index. Figure 9d shows results, where we see that even with rank of 2, the accuracy is just above 50%, showing the increase in difficulty of the constraint in split t the rank increases, we also observe huge decrease in performance, where LLMs again overwhelmingly predict dimensions of 0 or 1.
20

---

G
Diversity metric
Besides aceuracy (percentage of valid programs generated), also measure the diversity the valid programs generated. In particular, based on the exact API, we use different diversity metric:
For APIs where the number of possible valid outputs are large and not restricted (e.g.. torch. reshape), we sample the LLM multiple times (100) with high temperature (1) and com- pute the percentage of unique valid programs generated as discussed in Section 3.2.
ii) For APIs where the number of possible valid outputs are fixed (e.g., mp-max can only select valid dimensions within range), again sample the LLM multiple times and then compute the distance between uniformed valid distribution and the distribution produced by the LLM. For example, if the set of all valid answers is (-1, 1), and the model predicts (-3, -2,0,0,00000 ,,,0,,,,,,,,,, 1,,1,11 in 10 samples, then the LLM's distribution P={-1: O, 0: 0.3, 1: 0.5, others: 0.2), and the reference distribution is Q=(-1: 1/3, 1/3, 1/3, others: O}. Next, compute the Hellinger distance [52] between the two discrete probability distributions and compute the diversity distance.
Since it requires a large amount of sampling programs (100 samples per problem) to evaluate diversity, in Section 4.4, we evaluated the diversity of each LLM only on single difficulty level, i.e., the third level, either r or value in (8,16).
H GPT-4-Turbo infilling prompt
You an expert Python programmer and good writing correct PyTorch code. Please complete the program by filling in the correct API parameter(s). You should keep the exact same program unchanged, just fill the missing code part.
System Prompt
"python
import torch x = torch.randn(1, (1, 2, 3)
Example Input
Norch.nn.Conv2de. 2, groups=<insert code here>) y = mx)
"python
Example Output
import torch
torch.randn(1, 2) 2, groups=2)
m y y=m mx()
Figure Example GPT-4-Turbo prompt used for infilling and output
GPT-4-Turbo is instruction-following LLM and we do not have access to base version that supports direct infilling. Therefore, we use a infillspecific prompt to ask GPT-4-Turbo to only fill in the missing code without adding any additional text. This setup allows us compare against other infilling LLMs in the same setting. On the other hand, in this paper (e.g., Fig. 6h), we use SurTo--Turbo-B
to differentiate the free-form generation setting. GGPT-4-Turbo-Ing indicates that we allow to generate additional text (such as CoT or ReAct reasoning steps).
Figure 10 shows the prompt used by us perform infilling using GPT-4. Note we separate out the system prompt and how we format an example input. Additionally, we modify the library name the system prompt depending library of the API.
21

---

You are an expert Python programmer and are good at writing correct (library) code. Please complete the program by filling in the correct API parameter(s).
GPT-4-Turbo CoT Prompt
You are an Al programming assistant, utilizing the DeepSeek Coder model, developed by DeepSeek Company, and you only answer questions related to computer science. For politically sensitive questions, security and privacy issues, and other non-computer science questions, you will refuse to answer. ### Instruction:
Please complete the following Python program in a markdown style code block. Replace "[INSERT HERE]" with the correct API parameter(s). You should keep the exact same program unchanged, just fill in the missing code part.
Deepseek Coder-Instruct CoT Prompt
You are an expert Python programmer and are good at writing correct PyTorch code. Please think step by step and complete the following Python program in markdown style code block. Replace "[INSERT HERE]" with the correct API parameter(s).
CodeQwen1. 1-5hhat CoT Prompt

 You are an expert Python programmer and are good writing correct (library) code. Please think step by step and complete the following Python program in markdown style code block. Replace "[INSERT HERE] with the correct API parameter(s).
CodeLlame-Instruut CoT Prompt
Figure 11: CoT prompts used for the instruction LLMs
I Single API parameter results for instruction model with CoT prompting
Experiment setup. We design CoT prompts shown in Figure 11. Note that the prompt is slightly different for different models due their specific format requirements. Additionally, for torch.nn. nn. Conv2d, we use a custom prompt (shown in Figure 12) where we explicitly add a sentence "Don't set groups=1" to 512 for all models and all APIs, except for torch. nn. Fold we use LLMs tend to predict longer text before the actual code for this challenging API.
avoid trivial answers. We use greedy decoding and max_new_ tokens maw_tokens=1024 since
Results. Figure 13 shows the additional results on constraints chosen evaluate the instruction- tuned LLMs. Figure 13a, 13b, and 13c shows the result of the different variants on Conv2d. Furthermore, we also evaluate the constraint MaxPool2d with the result shown in Figure 13d.
J Single API parameter results for instruction model with ReAct prompting
We applied the ReAct prompting strategy [57] on the difficult APIs constraints studied using CoT in Section 4.3. Figure 14 shows our ReAcT prompt for GPT-4-Turbo. Note that the prompt is slightly
22

---

GPT-4-Turbo CoT Prompt for Conv2d.
You are an expert Python programmer and are good writing correct (library) code. Please complete the program filling in the correct API parameter(s) Don't set groups=1
Example Input
"python
import torch
randn(254, 180, 153, 189)
torch.nn. Conv2d(180, groups-<inser code here>)
y m(x)
Example Output
the given code snippet, you are trying use 'torch nn Conv2d' layer, which is 20 convolutional layer PyTorch. paremetere Conv2d are:
in_channels" (int)
input image.
oou aut_channels" (Int) Number of channels produced by the convolution.
kerme_siiee' (int tuple) - Size the convolving kernel.
'stidde' (int optional) Stride of the convolution. Default:
5. ppdding" (int or tupla, optional) - Zero padding added to both sides the input. Default:
dilationn' tuple, optional) Spacing
7. groups" (int, optional) Number of blocked connections from input channels to output channels. Default: (bool, optional)
output.
For the groups" parameter your code, you need specify an integer that divides both the number input channels ii__channels' and number of output channels ('out_charnela', The 'groups' parameter to control the connections between inputs and outputs. Setting groups' to means all inputs are convolved to outputs. 'roups''
chennels", filters (like depthwise convolution).
input chennel convolved
case, since channels' is 180 and outt, channels is 135, need to find divisor of both 180 and 135 to use as the 'groups" value. The greatest common divisor (GCD) of 180 and 45. Thus, you can set grroups445::
"python
import torch
x torch.randn/254, 180, 153, 189)
torch.nn. Cow.mn(1cn1nn, 135, groups=45)
y mmx)
configuration means convolution operation performed separately 45 groups, group hendling subeet the end output channels.
Figure 12: CoT prompt and example input/output for torch, nn. Conv2d used for the instruction LLMs. In this example, GPT-4-Turbo is able accurately recall the numeric constraint for the targeted pa- rameter groups, apply them on this specific input example sin_channells-180 out_ mut_channels=135, and generate valid answer 45.
different for different models due to their specific format requirements. Our ReAct prompt asks the LLM first generate thought and then produce the code output. We also provide the LLM with a single demonstration of the ReAct task. Furthermore, similar to the CoT experiments, we also explicitly add the sentence "Don" 't
groups-1" avoid trivial answers.
Figure compares the results the instmuction-folloiinng L.Ms using CoT versus ReAct as well their base variants. We see that for torch. MaxPool2d, ReAct prompting generally performs
23

---

1.0
0.4
0.2
(32.65)
(1.4)
[4.3)
(32.84) (211128)11288229
(a) Conv2d
(b) Conv2d (excl.
6.0
0.6
0.4
0.2
6.2
071
(c) Conv2d avoid
(d) O Porch MaxPool2d
Figure 13: Instruction model results. Figure 13a shows the result where we do not add the additional non-trivial requirement the prompt ("Don't set groups=1"), and also count groups-1 answers as correct. Figure 13b shows the result on the same setting and model samples as Figure 13a but we count groups=1 answers as incorrect. Figure 13c shows the result where we add the additional non-trivial requirement in the prompt ("Don't 't set groups=1"), but we still count groups=1 answers as correct.
better than CoT especially in more difficult problem settings (e.g., at highest difficulty setting, GPT- TTurbo-ReAct. 89.5% versus
56.0%). This demonstrates the effectiveness of ReAct in generating thoughts that can help with the correct API parameter generation. However, for torch. Conv2d, ReAct performs similarly to CoT prompting. The in Conv2d much more complex, requiring factorization. As such, smaller open-source LLMs cannot perform well even with reasoning steps. On the other hand, state-of-the- LLMs like GPT-4-Turbo show their powerful reasoning abilities by improving the performance over the base variant with both CoT and ReAct. Although ReAct performs better than CoT for the easier difficulty settings in wt..nnnoolllo... Fold, its performance quickly drops in higher difficulty settings (at best ~5% accuracy with the best GPT-4-Turbo).. Overall, this experiment results demonstrate that even more advanced prompting methods such as ReAct still cannot effectively handle more complex constraints.
that the constraint used
torch. nn.
K
Single API parameter results with dbcumenttation-aaggeened prompting
We conducted additional experiments using the documentation eecuumntatiion-augmentt setting across the 3 difficult API constraints used in the CoT experiments (Section 4.3). We provide the raw documentation of each API (obtained from the source code docstring) in the prompt and apply both base and instruction- following LLMs. Figure 16 shows example prompt to perform the documentation aagggeetttio--ggmeetteee setting for GPT-4-Turbo. Note that the prompt is slightly different for different models due to their specific format requirements. Furthermore, similar the CoT and ReAct experiments, also explicitly add the sentence "Don'tt set groups=1" to avoid trivial answers.
In Figure 17, compare the performance with and without documentation. We found that there are cases where documentation can improve performance. For example, in the most difficult setting of torch. nn, Conv2d, adding documentation is able to improve performance of CodeLlama-344

---

You are an expert Python programmer and are good at writing correct PyTorch code. Please refer the given examples and complete the following Python program using these two steps:
1. Generate a thought about the API parameter values required satisfy the numeric constraints for valid program.
2. Complete the program in markdown style code block. You should keep the exact same program unchanged, just fill in the missing code part.
Examples are listed as follows:
Program:
ppython
import torch
torch. torch.randnn19, 21, 23, m torch.m..BachhNoo@@@cnneeee code here>)
y. mx)
Thought:
The 'torch.nn.. Baacc.nn.BatchNorrc Borchh API in PyTorch expects the number of channels (C) as its argument.
The input tensor 'x' has a shape of '[N, C, H, WJ', which in this case (19, 21, 23, 3)'
Therefore, the valid API parameter value should be 21, corresponding the number of channels in the input tensor 'x".
Completed Program: pyyhon
import torch
x torch.randn(19,, 21, 23, 3)
m = (orch.m..BacchNom222(21)
y m(x)
GPT-4-Turbo ReAct Prompt
Figure 14: Example React prompt used for GPT-4-Turbo
Instruct from 20% to 45% accuracy (Figure 17d). However, there also similar cases where adding documentation decreases performance. For example the GPT-4-Turbo- Turto-Tuuro-I performance falls from 57.5% to 22.5 in the most difficult setting of torch. MaxPool2d (Figure 17b).
Since we provide the raw documentation text without further processing, the success rate adding documentation can vary depending the specific model as well as the quality of the documentation. As such, this demonstrates that naively adding API documentation cannot always achieve better performance on our tasks.
L Computation Environment
We perform both LLM generation and evaluation on an 64-core workstation with 256 GB RAM running Ubuntu 20.04.5 LTS. For local open-source LLMs, NVIDIA RTX A6000 GPUs. For GPT-4-Turbo experiments, we directly access the API endpoint provided by OpenAl.
25

---

9.23
9..5
9.0
14,0 1::00 16.30 23,641
19.10018.350
[L41 14U0
10.00 10132222
(a) O Py Torch MaxPool2d
(b) O PyTorch Conv2d
(c) Ó PyTorch Fold
Figure 15: Single API parameter result with chain-of-thouge (CoT) and ReAct prompting and the base LLM using greedy decoding with 200 problems for each difficulty setting. We follow the same generation and evaluation setting used Section 4.3.
You are an expert Python programmer and are good at writing correct PyTorch code. Please refer the given API documentation and complete Python program. Documentation the torch.nn Mexxoooll2 API: input
Applies pooling
signal composed
In the simplest case, the output value of the layer with input eize meth:"(N C, W)' math: (N. H_lout, W. (_(out)) and :atrr kemel, sizee :math: (KH, kW)
precisely described as:
math::
Vhegin(alignad)
oun(N)) C.j w) & ymax (m=0, Vddots, \max (n=0, Mdoots, kW-13 !I Stex//inuut(NNi CJ.. (text(stridaj]) ttimes h m.
teset(ttiid((1) Vtimes w +)
Vend(aligned)
aatt:: padding" non-zero, then the input is implicitly padded with negative infinity on both sides for attr paciiing" number of points. attr: dilation controls the spacing between the kernel points.
Note:
When moid---R- sliding windows allowed go bounds they
left padding
input. Siiding
right padded region ignored.
The parameters :ottr: kerne :attr: sstride,, attr: padding". :attr: dilation" can aither be:
a single "int" which case same value used for the height and wictt dimension
-a "tuple" of two ints in which case, the first 'int" is used for the height dimension, and the second lint" for the width dimension
Args:
kernel size: the size of the window to take max over
stride: the stride of window. Default value att:: kemeel size'
padding implicit negative infinity padding
dilation: parameter that controls the stride of clements the window iindioss if True" will raturn the max indices along with the outputs. MexUnpool2d later compute the output shape
Useful
ceiLmmode when
use "oell instead
complete program by filling correct parameter(s). should keep same program unchanged, just fill in misesng code part.
GPT-4-Turbo Documentation-aggeenned Prompt
Figure 16: Example documentation -ugeeiitee-mmeteeedd prompt used for GPT-4-Turbo
26

---

0.40
14.00
24:00
26.301
20.44
11.41
14.11
[8,0) DE.D 28.40
100TE 201
11.41
19.41
28.30
DI.12)
D2.40 IA,UREEUN
(a) O Porch MaxPool2d base
(b) O PyTorch MaxPool2d instruct.
(c) O PyTorch Conv2d base
6.48
0.34
9.34
0.15
0.1
6.41
0.80
12.40
14:30
18.35
10,301
1501
12.0
30.0
16,30
DE30 TE 4O
[C.0
14.05
D0.05 TOUNT DE 1901000
(d) Torch Conv2d instruct.
(c) PyTorch Fold base
(f) O PyTorh Fold instruct.
Figure 17: Single API parameter result for both instruction-followiii and base LLMSs with and without documentation. We follow the same generation and evaluation setting used in Section 4.3.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
In this paper, we perform a comprehensive evaluation on the proficiency of LLMs to handle numeric parameter constraints in data science libraries. The main claims made are accurately reflecting the paper's 's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 3.1 discuss the scope of our study well assumptions made, that is, we focus numeric constraints only and assumes all other parameters are default and valid.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't seekowleddger judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
28

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In our paper, we discuss our experimental settings in Section 2.2 as well as evaluation setups in Section 3. Furthermore, in the Appendix, we provide additional details regarding each API and their constraints reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
29

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes. The data and code will be made publicly available soon, along with detailed instructions to replicate the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https: /nipp.cc.. wwtpccccpppcccccccctcccccccccccc http:ppppppppppp///ppppppppapael http://ooeuumisiiiooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section 2, Section 3 and Appendix I, we provide experimental setups for benchmark creation and evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow prior benchmarking work like HumanEval [10] and directly report the LLM performance without including error bars. Furthermore, to conduct detailed statistical test, would be extremely costly on our large dataset especially with models such
GPT-4.
Guidelines:
30

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yss]
Justification:
We provide the list of computation resources used by us in Appendix L.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that NeurlPS Code of Ethics https : ///uuenuiip.../ cc cc/pubbic/tthiccsuudeell...
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: The research conducted in this paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
31

---

We discuss the societal impacts in introduction. Since data science libraries are widely used, we hope our work can facilitate future research improve language models' ability in this domain, and enable people to write data science programs faster.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Answer: [No]
Justification: This paper constructs a synthetic dataset, and does not have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite all the models we evaluated.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
32

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, the data and code will be made publicly available soon.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper evaluates language models and does not involve human subjects.
Guidelines:
33

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "yV29W8mu2p", "reviewer": "Reviewer_qmKc", "summary": "This paper constructs a benchmark, DSEval, of 19600 programs across 28 data science (DS) library APIs with numerical constraints and uses the benchmark to invesitigate the capability of LLMs in generating valid DS programs which satisfy those numerical constraints. Additionaly, this paper categories the constraints into four different groups: equality, inequality, arithmetic and set-related. The experiments include 3 different generating settings: 1) full program, 2) all parameters, and 3) individual parameters, and study 8 LLMs including both closed-source and open-source models. The paper shows that LLMs are great at generating simple DS programs, but the performance of LLM drops significantly when the difficulty increases.", "questions": "1. Why only 28 or 12 APIs are studied given a large number of APIs?\n\n2. Could you elaborate on the details of the API constraints used in constructing the benchmark? Please see the detailed comments for clarification points.\n\n3. How does the proposed work verify the correctness of the logic of a full program?", "limitations": "The paper provides a discussion on limitation.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The paper targets at an important problem.\n+ The paper is easy to follow in most places.", "weaknesses": "- The paper lacks important details.\n- Some claims need better justification.\n- The experiment is limited in scale in terms of the number of studied APIs.\n\nOne contribution of the paper is constructing the benchmark and the paper claims that the design of the benchmark is general and can be easily extended to additional libraries. However, the paper lacks details of how the benchmark is constructed, making it difficult to evaluate this claim. First, lines 11/63/165 state the benchmark contains 28 APIs while line 80 says 12 APIs. The authors need to clarify on this. Second, how the 28 or 12 APIs are selected? what does it mean by representative? Third, do these constraints exist? If so, were they written by developers? What format do these constraints take? Are they in natural language or a formal format? If they are in natural language, how are they converted into a formal format? Forth, are these 19600 programs from existing projects? Fifth, the number of such APIs is large, but the paper only studies 28 or 12 APIs which is pretty a small scale. \n\n\nRegarding the full program setting, how the 3-step instuction is contructed for individual program? From the example, I can see 3 lines of instructions and 3 lines of code. Are all the programs composed with 3 lines of code? I believe it should not be the case. If there are more than 3 lines of code, does it mean that there will be more than one line of code for each instruction? The paper uses SMT solver to validate the correctness of the generated program. It checks the validation based on constraits. Since the constraints for sepecific for one API. I believe SMT solver can verify the correctness of individul statement. How about the logic of the whole piece of the generated program? It would be better to clarify on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: Why only 28 or 12 APIs are studied given a large number of APIs? What does it mean by representative?**\n\nGood question! Please note that although there are a large number of APIs, the commonly used ones (e.g., `Conv2d`) are not that many. For example, the widely-used benchmark DS-1000 [29] contains 1000 DS problems collected from StackOverflow problems (including 68 PyTorch problems and 220 NumPy problems), reflecting realistic use cases; meanwhile, it only covers 16 PyTorch APIs and 59 NumPy APIs (after excluding data construction APIs like `np.ones`). NNSmith [32], a popular DNN generator for testing ML libraries, chooses to support only 73 core operators that are commonly used in DL programs. Furthermore, not all of the commonly used APIs have numeric parameter constraints. \n\nAs for our API selection process, first, we followed prior work NNSmith and examined all 73 core operators it supports. Next, we select the 22 core APIs that have numeric parameter constraints and add additional 6 APIs to obtain the 28 APIs used in our study in the full program prediction setting (Section 4.1) and the full API parameter prediction setting (Section 4.2). Furthermore, we choose 12 APIs to cover the representative types of numeric constraint for detailed analysis in the single API parameter prediction setting (Section 4.3) and in our DSeval benchmark (Section 4.4). We use “representative” to mean representative with respect to the numeric parameter constraints in DS library APIs. Table 1 shows the categorization of the different types of numeric constraints that exist in DS libraries. Our selection criteria aim to select a list of APIs that have interesting numeric parameter constraints that can cover all the major constraint categories. You can also find a complete list of the 12 APIs and their corresponding constraints in Table 3 in the Appendix. \n\n**Question-2: Could you elaborate on the details of the API constraints used in constructing the benchmark?**\n\nIn this work, we focus on the numeric constraints that are part of the popular DS APIs. These constraints are directly embedded into each individual API. In other words, to use these APIs, the generated DS code must satisfy these constraints. Please refer to Figure 1 for an example of a DS API and its corresponding constraint. \n\nThe constraints are defined by developers according to the functionality of each DS API. These constraints are usually specified in natural language within the API documentation. In our study, we manually encode these constraints as satisfiability modulo theory (SMT) formulas and use an SMT solver (Z3) to check if the parameter values generated by the LLM are correct. Please see Section 2.3 for more detail. \n\nThe 19,600 programs in our DSeval benchmark are randomly generated by us and are not taken from existing projects. Each benchmark problem requires the LLM to generate a single parameter for an API in order to produce a valid program. To create these problems, we randomly generate values for both the input data and the other parameters in the target API. Using the encoded constraints, we ensure the problem is valid (i.e., the constraints are satisfiable) and retry if it cannot be satisfied. Again, please see Section 2.3 for more detail on the input generation process. \n\n**Question-3. Regarding the full program setting, ... Are all the programs composed with 3 lines of code? I believe it should not be the case. ... How does the proposed work verify the correctness of the logic of a full program?**\n\nThanks for asking the question, and we believe there may be some misunderstanding. Please note that this paper focuses on ***simple DS programs with only a single API call*** in all settings, including the ***Full program*** setting (see Line-109: “For the full program setting, we want the LLM to synthesize a complete DS program using a specific API from scratch”). We present the full prompt in Figure 2. While LLM-generated programs may contain arbitrary lines of code (e.g., sometimes there are more computations after calling the target API), we extract and focus on the first few code statements only (i.e., the input data generation statements followed by a single API invocation statement). We apologize for any confusion and will revise our presentation for clarity.\n\n***Rationale of studying a single API***: First, isolating the evaluation to individual APIs or individual API parameters makes it easier to analyze the result. Such a fine-grained setting facilitates a detailed examination of the LLMs’ limitations with respect to various types of numerical constraint. Second, given that LLMs already struggle with constraints within a single API, we believe that expanding the benchmark to multiple APIs would likely yield accuracy too low and difficult to interpret meaningfully. We appreciate the reviewer's comments and will include a discussion of this future work in our revised manuscript.\n\n***Extend to multiple APIs***: Please note that our method can easily be extended to more complex programs, which is essentially a computation graph consisting of multiple operators and their connections. Since we already symbolically model each individual operator (including input, output, constraints, type transfers), we can also combine these operators and symbolically generate and validate a full computation graph. To achieve this, we can reuse and modify the generation and validation framework provided by NNSmith [32], a popular tool which generates diverse DL computation graphs via formal constraint solving. For more details, please refer to the NNSmith paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "izth26gkDJ", "reviewer": "Reviewer_xaud", "summary": "This paper investigates the ability of large language models (LLMs) to implicitly learn and apply numeric parameter constraints in data science (DS) APIs, focusing on PyTorch and NumPy libraries. The authors conduct a comprehensive study across 28 representative APIs, evaluating LLMs in three settings: full program generation, all parameter prediction, and individual parameter prediction. They introduce DSEVAL, a benchmark containing 19,600 problems across 12 APIs with varying difficulty levels. The study evaluates both open-source and closed-source LLMs, including state-of-the-art models like GPT-4. Results show that while LLMs perform well on simple constraints and common patterns, their performance degrades significantly with increased difficulty or unusual inputs. The authors conclude that current LLMs, including GPT-4, struggle with complex arithmetic constraints and often rely on memorization of common patterns rather than true understanding of the underlying constraints. This research highlights the limitations of LLMs in handling numeric API constraints and provides a benchmark for future improvements in this area.", "questions": "1. Why did you limit your study to PyTorch and NumPy? Wouldn't including more libraries make your findings more generalizable?\n\n2. Your study lacks a human baseline. How can we interpret the LLM performance without knowing how humans perform on similar tasks?\n\n3. You don't provide error bars or statistical significance tests. How confident are you in the reliability of your results?\n\n4. Your paper doesn't explore fine-tuning or more advanced prompting techniques. Couldn't these potentially improve LLM performance significantly?\n\n5. The paper lacks an in-depth error analysis. Wouldn't categorizing common mistakes provide more insights into LLM limitations?", "limitations": "Lack of human baseline: The study does not provide a comparison to human performance on similar tasks, making it challenging to contextualize the LLM performance.\n\nAbsence of error analysis: The study lacks a detailed categorization and analysis of common error patterns, which could provide deeper insights into specific LLM limitations. \n\nNo error bars or statistical significance tests: The paper does not include measures of statistical uncertainty, which could strengthen the validity of the findings.\n\nLimited discussion on potential solutions: While the paper identifies limitations in LLM performance, it offers few concrete suggestions for improving LLM capabilities in handling numeric constraints.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: First systematic study of LLMs' ability to handle numeric constraints in DS APIs. Propose a novel benchmark (DSEVAL) for evaluating this specific capability. Challenges previously untested assumptions about LLM capabilities\n\nQuality: Comprehensive evaluation across 28 diverse APIs. Rigorous methodology using SMT solvers for validation. Evaluation of both open-source and closed-source models, including state-of-the-art LLMs\n\nClarity:. Well-structured presentation with clear explanations of complex concepts. Effective use of examples, tables, and figures to illustrate key points. Accessible to readers without deep expertise in LLMs or DS APIs", "weaknesses": "Absence of error bars or statistical significance tests: Lacks quantification of result certainty. Suggestion: Include error bars or conduct statistical tests to strengthen the validity of findings\n\nLimited exploration of prompting techniques: Minimal investigation of advanced prompting methods. Suggestion: Experiment with more diverse prompting strategies to potentially improve LLM performance. \n\nLack of human baseline: No comparison to human performance on similar tasks. Suggestion: Include a human baseline to contextualize LLM performance \n\nLimited discussion on potential solutions: Doesn't offer many concrete suggestions for improving LLMs in this domain. Suggestion: Propose and discuss potential approaches to enhance LLM capabilities in handling numeric constraints", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: Why did you limit your study to PyTorch and NumPy?**\n\nWe chose PyTorch and NumPy due to their wide adoption in the data science community: PyTorch is used by over 480k open-source GitHub projects [a] and NumPy is installed more than 300 million times monthly [b]. Focusing on these two allowed for a more in-depth analysis within our budget. You are absolutely right that including more libraries will make our findings more generalizable, and we plan to include more DS libraries (e.g., Pandas, Matplotlib, Scikit-learn, SciPy, and Tensorflow) in future work.\n\n**Question-2. Your study lacks a human baseline. How can we interpret the LLM performance without knowing how humans perform on similar tasks?**\n\nPlease note that most code benchmarking work (e.g., DS-1000, MBPP, HumanEval) does not provide a human baseline. This is because it is expensive to conduct user studies on coding problems, especially for DS programs where domain expertise is required. Additionally, people with different experiences can have vastly different performances. Meanwhile, unlike prior benchmarks where the natural language problem description can be unclear and thus imposes an upper bound on LLM performance [c], our benchmark is unambiguous, and we could expect ~100% accuracy from sufficiently strong LLMs or expert developers.\n\nIn this work, we interpret the LLM performance by comparing different LLMs and different problem complexities. As nicely summarized by Reviewer-bNWw:\n\n> (1) Their main finding is that although it is widely assumed that current LLMs can solve arithmetic constraints, the performance of even state-of-the-art LLMs like GPT-4-Turbo drops drastically when the complexity of the constraints increases, and accuracy sometimes even drops to 0.\n\n> (2) Another interesting finding is that there is a huge gap between open source models and GPT-4-Turbo, which is not captured by benchmarks like HumanEval.\n\n**Question-3: You don't provide error bars or statistical significance tests. How confident are you in the reliability of your results?**\n\nDue to the expensive costs, we use greedy decoding in all our experiments except for the full program setting (Section 4.1). Therefore, in all other settings, the performance is deterministic.\n\nFor the full program setting, we vary the temperature from 0.2 to 1 and sample 200 DS programs across all 28 APIs. As a result, performing repeated experiments to compute error bars will be extremely costly especially for commercial models like GPT-4-Turbo.\n\nFurthermore, please note that in all settings of our benchmark, each problem is independently sampled from an identical distribution of its difficulty level. As such, the pass/fail outcome follows a Bernoulli distribution, and we draw N=200 samples for each task distribution. Therefore, using the normal approximation: $p\\approx\n\\hat{p}\\pm\\frac{z_{\\alpha}}{\\sqrt{N}}\\sqrt{\\hat{p}(1-\\hat{p})}$, where $\\hat{p}$ is the average accuracy (proportion of successes), $z_{\\alpha}$ is the $1-\\frac{\\alpha}{2}$ quantile of a standard normal distribution corresponding to the target error rate $\\alpha$. For a 95% confidence level, $z_{.05}=1.96$ and $p$ (i.e., accuracy in our results) can be estimated by $p \\approx \\hat{p} \\pm 0.0049$. Therefore, we believe our results are statistically meaningful.\n\n**Question-4: Your paper doesn't explore fine-tuning or more advanced prompting techniques. Couldn't these potentially improve LLM performance significantly?**\n\nFirst, kindly note that we study chain-of-thought (CoT) prompting in Section 4.3. Our results on the difficult APIs show that while CoT prompting can indeed boost performance (especially for SOTA LLMs like GPT-4-Turbo), it still struggles to solve more complex arithmetic constraints as difficulty increases.\n\nSecond, following reviewer YxwD’s suggestion, we evaluate the ReACT prompting strategy in Figure 12 (see attached PDF in global response). We observe that while ReAct can perform better than CoT, it still fails to solve more complex arithmetic constraints. We also follow reviewer WbTY’s suggestion and include API documentation in prompts (Figure 14). We found that adding documentation cannot always achieve better performance on our tasks. \n\nFurthermore, we totally agree that fine-tuning may significantly improve the accuracy of LLMs. However the focus of the paper is to evaluate the **implicit** ability for LLMs to model parameter constraints from pre-training. We hope to evaluate fine-tuning as future work.\n\n**Question-5: Wouldn't categorizing common mistakes provide more insights into LLM limitations?**\n\nThanks for this suggestion! In our paper, we mainly focus our in-depth analysis based on the constraint types (e.g., equality, inequality, arithmetic, and set-related). Regarding categorizing common mistakes made by LLMs, we did have interesting findings, such as:\n\n- **LLMs struggle with uncommon input tensors**: We found that across many APIs and constraints, LLMs struggle when provided with uncommon input tensor ranks (i.e., rank > 4) or uncommon shapes (e.g., `x = torch.rand(9, 30, 23, 4)`). The reason is that LLMs are mostly trained with data that contains very common shapes or ranks. As such, LLMs can easily make mistakes on uncommon inputs.\n\n- **LLMs tend to predict common parameter values blindly**:  We also observe that LLMs tend to generate common parameter values (e.g., 0, 1, powers of 2) which often turn out to be incorrect. This is again because LLMs are trained with pre-training code that frequently contains such parameter patterns and thus are likely to predict them even given a different input context.\n\nThanks again for all the great suggestions and we will work towards adding them in the next version of the paper!\n\n**References**\n\n[a] https://github.com/pytorch/pytorch/network/dependents \n\n[b] https://pypi.org/project/numpy/ \n\n[c] L2ceval: Evaluating language-to-code generation capabilities of large language models. https://arxiv.org/abs/2309.17446 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rg6gbmD5pn", "reviewer": "Reviewer_bNWw", "summary": "The authors systematically investigate how well current LLMs learn numeric\nparameter constraints of functions and deep learning operators in the Numpy and\nPyTorch libraries. Their main finding is that although it is widely assumed that\ncurrent LLMs can solve arithmetic constraints, the performance of even\nstate-of-the-art LLMs like GPT-4-Turbo drops drastically when the complexity of\nthe constraints increases, and accuracy sometimes even drops to 0.\n\nAnother interesting finding is that there is a huge gap between open source\nmodels and GPT-4-Turbo, which is not captured by benchmarks like HumanEval. The\nauthors introduce a public benchmark called DSeval based on these findings which\ndemonstrates this gap and allows future research to measure and narrow it.", "questions": "In Figure 5, why does the accuracy drop much more significantly for Linear than\nfor the other functions/constructors? Is this only true for DeepSeekCoder-33B?", "limitations": "There is no section on limitations but I don't believe that one would be\nnecessary as the whole work is about the limitations of LLMs.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a first, thorough and systematic study of these constraints,\nwhich shows that in many cases LLMs perform worse than what is currently widely\nbelieved. The paper also illuminates the strong and weak points of these LLMs,\nand introduces a new benchmark dataset which shows a huge gap between\nopen-source LLMs and GPT-4-Turbo, even when they seem to perform similarly on\nother benchmarks like HumanEval.", "weaknesses": "Some of the terminology is not aligned with the widely used meaning of the\nconcepts, which makes the paper harder to read. Most significantly, the paper\ntalks about APIs like sum, max, or reshape. These are usually considered\nfunctions and/or deep learning operators , and not APIs. APIs are usually\nconsidered to be larger, like the API of a library such as PyTorch, or the API\nof OpenAI. Also, PyTorch is usually thought of as a deep learning library, not a\ndata science library.\n\n\n\nBased on this I would probably not call the benchmark DSeval. Also, the\nparameter constraints are probably also a salient feature of the benchmark and\nthe work, so I would call it something along the lines of DLParamEval, or\nDLParamConstraintEval. Naming the benchmark is of course the authors's choice,\nso this is just a suggestion and won't influence my score.\n\nThe paper doesn't distinguish between GPT-4 and GPT-4-Turbo. The authors usually\nwrite about GPT-4 while GPT-4-Turbo is used in the evaluations.\n\nSome smaller remarks:\n- I believe that the last three lines of the input are actually part of the\n  output in Figure 2a.\n- The explanation of subfigures could be added to the caption of Figure 2\n- full program, all parameters, and individual parameters could be explained\n  briefly also near line 66", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: In Figure 5, why does the accuracy drop much more significantly for Linear than for the other functions/constructors? Is this only true for DeepSeekCoder-33B?**\n\nThanks for bringing it up! This is an interesting result. For `torch.nn.Linear(in_features, out_features)`, the only constraint is that the `in_features` should match the last dimension of the input tensor. However, the DeepSeekCoder-33B model tends to copy the wrong dimension of the input tensor, likely because it hasn't seen lots of high-rank tensors (rank > 4) in the pre-training data. \n\nWe further evaluate this phenomenon across different LLMs. Figure 13 (see attached PDF in global response) shows the results for both the full API parameter and single API parameter setting for `torch.nn.Linear` as we increase the difficulty (rank of the input data) across 8 LLMs. Similar to DeepSeekCoder-33B, the performance of other LLMs also drops significantly when rank reaches 4. Afterwards, the performance stabilizes for higher difficulties (i.e., rank > 4) especially for open-source LLMs. This is true for both the full API parameter and single API parameter setting. Surprisingly, we found that even the state-of-the-art GPT-4-Turbo drops in performance when the rank reaches 4. However, we see that GPT-4-Turbo was able to improve its performance in higher difficulties (i.e., rank > 6). After looking at the results, we found that for lower ranks, GPT-4-Turbo tends to use other APIs as “short-cuts” and forgo the analysis on `torch.nn.Linear` directly as shown in an example below:\n\n```python\nimport torch\nx = torch.randn(1, 8, 10, 10)\nm = torch.nn.Linear(8*10*10, 3)\ny = m(x.view(1, -1))\n```\n\nIn the above example, instead of using only the last dimension (10), GPT-4-Turbo multiplies all the previous dimensions together and performs a flattening operation (`x.view(1, -1)`). This does not reflect the original meaning of the code and as such is evaluated as incorrect. \n\nThanks for this great question again! We will add this experiment results in the next revision of the paper.\n\n**Question-2: Most significantly, the paper talks about APIs like sum, max, or reshape. These are usually considered functions and/or deep learning operators , and not APIs. APIs are usually considered to be larger, like the API of a library such as PyTorch, or the API of OpenAI. Also, PyTorch is usually thought of as a deep learning library, not a data science library.**\n\nThanks for the suggestions! First, please note that deep learning/machine learning are generally considered as subfields of data science [48]. Libraries like PyTorch contain many data process, manipulation, and transformation APIs used for data science operations. Furthermore, the widely-used data science benchmark -- DS-1000 [29], contains both TensorFlow and PyTorch problems. Additionally, please note that we are not only targeting PyTorch, we also include NumPy in our study and we plan to generalize this study to more data science libraries in the future. For libraries like PyTorch and NumPy, indeed “smaller” APIs like `sum`, `max`, or `reshape` are called operators as well, and there are “larger” APIs like `SoftMax`, `BatchNorm`, and `Conv2d` which combine multiple operators; Meanwhile, they can be all considered APIs as well (please see https://pytorch.org/docs/stable/index.html - Python API). We use the term “API” instead of “operator”, because it is more general and accurately describes our target: the *publicly* exposed operators or functions for which we expect LLMs to generate correct parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mQljtYLZc2", "reviewer": "Reviewer_YxwD", "summary": "This paper investigates a problem, precisely the title, can LLMs implicitly learn numeric parameter constraints in data science APIs? To investigate this problem, this paper constructs a benchmark, DSEVAL, which contains a series of API calling based code completion tasks. Then, this paper evaluates several LLMs to derive the conclusion: LLMs can implicitly learn the constraint pattern during training, but lack genuine comprehension of the underlying constraints.", "questions": "See weakness.", "limitations": "I donot find any discussion about limitation in this paper.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The investigated problem is interesting.\n- The technique is sound.\n- The writing is good.", "weaknesses": "- I am not sure whether this paper should be submitted to Dataset & Benchmark track, since this paper focuses on evaluating LLMs’ underlying capabilities for satisfying parameter constraints when calling data science APIs, rather than proposing any novel algorithmic advances, analysis, or applications. \n\n- This paper should provide some naïve solutions as the starting point for this problem. However, this paper just provides a comprehensive evaluation on existing LLMs, without proposing any possible solution for this problem.\n\n- I think one natural question is that whether LLMs can solve this problem by using ReAct [1] prompt strategy, i.e., each time LLMs call a data science API, LLMs are prompted to first output a thought about the constraint for this API, and then generate the parameter. I am curious about the results about this setting.\n\n[1] ReAct: Synergizing reasoning and acting in language models, ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: I am not sure whether this paper should be submitted to Dataset & Benchmark track, since this paper focuses on evaluating LLMs’ underlying capabilities for satisfying parameter constraints when calling data science APIs, rather than proposing any novel algorithmic advances, analysis, or applications.**\n\nWe thank the reviewer for this suggestion. We would like to point out that although there is a separate Dataset & Benchmark track, the main conference still accepts a primary area for evaluation: “Evaluation (methodology, meta studies, replicability and validity)”. Given the wide usage of DS libraries/APIs, automatically synthesizing valid DS programs with LLMs has been a critical research area in order to improve DS development efficiency [29] or improve the reliability of ML systems [16]. Therefore, it is critical to rigorously evaluate whether LLMs can implicitly learn the numeric constraints in data science APIs -- an untested assumption of LLMs’ capabilities. Our paper aims to propose a novel fine-grained analysis for DS API generation by isolating each API call and API parameter generation, as opposed to the previous coarse-grained benchmarks such as DS-1000. We believe our proposed evaluation methodology is novel and would be highly beneficial for the research community to accurately evaluate and analyze the LLMs’ ability in the important domain of DS code generation. \n\nAlso, we do not aim to propose novel code generation algorithms in this work, but rather to point out interesting results and inspire future work. Some prior work [a,b,c,d] with similar contributions (i.e., discovering important findings/limitations for LLMs) were also published at NeurIPS main track.\n\n**Question-2: This paper should provide some naïve solutions as the starting point for this problem. However, this paper just provides a comprehensive evaluation on existing LLMs, without proposing any possible solution for this problem.**\n\nWe understand that there are strategies at inference time such as ReAct which can boost the performance of LLMs. However, the goal of this paper is to investigate the ability of LLMs to ***implicitly*** model the parameter constraints using zero-shot inference, without adding ***explicit*** reasoning steps such as recalling the constraints with ReAct. Different models may perform differently depending on prompting techniques used, and the performance also highly depends on the wording and in-context demonstrations of the prompt. On the other hand, evaluating them under a naïve zero-shot auto-completion setting provides a fair assessment of the ability of the LLM to implicitly learn the numeric constraints.\n\nMeanwhile, please kindly note that we have applied Chain-of-Thought (CoT) prompting strategy to elicit thought steps at **the end of Section 4.3**. More specifically, we evaluate instruction-tuned LLMs using CoT prompting (“Please think step by step”) on multiple particularly difficult DS APIs. We found that although CoT prompting does help improve performance (especially when using state-of-the-art GPT-4-Turbo), it still struggles for more complex arithmetic constraints such as the ones in `torch.nn.Fold` (less than 5% accuracy in more difficult settings).\n\n**Question-3: I think one natural question is that whether LLMs can solve this problem by using ReAct [1] prompt strategy.**\n\nThanks for the great suggestion! Although techniques that elicit explicit reasoning at inference time (like ReAct) is not the main focus of this paper, we agree that such an analysis is valuable and can provide additional insights to the goal of this paper.\n\nWe applied the ReAct prompt strategy on the difficult APIs/constraints studied using CoT in the paper. Our ReAct prompt setup follows the reviewer’s suggestion (i.e., asking the LLM to generate a thought first then the code output). We also provide the LLM with a single demonstration of the ReAct task. \n\nFigure 12 (see attached PDF in global response) compares the results of the instruction-following LLMs using CoT versus ReAct as well as their base variants:\n- We see that for `torch.nn.MaxPool2d`, ReAct prompting generally performs better than CoT especially in more difficult problem settings (e.g., at highest difficulty setting, GPT-4-ReAct: 89.5% GPT-4-CoT: 56.0%). This demonstrates the effectiveness of ReAct in generating thoughts that can help with the correct API parameter generation. \n- However, for `torch.nn.Conv2d`, ReAct performs similarly to CoT prompting. The reason is that the constraint used in Conv2d is much more complex, requiring factorization. As such, smaller open-source LLMs cannot perform well even with reasoning steps. On the other hand, state-of-the-art LLMs like GPT-4-Turbo show their powerful reasoning abilities by improving the performance over the base variant with both CoT and ReAct.\n- Although ReAct performs better than CoT for the easier difficulty settings in `torch.nn.Fold`, its performance quickly drops in higher difficulty settings (at best ~5% accuracy with the best GPT-4-Turbo). Overall, this experiment results demonstrate that even more advanced prompting methods such as ReAct still cannot effectively handle more complex constraints.\n\nWe thank the reviewer again for this suggestion and will work towards adding these results to the new revision of the paper.\n\n\n**References**\n\n[a] Testing the General Deductive Reasoning Capacity of Large Language Models Using OOD Examples. https://openreview.net/forum?id=MCVfX7HgPO NeurIPS 2023 (poster).\n\n[b] Are emergent abilities of large language models a mirage? https://openreview.net/forum?id=ITw9edRDlD NeurIPS 2023 (oral).\n\n[c] Statistical knowledge assessment for large language models. https://openreview.net/forum?id=pNtG6NAmx0 NeurIPS 2023 (poster).\n\n[d] Exploring length generalization in large language models. https://openreview.net/forum?id=zSkYVeX7bC4 NeurIPS 2022 (poster).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "on9fcAjXgS", "reviewer": "Reviewer_WbTY", "summary": "In data science library APIs, there is a numerical parameter constraint between input data and parameters. This paper presents an empirical study on whether large language models (LLMs) can implicitly learn numerical constraints in data science library APIs. The findings indicate that LLMs demonstrate the ability to memorize common patterns in the use of popular data science APIs, and their performance significantly declines when the rank or dimensions of the input data increase, suggesting a general lack of true understanding of the API's numerical constraints. Additionally, the paper reports several other findings. It also introduces a dataset named DSEVAL, designed to evaluate LLMs' capabilities in understanding numerical API constraints for some popular data science libraries.", "questions": "1. I believe that by incorporating API documentation into fine-tuning or including it within prompts, LLMs might better understand numerical parameter constraints, which could potentially address the issue you're exploring.\n\n2. API parameters can vary across different versions of a library. Could these variations impact the results of your study?\n\n3. Why do you choose to conduct the study with 28 APIs, whereas only 12 APIs are used in the DSEVAL dataset? What accounts for this difference?\n\n4. The specific LLM used during input creation is not mentioned. Could you clarify which LLM is employed?\n\n5. In Figure 5, why do different sets of APIs get affected as the input rank and dimensionality increase? Furthermore, how do rank and dimension affect the complexity of these APIs?", "limitations": "The authors have not dedicated a separate section to discussing the limitations; however, they consider the limitations to be associated with the scope of their study. I consider this sufficient.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper explores a new and interesting problem regarding whether LLMs comprehend the numerical parameter constraints between input data and parameters in Data Science library APIs during the code generation process.\n\n2. The study is thoroughly conducted, encompassing three scenarios (Full program, all parameters, Individual parameter) and various series, different sizes of LLMs such as GPT-4-Turbo, DeepSeek, CodeLlama, StarCoder, and CodeQwen 1.5.\n\n3. It contributes a new dataset designed to assess LLMs’ capabilities in understanding the numerical API constraints for popular Data Science libraries like PyTorch and NumPy, with both the dataset and source code made publicly available.", "weaknesses": "1. While the problem of numerical parameter constraints in the data science APIs discussed in this paper exists, it may not be particularly significant and appears to be easily resolvable. This raises concerns about the potential for further research.\n2. Some of the experimental details of the paper are unclear and require additional elaboration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question-1: I believe that by incorporating API documentation into fine-tuning or including it within prompts, LLMs might better understand numerical parameter constraints, which could potentially address the issue you're exploring.**\n\nThanks for this suggestion. We conducted additional experiments using the documentation-augmented setting across the 3 difficult API/constraints used in our CoT experiments. We provide the raw documentation of each API (obtained from the source code docstring) in the prompt and apply both base and instruction-following LLMs. \n\nIn Figure 14 (see attached PDF in global response), we compare the performance with and without documentation. We found that there are cases where documentation can improve performance. For example, in the most difficult setting of `torch.nn.Conv2d`, adding documentation is able to improve performance of CodeLlama-34B-Instruct from 20% to 45% accuracy (Figure 14 (d)). However, there are also similar cases where adding documentation decreases performance. For example the GPT-4-Turbo-Instruct performance falls from 57.5% to 22.5% in the most difficult setting of `torch.nn.MaxPool2d` (Figure 14 (b)).\n\nSince we provide the raw documentation text without further processing, the success rate of adding documentation can vary depending on the specific model as well as the quality of the documentation. As such, this demonstrates that naively adding API documentation cannot always achieve better performance on our tasks. \n\nAgain, we want to emphasize that we investigate the LLMs’ ability to implicitly reason about such constraints, given that LLMs have been trained on massive parameter combinations. Our findings in the paper shows that current LLMs cannot implicitly reason with more complex API constraints or unusual inputs. This can inspire many follow-up works on improving LLM performance, including the reviewer’s suggestion of incorporating API documentation or performing further fine-tuning.\n\n**Question-2: API parameters can vary across different versions of a library. Could these variations impact the results of your study?**\n\nGreat question! Indeed, some APIs are not stable and their parameters and constraints could vary across versions. However, in this study we focus on the core functional APIs which should remain stable across library versions. We manually examined all 28 APIs across a 2 year period of major releases and found that only one API (`torch.squeeze`) received a small update to its parameter (removing the optional `out` parameter used to store the output tensor) while its numerical parameter constraints stayed the same. Therefore, we believe the API parameter variations across library versions have minimal impact on our study.\n\n**Question-3: Why do you choose to conduct the study with 28 APIs, whereas only 12 APIs are used in the DSEVAL dataset? What accounts for this difference?**\n\nOur 28 APIs are obtained by first selecting the 22 core APIs that have numeric parameter constraints in NNSmith, and adding additional 6 core APIs by examining the API list. Our chosen APIs are the core API commonly used by users. While there are a large number of APIs in DS libraries, the commonly used ones (e.g., Conv2d) are not that many. For example, the widely-used benchmark DS-1000 [29] contains 1000 DS problems collected from StackOverflow problems (including 68 PyTorch problems and 220 NumPy problems), reflecting realistic use cases; meanwhile, it only covers 16 PyTorch APIs and 59 NumPy APIs (after excluding data construction APIs like “np.ones”).\n\nFurthermore, some APIs have similar constraints or same constraint types. For example, `numpy.max` and `numpy.min` have the same constraints, whereas `torch.nn.Conv2d` and `torch.nn.Conv1d` have very similar constraints. Therefore, in our DSeval benchmark, we select 12 representative APIs to keep the experiments at an affordable scale, while still covering all major constraint categories in Table 1. We will clarify this further in the next revision of the paper. \n\n**Question-4: The specific LLM used during input creation is not mentioned. Could you clarify which LLM is employed?**\n\nSorry for the confusion. We do not use LLMs during input creation. Instead, we have a template-based prompt and we generate random input values using built-in random number generators. We then use the Z3 constraint solver to filter out the invalid inputs and ensure that every input has at least one valid answer. For more details, please refer to Section 2.3.\n\n**Question-5: In Figure 5, why do different sets of APIs get affected as the input rank and dimensionality increase? Furthermore, how do rank and dimension affect the complexity of these APIs?**\n\nInput rank or dimensionality can affect different APIs depending on the type of numeric constraint of each API. Table 1 shows the categorization of the different types of constraints. For example, an API like `torch.nn.SoftMax` that has a constraint of $-rank \\leq dim < rank$ will have its difficulty influenced by the actual rank of the input tensor. On the other hand, an API like `torch.nn.Conv2d` has a constraint of $in\\underline{ }channels \\% groups = 0$, which depends on the actual dimension value of the input (i.e., `in_channels`). As the dimension value of `in_channels` increases, it will be more difficult to select the `groups` parameter that can divide it evenly. Therefore, we increase the difficulty of different APIs based on whether the constraint depends on the rank, dimension, or both.\n\nThanks for the question and we will clarify this further in the next revision of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their insightful comments and suggestions to improve the paper! We address the main questions and concerns in this rebuttal. Furthermore, we also plan to revise the paper accordingly to address all other minor suggestions and comments.

We have also attached a PDF corresponding to the new experimental results requested by the reviewer, please kindly see the attached PDF for the detailed result figures. 

Please kindly let us know if there is any misunderstanding of the questions, and we are very happy to provide further updates or clarifications during the reviewer-author discussion period.

Meta-review: The authors create a new data set called DS_EVAL, which tests LLMs ability to reason about numeric constraints when doing code generation. The data set consists of 19,600 programs, which call 28 different APIs from pytorch and numpy, and each call is ranked in terms of difficulty.  Constraints refer to things like specifying valid dimension numbers given the shape of arrays, specifying valid argument to reshape, specifying valid permutations when doing transpose, etc.   An SMT solver is used to validate that the LLM output satisfies the constraints.

The authors compare a number of LLMs, and find that LLMs are generally able to handle easy cases using common shapes that occur frequently in training data, but cannot reason about constraints in harder and less common cases.  

Reviewer responses to this paper were mixed.  Three reviewers recommend "weak accept" (6), one recommends "weak reject" (4), and the last recommends "reject" (3).   Praise for the paper was quite uniform: 

"Originality: First systematic study of LLMs' ability to handle numeric constraints" (reviewer xaud)
"first, thorough and systematic study of these constraints"  (reviewer bNWw)
"The paper explores a new and interesting problem" and "The study is thoroughly conducted"  (reviewer WbTY)

Reviewer (qmKc) [weak reject] is concerned that "the paper lacks important details." -- namely how the constraints were determined and formalized, and how the example programs are constructed.  They also complain that "the experiment is limited in scale in terms of the number of studied APIs."  IMHO, these are valid criticisms, but are not strong enough to reject the paper.  

Reviewer (YxwD) [reject] complains that the "paper just provides a comprehensive evaluation on existing LLMs, without proposing any possible solution for this problem" and suggests that the paper should have been submitted to the datasets and benchmarks track instead.

Overall, I personally feel that this is a valuable paper, which is worthy of publication.  The paper examines a particular aspect of code generation that has not been previously explored in the literature, and is likely to be increasingly important as LLMs are applied to more complex coding tasks.  It provides a comprehensive evaluation of existing SOTA models.  Moreover, because this is a benchmark, it can be used by other researchers to test and improve LLM coding capabilities going forward.  

However, I do think that reviewer (YxwD) has a point.  This is clearly a "datasets and benchmarks" paper, and perhaps should have been submitted to that track.
Decision: Accept (poster)