Can LLMs Implicitly Learn Numeric Parameter Constraints in Data Science APIs?

Authors: yinlin  deng chunqiu
steven  xia
zhezhen  cao
meiziniu  li
lingming  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 18:15:23

Abstract: Data science (DS) programs, typically built on popular DS libraries (such as PyTorch and NumPy) with thousands of APIs, serve as the cornerstone for various mission-critical domains such as financial systems, autonomous driving software, and coding assistants. Recently, large language models (LLMs) have been widely applied to generate DS programs across diverse scenarios, such as assisting users for DS programming or detecting critical vulnerabilities in DS frameworks. Such applications have all operated under the assumption, that LLMs can implicitly model the numerical parameter constraints in DS library APIs and produce valid code. However, this assumption has not been rigorously studied in the literature. In this paper, we empirically investigate the proficiency of LLMs to handle these implicit numerical constraints when generating DS programs. We studied 28 widely used APIs from PyTorch and NumPy, and scrutinized the LLMs’ generation performance in different levels of granularity: full programs, all parameters, and individual parameters of a single API. We evaluated both state-of-the-art open-source and closed-source models. The results show that LLMs are great at generating simple DS programs, particularly those that follow common patterns seen in training data. However, as we increase the difficulty by providing more complex/unusual inputs, the performance of LLMs drops significantly. We also observe that GPT-4-Turbo can sustain much higher performance overall, but still cannot handle arithmetic API constraints well. In summary, while LLMs exhibit the ability to memorize common patterns of popular DS API usage through massive training, they overall lack genuine comprehension of the underlying numerical constraints.

Main paper: 
 introduction: 
 Data science (DS) is an emerging and important area that combines classic fields like statistics, databases, data mining, and machine learning (ML) to gain insights via complex operations on the abundance of available data [49] . DS libraries (such as PyTorch [41] and NumPy [38] ) contain thousands of APIs used by developers and data scientists to process/analyse data. These DS APIs serve as the fundamental building blocks for almost all important ML/DS pipelines, and have penetrated into almost every corner of modern society, including financial systems [18, 4] , autonomous driving software [9, 27, 46] , coding assistants [45, 37] , etc. Due to their high importance and wide usage, automatically synthesizing valid DS programs has been a critical research area [29, 21, 47] .One key challenge of DS code generation is to satisfy the complex constraints within each DS library API. DS library APIs perform transformations (e.g., matrix multiplication) on inputs (i.e., arrays or array-like objects) with numeric constraints on API parameters and inputs. Figure 1 shows an example of a typical DS program where the DS library (i.e., PyTorch) is first imported, followed by creating some input_data, and then performing the data manipulation operation on the input_data using a DS API (torch.nn.Conv2d). The parameters of the API (e.g., kernel_size, groups) must satisfy the corresponding constraints between API parameters and the properties of the input_data. We refer to API constraints as the set of relationships between properties of input_data and API parameters that, if and only if when satisfied, leads to a valid DS API invocation. As seen in Figure 1 , not only are there constraints between the properties of the input_data and API parameters (e.g., kernel_size ≤ H + 2*padding), but there are also constraints within API parameters (e.g., out_channel % groups = 0). These constraints are defined by developers according to the functionality of each DS API, and are usually specified in natural language within the API documentation. Such complex constraints are critical for DS applications, and DS users or even DS experts may unintentionally violate such constraints [29, 26] . Large language models (LLMs) have achieved tremendous success in processing code [10, 2] . Due to their powerful code understanding and generation ability, LLMs have been applied to various coding tasks [34] , such as code completion [20, 6] , program repair [19, 54] , and test generation [16, 17, 48] . For DS libraries, LLMs have been applied to solve practical user queries on StackOverflow [29] and even generate test programs to detect bugs in modern ML frameworks [16] . Prior work assumes LLMs, through massive training, can already implicitly model constraints in DS APIs by learning from numerous correct DS API uses [47, 21, 16] . However, this assumption has not been systematically verified. Furthermore, popular DS-specific benchmarks like DS-1000 [29] do not specially test the LLM's ability to satisfy implicit constraints and instead focus on how to apply DS APIs to solve data analysis tasks. These gaps in prior research raise a critical question: Can LLMs implicitly learn the numeric constraints in data science APIs?Our work. To answer the question, we conduct a rigorous study on the performance of LLMs in generating valid DS programs satisfying diverse numerical API constraints. We collected a set of 28 representative DS library APIs across two widely-used Python DS libraries (PyTorch and NumPy), each with their unique constraints/setup. Additionally, we categorize each API's constraints into different categories (e.g., equality and arithmetic) and perform in-depth experiments on each constraint type. To support our analysis, we systematically created 3 generation settings: full program, all parameters, and individual parameters, designed to test the LLMs under different evaluation scenarios. Additionally, we vary the difficulty level by adjusting the inputs to explore LLM behaviours when asked to solve more complex API constraints or given more unnatural inputs.Interestingly, contrary to the popular assumption in prior work, while LLMs can easily satisfy constraints when the inputs are simple, we observe that the performance drops drastically as we increase the difficulty or provide more unusual inputs. We found that LLMs tend to generate simple and common inputs seen during training, highlighting that LLMs are often memorizing patterns instead of truly understanding the actual DS API constraints. For example, for the widely used Conv2d API shown in Figure 1 , when max(in_channels,out_channels) is set to [128, 256) , even GPT-4-Turbo [1] can only predict the correct value of groups ∼24% of the time, while the other models are below 14%. Furthermore, based on our experimental findings, we constructed DSEVAL, the first benchmark for systematically evaluating LLMs' capabilities in understanding the important numerical API constraints for popular DS libraries. DSEVAL contains 19,600 different problems across 12 representative APIs to extensively compare and contrast the performance of different LLMs. DSEVAL supports lightweight and fast evaluation by extracting LLM generated parameters and quickly verifying the correctness using state-of-the-art SMT solvers (such as Z3 [13] ) to avoid time-consuming execution-based evaluations. Our evaluation on eight state-of-the-art open-source and closed-source models shows that while all studied models struggle with more difficult problems, GPT-4-Turbo consistently achieves the highest accuracy across all difficulty levels. For example, GPT-4-Turbo achieves an average accuracy of 57.5% for hard constraints of PyTorch APIs, while the best open-source model can only achieve 39.2%, demonstrating the huge gap between large proprietary models and other open-source LLMs. Our design of DSEVAL is general and can be easily extended to additional libraries and APIs for the DS domain and even beyond. We specifically target the core APIs commonly used by users that perform operations on the input_data. Additionally, we also only consider numeric API constraints: constraints with only numeric parameters such as integers. We ignore any other types of parameters (e.g., string) since they do not affect the validity of numeric constraints. As such, any non-numeric parameters produced by the model will be discarded during constraint validation.Table 1 shows the types of constraints we considered in the study with the corresponding categories. We group the constraints into i) Equality: constraints where the values have to match exactly. We see that equality constraints are related to selecting or generating the right shape in the input_data. ii) Inequality: constraints where values have to be greater or less than. Inequality constraints include mainly rank related operations to stay within the valid rank range. iii) Arithmetic: constraints involving arithmetic operations such as division, modulus or products. There are also more complex API constraints that includes combination of many arithmetic operations. iv) Set-related: constraints where the satisfaction criteria depend on different set-based properties. For example, there are constraints that require parameters to be unique or complete with respect to input_shapes. Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In this paper, we perform a comprehensive evaluation on the proficiency of LLMs to handle numeric parameter constraints in data science libraries. The main claims made are accurately reflecting the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
evaluation settings: 
 Next, we describe our settings to evaluate the performance of LLM on handling the numeric constraints. In total, we have 3 settings: i) full program, ii) all parameters, and iii) individual parameter.Full program. For the full program setting, we want the LLM to synthesize a complete DS program using a specific API from scratch. To do this, we provide a 3-step instruction and the basic starting code of importing the DS library. Figure 2a shows an example of the full program input for the API torch.sum as well as an example LLM output. We note that in this setting, the LLM has full freedom to generate any type or size for the input_data. As such, the LLM may choose very simple input_data and API parameter values that can easily satisfy the constraint.All parameters. In the all parameters setting, we directly provide the input_data for the API. Figure 2b also shows an example of the input for the API torch.nn.MaxPool2d where the LLM just needs to output the API parameters. This setting evaluates if/how LLMs can accurately solve the constraints as we vary the input_data with more difficult or uncommon cases. Still the LLM has full freedom to pick the full combination of parameters to satisfy the required constraint.Individual parameter (main setting). To perform a finer-grained evaluation, we introduce the individual parameter setting where we ask the LLM to generate a single parameter of the API. Figure 2c additionally demonstrates an example for np.reshape where we only allow the LLM to fill in a single parameter value of newshape. Furthermore, we can also add an additional constraint by directly providing the first value of newshape (2 in the example). This makes the problem even more challenging where instead of being able to simply copy the input_shapes, the LLM now has to reason with the partial shape given and compute the final correct shape to satisfy the constraint.Compared to the prior two settings, the choices here are much limited. This makes the task harder to fully evaluate how LLMs solve complex API constraints, and serves as our main setting.Creation. To produce the inputs for each of the 3 settings, we use a fixed set of templates for each API. For the full program setting, we produce one input per API, changing only the API name in the input instruction. For the all parameters setting, we vary the input_data given to the API. In particular, we focus on two properties of the input_data: 1) rank of the input_data and 2) each dimension value. We create randomized inputs and increase the difficulty by either increasing the rank or the dimension values to measure the LLM performance. Note that input rank or dimensionality can affect different APIs depending on the specific numeric constraints (Table 1 ). For example, an API like torch.nn.SoftMax that has a constraint of -rank ≤ dim < rank will have its difficulty influenced by the actual rank of the input tensor. On the other hand, an API like torch.nn.Conv2d has a constraint of in_channels % groups = 0, which depends on the actual dimension value of the input (i.e., in_channels). As the dimension value of in_channels increases, it will be more difficult to select the groups parameter that can divide it evenly. Therefore, we increase the difficulty of different APIs based on whether the constraint depends on the rank, dimension, or both. Similarly, for the individual parameter setting, we also randomize the input_data based on the previous two properties. Additionally, we pick the parameters with interesting constraints for the LLM to predict in order to be representative and cover the major constraint types. Furthermore, since we only ask the LLM to produce a single parameter value, we also vary the other parameter values in the API to add additional constraints (details discussed in Section 4.3). To ensure the input is valid, we leverage satisfiability modulo theory (SMT) solvers as shown in Figure 3a . SMT solvers, such as Z3 [13] , are tools which can be used to solve an SMT problem of determining whether a mathematical or first-order logic formula is satisfiable [5] . We first encode the API constraints into an SMT formula.We then randomly generate concrete values for the input_shapes and leave the other parameters that we want the LLM to generate as symbolic variables. Next, we use an SMT solver to check if the constraints are satisfiable (i.e., there exists a set of values for each symbolic variable that can satisfy the constraint). If it is satisfiable, the input we provide to the LLM is valid, otherwise we restart the process by randomly selecting the concrete values. In our study, we reuse the encoded API constraints provided by NNSmith [32] (a popular tool for testing ML libraries via formal constraint solving) and add additional ones when needed.Evaluation. To evaluate the validity of the DS programs generated by the LLMs, we first parse the output to extract the input_data and API parameters. We then check if the LLM predicted values are valid. This is also done via SMT solving as demonstrated in Figure 3b where we use the SMT formulas and, this time, check if all the concrete values generated are valid according to the constraints. Note that such light-weight constraint solving can support much faster validation than actually executing the generated DS programs, while still providing the same guarantee.3 Experimental SetupQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: In Section 3.1 we discuss the scope of our study as well as assumptions made, that is, we focus on numeric constraints only and assumes all other parameters are default and valid.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
subjects: 
 We construct a dataset with 28 representative APIs in total from two popular DS libraries: PyTorch (18) and NumPy (10) . For our API selection process, we begin by referencing prior work NNSmith [32] and examined all 73 core operators it supports. From these, we select 22 core APIs that have numeric parameter constraints and add additional 6 APIs to obtain the 28 APIs used in our study for both the full program prediction setting (Section 4.1) and the full API parameter prediction setting (Section 4.2). For a more detailed analysis, we select 12 APIs to cover the representative types of numeric constraint for examination in the single API parameter prediction setting (Section 4.3) and in our DSEVAL benchmark (Section 4.4). We use "representative" to mean representative with respect to the numeric parameter constraints in DS library APIs. Table 1 shows the categorization of the different types of numeric constraints that exist in DS libraries. Our selection criteria aim to select a list of APIs that have interesting numeric parameter constraints that can cover all the major constraint categories. A complete list of the 12 APIs and their corresponding constraints is provided in Table 3 in the Appendix.We focus on the 3 settings described previously to analyse the performance of LLMs. For the full program setting, we generate a single input prompt per each studied API and ask the LLMs to synthesize the complete DS program by varying the sampling temperature. For the all parameters setting, we have 14 difficulty settings, each with 200 different inputs per API, and use greedy decoding to obtain the LLM solutions. The difficulty setting is controlled by increasing the rank of input_data (from 2 to 8 in intervals of 1) with default dimension value as [1, 16] , and increasing the dimension value (i.e., [1, 4) , [4, 8) ,... , [128, 256] ) with default rank as 3, separately. Finally, in the single parameter setting, we select one parameter for each API for the LLM to generate. For any parameters irrelevant to the constraint, we use the default value if it is an optional parameter, and randomly choose from a reasonable value range if it is a required parameter (Appendix C). We adopt the same difficulty setup and greedy decoding strategy as the all parameter setting.Validity. To measure validity, we directly extract the LLM output predictions and evaluate according to the process described in Section 2.3. We define accuracy as the percentage of valid programs produced by the LLMs in each difficulty setting.Diversity. To measure diversity, we compute the unique valid rate: the percentage of unique valid programs generated via sampling. Note that we deduplicate by extracting the input shapes and numeric parameters, ignoring the irrelevant parameters and irrelevant code suffix.We evaluate 8 popular state-of-the-art LLMs, including both closed-source and open-source models (detailed list shown in Table 2 ). For both the full program and all parameter settings, we only present the results for DeepSeek Coder-33b [22] , state-of-the-art open-source model, due to the space limit (other models follow similar trends). For the individual parameter setting (the main setting), we focus on the DeepSeek Coder family models (33b, 6.7b, and 1.3b) as well as GPT-4-Turbo (2024-04-09), covering both state-of-the-art open-source and close-source models, as well as models with different sizes. Apart from the full program setting, where the LLM generates a complete program, we perform infilling using the studied LLMs' model-specific infilling format. To perform infilling using GPT-4-Turbo, we design a specialized prompt (see Appendix H). Unless otherwise stated, we use greedy decoding (i.e., temperature = 0) and temperature of 1 when sampling for diversity evaluation. To start with, we ask the LLM (DeepSeek Coder-33b) to predict the entire DS program from scratch given just simple instructions. Figure 4a shows the overall accuracy of the 18 APIs in PyTorch and 10 APIs in NumPy. We see that with low temperature the model has near perfect accuracy on almost all the APIs and as temperature slowly increases, the accuracy tends to drop (ending with around 0.5∼0.8 with temperature=1). Surprisingly, we found that for torch.nn.Fold, which contains the most complex constraint, the LLM failed to produce any valid DS programs. This demonstrates that LLMs may still struggle with satisfying the extremely difficult constraints even when given the full freedom of generating any input values. Furthermore, in Figure 4b , we plot the proportion of unique valid programs generated by the model as we vary temperature. Of course when sampling at low temperatures, many of the inputs will be repeated, leading to low number of unique programs in general. In particular, the input shapes are often from widely-used computer vision datasets like 3*224*224 from ImageNet [15] . This indicates the LLMs tend to memorize some common patterns from either documentation or user programs. However, we see that even though the unique valid rate increases with high temperatures to give more diverse and creative outputs, the percentage of unique valid programs can still be mostly below 50%. This demonstrates that while models are successful in generating a high percentage of valid programs, a lot of generated programs are repeated. Figure 5 shows the setting where we randomly provide an input_data and ask DeepSeek Coder-33b to complete the valid parameters of the API. We vary the difficulty by changing either the rank or the dimension value ranges of the input_data to produce more complex and unnatural inputs. We use greedy decoding (temperature 0) to generate one solution per problem, and compute the average valid rate across the randomly created problems to compute accuracy for each difficulty level. Compared to Section 4.1 where LLMs achieve near-perfect accuracy for almost all APIs with low temperature like 0.2, we observe that the accuracy quickly drops when simply randomizing the input shape, especially for APIs with more complex constraints. This indicates that the learned patterns cannot easily generalize to less common input shapes. We further performed an interesting case study on the PyTorch API Linear, and found this phenomenon holds true across different models (Appendix D). However, we see that the majority of APIs maintain high accuracy even as difficulty increases (others(x) in Figure 5 ). This is because these APIs have relatively easy constraints. For example, APIs like max or argmax only require predicting a single integer representing the dimension to operate on, and the LLMs learn to predict dim=1 or just rely on the default parameter values of the API which are always valid.some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
single api parameter prediction: 
 We now focus on the main finer-grained evaluation setting where we ask LLMs to predict a single parameter value and discuss the input setup, results, and findings for each API separately. Here, we only discuss representative API constraints from each category and full results are in Appendix F. (h) Conv2d on IT models (i) PyTorch Fold on IT models Figure 6 : Single API parameter result. Solid lines (except Fig. 6c ) show the accuracy of using greedy decoding (temp=0). In Fig. 6b , dashed lines show the pass@1 accuracy in sampling experiments with temp=1. In Fig. 6d , dotted lines show the accuracy after excluding trivial solutions. In Fig. 6h and 6i , we use *-Inst. to distinguish between the generation settings: infilling (GPT4-Turbo) and free-form generation (GPT4-Turbo-Inst.). More details are provided in Appendix H and I.Equality. BatchNorm2d in PyTorch applies batch normalization [25] on a 4D input tensor, with the second dimension as the number of features. We select the parameter num_features for the models to predict, with the equality constraint that num_features = input_shapes [1] . Figure 6a shows the results as we increase the difficulty by changing the maximum possible value for each input dimension. We observe that the DeepSeek Coder models drop from around 0.7∼0.8 to less than 0.5, while GPT-4's performance stays around 0.9 throughout different difficulty levels.Finding: Overall, we found that smaller LLMs even struggles with even the simple constraint of copying an existing value, while large state-of-the-art LLMs can maintain its high performance.Inequality. max in PyTorch computes the maximum value along a dimension. The parameter we target is dim with the valid range being [-rank, rank). In Figure 6b , when using greedy decoding, all 4 LLMs achieve close to perfect accuracy. Therefore, we also conduct sampling experiments and present the pass@1 accuracy and diversity in Figure 6b and 6c . For max we compute the diversity differently from Section 3.2 (see Appendix G), since the number of possible unique valid outputs is very small. Interestingly, the smaller DeepSeek Coder-1.3b model achieves highest sampling accuracy for rank=8, but has the lowest diversity. This is because the smaller model often predicts common values like 1, whereas the larger model (33b) can explore various correct answers like -1,2.Findings: We found that larger models are indeed better at capturing the simple inequality constraints and modeling the true probability of various possible values, while smaller models tend to memorize common patterns, leading to less diverse predictions.Arithmetic. reshape in both PyTorch and NumPy attempts to rearrange the dimensions in the input_data, with the constraint being i input_shapes[i] == j new_shape[j]. Since we found that it is common for the LLMs to simply predict the same shape or a permutation of the original, we add an additional constraint: we specify the first dimension of the new_shape to be different from any dimensions in input_shapes. Figure 6d shows the results as we vary the ranks of the input_data for PyTorch (similar trend in NumPy). We observe that most LLMs in the beginning perform well; however, as the difficulty increases, their performance drastically lowers. Meanwhile, GPT-4-Turbo performance does not drop even with more difficult inputs. We found the reason is that GPT-4-Turbo tends to always predict the special -1 value for reshape where the new_shape will be automatically inferred by the library. Figure 6d showcases this exact phenomenon in PyTorch (similar trend as NumPy) where dotted lines present the accuracy of any outputs without -1. We see that now even GPT-4-Turbo struggles in generating valid parameters without using the -1 crutch for the constraint.Conv2d in PyTorch applies a 2D convolution over a 4D input tensor. The LLMs are asked to predict the parameter groups, where they have to divide both in_channels and out_channels evenly. The default value for groups is the trivial 1 (and therefore always valid). To ensure that there is at least one non-trivial value for groups, we randomly sample in_channels and out_channels within the value range such that their greatest common divisor is greater than 1. Figure 6e shows that the accuracy steadily drops as we increase the magnitude of values: even GPT-4-Turbo can only solve ∼24% of the hardest subset of problems, which other models drop below 14% for the same problems.Fold in PyTorch aims to combine an array of sliding local blocks into a large containing tensor. The constraint required for fold is the most complex out of all studied APIs where the LLM tries to generate a k_size tuple, and the product of the tuple must divide the 2nd index of the input_shapes evenly. Furthermore, it also needs to satisfy a complex equation over multiple parameters as shown in Figure 6f . We use the default values for all parameters other than out_size and ask LLMs to produce the correct k_size. Shown in Figure 6f , due to the complexity of the constraint, even on the lowest difficulty with small values, LLMs achieve relatively poor accuracy compared to other APIs. As we increase the values, the accuracy drops to nearly 0%. This highlights the high degree of difficulty in many DS APIs which current LLMs cannot reliably solve.Findings: Arithmetic parameter constraints in DS APIs are extremely challenging for all LLMs. Our results show that current state-of-the-art LLMs cannot effectively solve such complex constraints with their performance drops drastically and even sometimes drops to zero as we increase the difficulty.Set-related. transpose in NumPy attempts to rearrange/transpose the input_data according to the given new_dim. In transpose, the constraint is that the model-predicted new_dim must be a permutation of the original dimensions in input_data. We found that the LLMs tend to predict very simple permutations; as such, similar to reshape, we directly provide the first dimension of new_dim to increase the difficulty. We see that in Figure 6g , LLMs generally perform well on solving this constraint, and their performance improves with larger model sizes. Interestingly, the lowest difficulty of rank = 2 has a drop in performance. We theorize that this is because when the rank is 2, it is more common to directly call transpose() without any additional arguments. Therefore, the LLMs struggle a bit when given this unnatural task when asked to predict new_dim in low ranks.Findings: We found that LLMs generally perform well across the set-related constraints, and their performance scales with increasing model sizes. However, they still struggle with uncommon or unnatural inputs that are no commonly seen during training.Instruction-tuned models. We additionally investigate the performance of instruction-tuned (IT) LLMs [59] with chain-of-thought (CoT) prompting [51] . Due to computational limitations, we selected 3 constraints from PyTorch on which GPT-4-Turbo (without CoT) performs poorly for this experiment and analysis. The detailed experimental setup is described in Appendix I. Recall that for Conv2d, the task is essentially to predict groups such that it is a common divisor of two integers.As we observe that some models tend to predict a trivial answer 1, we specifically mention "Don't set groups=1" in the prompt and consider such answer as invalid in evaluation. From Figure 6h , we observe that GPT-4-Turbo with CoT performs well at this non-trivial task, maintaining over 85% accuracy even with values up to 255. By contrast, the best open-source model can only solve 22%! This shows that although models like CodeQwen achieves close performance to GPT-4-Turbo on existing popular benchmarks like HUMANEVAL [10] , there is still a huge gap in terms of coding and math reasoning ability between GPT-4-Turbo and other open-source models. Meanwhile, when we use the same setup on the extremely difficult constraint in Fold, we see that even GPT-4-Turbo fails to perform well (less than 5% accuracy in later difficulty settings). This demonstrates that while CoT prompting may elicit better performance in constraints like in Conv2d, it still cannot effectively handle other more complex arithmetic constraints. In addition to CoT, we also test ReAct [57] , another prompting strategy to elicit more reasoning process from LLMs. We observe that while ReAct can perform better than CoT, it still fails to solve more complex arithmetic constraints (detailed in Appendix J). Additionally, we attempt to include API documentation in prompts, but found that this does not always improve performance on our tasks (detailed in Appendix K).Based on the above findings, we further construct a public benchmark -DSEVAL with the same individual parameter prediction setting and the same representative set of APIs as studied in the Section 4.3. For each API in the benchmark, there are 7 different difficulty settings (grouped as 2 easy, 3 medium, and 2 hard ones) and each with 200 randomly created problems. In total, this gives us 19,600 problems in DSEVAL to extensively evaluate the performance of different LLMs.Table 2 shows the accuracy and diversity of all 8 models. First, we observe that the LLMs' accuracy drops when increasing the difficulty levels on the benchmark problems. This is also reflected by prior results where LLMs across the board struggle with more difficult problems. Next, we see that GPT-4-Turbo consistently achieves the highest accuracy across all difficulty levels, showing the gap between state-of-the-art proprietary models and other open-source LLMs. Furthermore, we observe some interesting ranking changes across difficulty levels. For example, while CodeQwen1.5 [3] achieves the second-best performance in the lowest difficulty level, its performance drops substantially on the medium and hard problems (second worst on PyTorch medium and hard). Other models like StarCoder [31] improve their relative performance and achieve higher ranking on more difficult problems, showing that different LLMs can perform differently depending on the input and constraint required to satisfy.We also study the diversity (see Appendix G for more details) of the LLM outputs, except we do not study GPT-4-Turbo due to its cost. Interestingly, LLMs which achieve high ranking in accuracy do not necessarily perform well in generating diverse correct solutions. This indicates that certain LLMs generate similar solutions to satisfy the constraint, without paying attention to the specific context. Therefore, they are not suitable for tasks like fuzz testing [16] which requires efficiently exploring a large solution space, or for tasks involving uncommon API usage. We further categorize some common mistakes made by LLMs on DSEVAL and provide additional insights in Appendix E. Overall, DSEVAL serves as the first benchmark to systematically evaluate the performance of LLMs on satisfying complex numeric API constraints for popular DS libraries and can be extended to support additional APIs and DS libraries. 
related work: 
 LLMs for code. LLMs have made remarkable advancements in a wide range of coding tasks, including code synthesis [60, 10, 2] , debugging [11, 8] , repair [53, 54, 7] , and analysis [36, 56, 55] . Notably, recent works [29, 16] also demonstrated LLMs' effectiveness in synthesizing DS code, which requires programming proficiency in DS APIs from specialized libraries such as NumPy [38] and PyTorch [41] . Trained on billions of code including such DS code, LLMs, such as StarCoder [31] and DeepSeek Coder [22] , have been extensively evaluated on DS code synthesis tasks. However, no prior study has systematically examined whether LLMs can indeed understand numerical API constraints of these scientific libraries instead of just memorizing the trained data [14] .Coding benchmarks for LLMs. Most code generation benchmarks [10, 33, 2, 22] are formulated with a natural language description and tests to verify the functional correctness of LLM-generated code. However, these benchmarks mostly target general-purpose code. To access LLM code generation for DS tasks, DS-1000 [29] is created by collecting real DS problems from StackOverflow, and ARCADE [58] evaluates LLMs' ability to solve multiple interrelated problems within DS notebooks. Compared to existing DS benchmarks, our study explores different granularity levels to systematically evaluate to what extent LLMs can implicitly learn DS APIs' numeric parameter constraints.Math reasoning of LLMs. To evaluate LLMs' arithmetic reasoning performance, GSM8K and other benchmarks [12, 42, 35, 24, 28] construct math problems in natural language requiring mathematical computations to solve. Compared to these existing benchmarks, problems designed in our study implicitly encode the arithmetic logic inside the DS library API, and thus can evaluate the LLMs' capability in understanding and solving numerical API constraints in the important DS libraries.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: Yes. The data and code will be made publicly available soon, along with detailed instructions to replicate the main experimental results.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we present the first systematic study on how LLMs understand the numerical API constraints for important DS libraries. Our study results show that current LLMs often memoize common patterns rather than truly understanding the actual numerical API constraints. Moreover, GPT-4-Turbo largely outperforms other open-source models and can well understand some simple arithmetic constraints using CoT. Based on our finding results, we also constructed DSEVAL, the first benchmark (with 19,000 problems) for systematically evaluating LLMs' capabilities in understanding the important numerical API constraints for popular DS libraries (such as PyTorch and NumPy).Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: In Section 2, Section 3 and Appendix I, we provide all experimental setups for benchmark creation and evaluation.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow prior benchmarking work like HumanEval [10] and directly report the LLM performance without including error bars. Furthermore, to conduct a detailed statistical test, it would be extremely costly on our large dataset especially with models such as GPT-4.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the societal impacts in introduction. Since data science libraries are widely used, we hope our work can facilitate future research to improve language models' ability in this domain, and enable people to write data science programs faster. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [No] Justification: This paper constructs a synthetic dataset, and does not have a high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all the models we evaluated. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: Yes, the data and code will be made publicly available soon.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper evaluates language models and does not involve human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We first begin by describing the type of programs we are targeting as well as any terminology definitions. In our study, we aim to verify the ability of LLMs on satisfying the numerical constraints for DS library APIs. Figure 1 shows an example of a typical DS program consisting of different DS APIs created by following these steps: i) importing the DS library (e.g., PyTorch) to be used in the program; ii) obtaining or generating input data; iii) performing data manipulation operation using the input data to produce outputs. In order for a DS program to be valid, it needs to satisfy the constraints required in the DS library APIs used. Next, we will describe each component of a DS program in more detail.DS APIs. After importing the DS library, we start by obtaining or creating some input data to be used in the program. The data creation process is done using a specific type of DS APIs, refer to by us as generation APIs. The output of generation APIs is a specific data structure (e.g., tensors, arrays, dataframes) used by each DS library defined by the parameters of the APIs. In the example from Figure 1 , the generation API is torch.randn to produce the input data of x. The DS library specific input_data commonly has the following properties: i) shape: the size, rank or dimensions of the data structure (e.g., [20, 16, 50 , 1000]) ii) dtype: the type of primitive data in the input_data (e.g., float) iii) value: the exact data values in the input_data. While most of the constraints in DS library APIs focus on relationship of the shape, both dtype as well as value can be important to satisfy additional constraints. Furthermore, more complex generation APIs can create heterogeneous input_data that contains different dtypes or even nested structures.Using the input_data created by the generation API, DS programs then use manipulation APIs to perform additional operations. Different from generation APIs where the output produced depends only on the provided API parameters, manipulation APIs create the output based on both the API parameters and the input data. We refer to API parameters as the options used to initialize the behaviour of the manipulation API. In the example, the manipulation API is Conv2d and is first defined with a set of parameters (e.g., kernel_size) and then in the next line applied on x to create the output y. For the program to be valid, the parameters of the manipulation API must satisfy the corresponding constraints between API parameters and the properties of the input_data. Note that we consider both model manipulation APIs (like Conv2d in the example) as well as sequential manipulation APIs where the input_data is directly provided as a parameter of the API.API constraints. We refer to API constraints as the set of relationships between properties of input_data and API parameters that, if and only if when satisfied, leads to a valid DS API invocation.Figure 1 provides some of the example API constraints for Conv2d. We observe that not only are there constraints between the properties of the input_data and API parameters (e.g., in_channel % groups = 0, kernel_size ≤ H + 2 * padding), but there are also constraints within API parameters (e.g., out_channel % groups = 0). Failure to satisfy any one of those constraints will lead to an invalid DS API innovation where, when executed by the DS program, will lead to a runtime error.Table 3 lists all the 12 APIs we studied in Section 4.3 and Section 4.4, where we ask LLMs to predict a single API parameter. In Column "Constraint", the underlined parameter is the one that the models need to predict, and we also list all the parameter constraints related to it. Column "Category" presents the categorization following Table 1 , highlighting that our selected APIs and API parameters can cover all the categories and are representative of their group.Difficulty settings. As discussed in Section 3.1, we have two different difficulty settings depending on the specific APIs, namely rank and rank. Below are the detailed setups for each API.• For torch.max, np.squeeze, np.argmax, np.transpose, np.max whose constraints that are more related to tensor dimensions, we design the difficulty level by increasing the rank of input data (i.e., how many dimensions it has). • For torch.nn.BatchNorm2d, torch.nn.Fold, torch.nn.Conv2d, torch.nn.MaxPool2d, np.split whose constraints that are more related to the actual values (either API parameter or the dimensions of input data), we design the difficulty level by increasing the range of relevant values.• For torch.reshape, np.reshape, since their constraints are closely related to both rank and value, we study both settings for each of them. To create a set of diverse problems for LLMs to predict a single API parameter, we randomize the context, namely the input data shape and other API parameters. Since the goal of our study is to focus on numeric API constraints, we want to control the complexity or naturalness of the constraintrelated variables, and ensure that the other unrelated parameters are always within a reasonable and common range. More specifically, during problem creation, if the irrelevant API parameter is an optional parameter, we just use its default value. If the irrelevant API parameter is a required API parameter, then we randomly choose a value according to the common value range listed in Table 4 . For example, for torch.nn.Conv2d(in_channels, out_channels, kernel_size, groups=1), the targeted parameter is groups, and kernel_size is a irrelevent but required parameter. Therefore, across all difficulty levels, we pick kernel_size randomly from [1, 10] .To obtain Table 4 , we perform an extensive study and refer to developer-written unit tests [44, 40] , API documentations [43, 39] , and existing DS library fuzzing literature [50, 23, 32, 30] to gather the common value range for each API parameter.When evaluated using the full API parameter setting (Section 4.2), we observe a much more significant accuracy drop for Linear compared to the other APIs (Figure 5 ). To gain deeper insights, we conduct an in-depth case study of this API.For torch.nn.Linear(in_features, out_features), the only constraint is that the in_features should match the last dimension of the input tensor. However, the DeepSeek Coder-33b model tends to copy the wrong dimension of the input tensor, likely because it has not seen lots of high-rank tensors (rank > 4) in the pre-training data.tends to use other APIs as "short-cuts" and forgo the analysis on torch.nn.Linear directly as shown in Figure 8 . Figure 8 : Example of GPT-4-Turbo's incorrect response. In this example, instead of operating on the original input tensor x and set in_feature to its last input dimension (10) , GPT-4-Turbo multiplies all the dimensions together and performs a flattening operation (x.view(1, -1) before invoking Linear). This violates the instruction to the model, which prohibits modifying the API invocation code. As such, this model response is evaluated as incorrect.In this section, we categorize some common mistakes made by LLMs during our experiments and offer some additional insights and explanations.• LLMs struggle with uncommon input tensors: We found that across many APIs and constraints, LLMs struggle when provided with uncommon input tensor ranks (i.e., rank > 4) or uncommon shapes (e.g., x = torch.rand (9, 30, 23, 4) ). The reason is that LLMs are mostly trained with data that contains very common shapes or ranks. As such, LLMs can easily make mistakes on uncommon inputs.• LLMs tend to predict common parameter values blindly: We also observe that LLMs tend to generate common parameter values (e.g., 0, 1, powers of 2) which often turn out to be incorrect. This is again because LLMs are trained with pre-training code that frequently contains such parameter patterns and thus are likely to predict them even given a different input context.• LLMs pay attention to the wrong tokens/irrelevant parameters: LLMs can learn spurious correlations and pay attention to the wrong context tokens. For example, opensource LLMs struggle with the simple equality constraint in_features=input.shape[-1] in torch.nn.Linear because the attention weights are focused on the irrelevant parameters.In this section, we provide the additional results for the individual API parameter setting.Reshape. reshape in NumPy contains the same functionality and constraint as the PyTorch. Figure 9a shows the result across two difficulty dimensions. Similar to the result discussed in Section 4.3, we also observe the same trend in NumPy where GPT-4 is able to achieve superior performance.MaxPool2d. MaxPool2d in PyTorch applies a 2D max pooling over a 4D input tensor. We focus on predicting the API parameter padding, where it needs to satisfy the following constraint: kernel_size ≤ H + 2*padding (the problem is already simplified by setting stride to 1 and setting W=H). Figure 9b shows that even GPT-4 is incapable of getting this type of non-trivial linear constraints right when the value range increases to [128, 256), and we observe that it tends to predict 0 which is the default value for padding and therefore fails to satisfy the second constraint. Meanwhile, DeepSeek Coder models, especially the 6.7B variant, do surprisingly well in the highest difficulty level we studied. After inspecting the outputs, we find that the DeepSeek Coder-6.7B model is able to predict an expression kernel_size//2 instead of a constant number for padding, and since the expression correctly characterizes the constraints it always leads to a valid solution.Squeeze. squeeze in NumPy aims to remove any dimension of length one from the input_data.The constraint is for the LLM to predict a dimension dim where input_shapes[dim] = 1. We add an additional constraint when generating the input_data such that there is only one dimension with length one (only one correct dimension). Figure 9c shows the result as we increase the rank in input_data. We observe that while in smaller ranks (2-3), the LLMs achieve close to perfect accuracy, the performance quickly drops off when we increase the rank to be >4, where all 3 LLM sizes perform similarly. We found that this is because LLMs tends to generate dimensions of 0 or 1 (commonly seen in example code and pre-training data). As such, when given a high ranked tensor where the correct squeeze dimension can be much higher than 0 or 1, the LLM struggles to satisfy the simple constraint. This again demonstrate the memorization issue where LLMs tend to predict commonly seen parameters during training instead of reasoning over the actual API constraint.Besides accuracy (percentage of valid programs generated), we also measure the diversity of the valid programs generated. In particular, based on the exact API, we use different diversity metric: i) For APIs where the number of possible valid outputs are large and not restricted (e.g., torch.reshape), we sample the LLM multiple times (100) with high temperature (1) and compute the percentage of unique valid programs generated as discussed in Section 3.2.ii) For APIs where the number of possible valid outputs are fixed (e.g., np.max can only select valid dimensions within a range), we again sample the LLM multiple times and then compute the distance between uniformed valid distribution and the distribution produced by the LLM. For example, if the set of all valid answers is {-1,0,1}, and the model predicts {-3,-2,0,0,0,1,1,1,1,1} in 10 samples, then the LLM's distribution is P={-1: 0, 0: 0.3, 1: 0.5, others: 0.2}, and the reference distribution is Q={-1: 1/3, 0: 1/3, 1: 1/3, others: 0}. Next, we compute the Hellinger distance [52] between the two discrete probability distributions and compute the diversity as 1 -distance.Since it requires a large amount of sampling programs (100 samples per problem) to evaluate diversity, in Section 4.4, we evaluated the diversity of each LLM only on a single difficulty level, i.e., the third level, either rank=4 or value in [8, 16) . GPT-4-Turbo is an instruction-following LLM and we do not have access to a base version that supports direct infilling. Therefore, we use a infill-specific prompt to ask GPT-4-Turbo to only fill in the missing code without adding any additional text. This setup allows us to compare against other infilling LLMs in the same setting. On the other hand, in this paper (e.g., Fig. 6h ), we use "GPT-4-Turbo-Inst" to differentiate the free-form generation setting. "GPT-4-Turbo-Inst" indicates that we allow it to generate additional text (such as CoT or ReAct reasoning steps).Figure 10 shows the prompt used by us to perform infilling using GPT-4. Note we separate out the system prompt and how we format an example input. Additionally, we modify the library name in the system prompt depending library of the API. Experiment setup. We design CoT prompts shown in 11. Note that the prompt is slightly different for different models due to their specific format requirements. Additionally, for torch.nn.Conv2d, we use a custom prompt (shown in Figure 12 ) where we explicitly add a sentence "Don't set groups=1" to avoid trivial answers. We use greedy decoding and set max_new_tokens to 512 for all models and all APIs, except for torch.nn.Fold we use max_new_tokens=1024 since LLMs tend to predict longer text before the actual code for this challenging API.Results. Figure 13 shows the additional results on constraints chosen to evaluate the instructiontuned LLMs. Figure 13a, 13b, and 13c shows the result of the different variants on Conv2d. Furthermore, we also evaluate on the constraint in MaxPool2d with the result shown in Figure 13d .We applied the ReAct prompting strategy [57] on the difficult APIs constraints studied using CoT in Section 4.3. Figure 14 shows our ReAcT prompt for GPT-4-Turbo. Note that the prompt is slightly For the `groups` parameter in your code, you need to specify an integer that divides both the number of input channels (`in_channels`) and the number of output channels (`out_channels`). The `groups` parameter is used to control the connections between inputs and outputs. Setting `groups` to 1 means all inputs are convolved to all outputs. If `groups` is set to `in_channels`, it means each input channel is convolved with its own set of filters (like a depthwise convolution).In your case, since `in_channels` is 180 and `out_channels` is 135, you need to find a divisor of both 180 and 135 to use as the `groups` value. The greatest common divisor (GCD) of 180 and 135 is 45. Thus, you can set In this example, GPT-4-Turbo is able to accurately recall the numeric constraint for the targeted parameter groups, apply them on this specific input example (in_channels=180, out_channels=135), and generate valid answer 45.different for different models due to their specific format requirements. Our ReAct prompt asks the LLM to first generate a thought and then produce the code output. We also provide the LLM with a single demonstration of the ReAct task. Furthermore, similar to the CoT experiments, we also explicitly add the sentence "Don't set groups=1" to avoid trivial answers. Instruct from 20% to 45% accuracy (Figure 17d ). However, there are also similar cases where adding documentation decreases performance. For example the GPT-4-Turbo-Instruct performance falls from 57.5% to 22.5 in the most difficult setting of torch.nn.MaxPool2d (Figure 17b ).Since we provide the raw documentation text without further processing, the success rate of adding documentation can vary depending on the specific model as well as the quality of the documentation. As such, this demonstrates that naively adding API documentation cannot always achieve better performance on our tasks.We perform both LLM generation and evaluation on an 64-core workstation with 256 GB RAM running Ubuntu 20.04.5 LTS. For local open-source LLMs, we use NVIDIA RTX A6000 GPUs. For GPT-4-Turbo experiments, we directly access the API endpoint provided by OpenAI.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Split. For split in NumPy, the goal is to predict a dimension dim which can divide the index parameter index evenly. Just like squeeze, we also add an additional constraint when generating the input_data such that there is only one dimension that is divisible by index. Figure9dshows the results, where we see that even with rank of 2, the accuracy is just above 50%, showing the increase in difficulty of the constraint in split As the rank increases, we also observe a huge decrease in performance, where LLMs again overwhelmingly predict dimensions of 0 or 1. 


Decision: The authors create a new data set called DS_EVAL, which tests LLMs ability to reason about numeric constraints when doing code generation. The data set consists of 19,600 programs, which call 28 different APIs from pytorch and numpy, and each call is ranked in terms of difficulty.  Constraints refer to things like specifying valid dimension numbers given the shape of arrays, specifying valid argument to reshape, specifying valid permutations when doing transpose, etc.   An SMT solver is used to validate that the LLM output satisfies the constraints.

The authors compare a number of LLMs, and find that LLMs are generally able to handle easy cases using common shapes that occur frequently in training data, but cannot reason about constraints in harder and less common cases.  

Reviewer responses to this paper were mixed.  Three reviewers recommend "weak accept" (6), one recommends "weak reject" (4), and the last recommends "reject" (3).   Praise for the paper was quite uniform: 

"Originality: First systematic study of LLMs' ability to handle numeric constraints" (reviewer xaud)
"first, thorough and systematic study of these constraints"  (reviewer bNWw)
"The paper explores a new and interesting problem" and "The study is thoroughly conducted"  (reviewer WbTY)

Reviewer (qmKc) [weak reject] is concerned that "the paper lacks important details." -- namely how the constraints were determined and formalized, and how the example programs are constructed.  They also complain that "the experiment is limited in scale in terms of the number of studied APIs."  IMHO, these are valid criticisms, but are not strong enough to reject the paper.  

Reviewer (YxwD) [reject] complains that the "paper just provides a comprehensive evaluation on existing LLMs, without proposing any possible solution for this problem" and suggests that the paper should have been submitted to the datasets and benchmarks track instead.

Overall, I personally feel that this is a valuable paper, which is worthy of publication.  The paper examines a particular aspect of code generation that has not been previously explored in the literature, and is likely to be increasingly important as LLMs are applied to more complex coding tasks.  It provides a comprehensive evaluation of existing SOTA models.  Moreover, because this is a benchmark, it can be used by other researchers to test and improve LLM coding capabilities going forward.  

However, I do think that reviewer (YxwD) has a point.  This is clearly a "datasets and benchmarks" paper, and perhaps should have been submitted to that track.

Reviews: review: paper_summary: paper_summary: This paper constructs a benchmark, DSEval, of 19600 programs across 28 data science (DS) library APIs with numerical constraints and uses the benchmark to invesitigate the capability of LLMs in generating valid DS programs which satisfy those numerical constraints. Additionaly, this paper categories the constraints into four different groups: equality, inequality, arithmetic and set-related. The experiments include 3 different generating settings: 1) full program, 2) all parameters, and 3) individual parameters, and study 8 LLMs including both closed-source and open-source models. The paper shows that LLMs are great at generating simple DS programs, but the performance of LLM drops significantly when the difficulty increases.
strength_weakness: strength_weakness: - The paper lacks important details.
- Some claims need better justification.
- The experiment is limited in scale in terms of the number of studied APIs.

One contribution of the paper is constructing the benchmark and the paper claims that the design of the benchmark is general and can be easily extended to additional libraries. However, the paper lacks details of how the benchmark is constructed, making it difficult to evaluate this claim. First, lines 11/63/165 state the benchmark contains 28 APIs while line 80 says 12 APIs. The authors need to clarify on this. Second, how the 28 or 12 APIs are selected? what does it mean by representative? Third, do these constraints exist? If so, were they written by developers? What format do these constraints take? Are they in natural language or a formal format? If they are in natural language, how are they converted into a formal format? Forth, are these 19600 programs from existing projects? Fifth, the number of such APIs is large, but the paper only studies 28 or 12 APIs which is pretty a small scale. 


Regarding the full program setting, how the 3-step instuction is contructed for individual program? From the example, I can see 3 lines of instructions and 3 lines of code. Are all the programs composed with 3 lines of code? I believe it should not be the case. If there are more than 3 lines of code, does it mean that there will be more than one line of code for each instruction? The paper uses SMT solver to validate the correctness of the generated program. It checks the validation based on constraits. Since the constraints for sepecific for one API. I believe SMT solver can verify the correctness of individul statement. How about the logic of the whole piece of the generated program? It would be better to clarify on this.
questions: questions: 1. Why only 28 or 12 APIs are studied given a large number of APIs?

2. Could you elaborate on the details of the API constraints used in constructing the benchmark? Please see the detailed comments for clarification points.

3. How does the proposed work verify the correctness of the logic of a full program?
limitations: limitations: The paper provides a discussion on limitation.
paper_summary: paper_summary: This paper investigates the ability of large language models (LLMs) to implicitly learn and apply numeric parameter constraints in data science (DS) APIs, focusing on PyTorch and NumPy libraries. The authors conduct a comprehensive study across 28 representative APIs, evaluating LLMs in three settings: full program generation, all parameter prediction, and individual parameter prediction. They introduce DSEVAL, a benchmark containing 19,600 problems across 12 APIs with varying difficulty levels. The study evaluates both open-source and closed-source LLMs, including state-of-the-art models like GPT-4. Results show that while LLMs perform well on simple constraints and common patterns, their performance degrades significantly with increased difficulty or unusual inputs. The authors conclude that current LLMs, including GPT-4, struggle with complex arithmetic constraints and often rely on memorization of common patterns rather than true understanding of the underlying constraints. This research highlights the limitations of LLMs in handling numeric API constraints and provides a benchmark for future improvements in this area.
strength_weakness: strength_weakness: Absence of error bars or statistical significance tests: Lacks quantification of result certainty. Suggestion: Include error bars or conduct statistical tests to strengthen the validity of findings

Limited exploration of prompting techniques: Minimal investigation of advanced prompting methods. Suggestion: Experiment with more diverse prompting strategies to potentially improve LLM performance. 

Lack of human baseline: No comparison to human performance on similar tasks. Suggestion: Include a human baseline to contextualize LLM performance 

Limited discussion on potential solutions: Doesn't offer many concrete suggestions for improving LLMs in this domain. Suggestion: Propose and discuss potential approaches to enhance LLM capabilities in handling numeric constraints
questions: questions: 1. Why did you limit your study to PyTorch and NumPy? Wouldn't including more libraries make your findings more generalizable?

2. Your study lacks a human baseline. How can we interpret the LLM performance without knowing how humans perform on similar tasks?

3. You don't provide error bars or statistical significance tests. How confident are you in the reliability of your results?

4. Your paper doesn't explore fine-tuning or more advanced prompting techniques. Couldn't these potentially improve LLM performance significantly?

5. The paper lacks an in-depth error analysis. Wouldn't categorizing common mistakes provide more insights into LLM limitations?
limitations: limitations: Lack of human baseline: The study does not provide a comparison to human performance on similar tasks, making it challenging to contextualize the LLM performance.

Absence of error analysis: The study lacks a detailed categorization and analysis of common error patterns, which could provide deeper insights into specific LLM limitations. 

No error bars or statistical significance tests: The paper does not include measures of statistical uncertainty, which could strengthen the validity of the findings.

Limited discussion on potential solutions: While the paper identifies limitations in LLM performance, it offers few concrete suggestions for improving LLM capabilities in handling numeric constraints.
paper_summary: paper_summary: The authors systematically investigate how well current LLMs learn numeric
parameter constraints of functions and deep learning operators in the Numpy and
PyTorch libraries. Their main finding is that although it is widely assumed that
current LLMs can solve arithmetic constraints, the performance of even
state-of-the-art LLMs like GPT-4-Turbo drops drastically when the complexity of
the constraints increases, and accuracy sometimes even drops to 0.

Another interesting finding is that there is a huge gap between open source
models and GPT-4-Turbo, which is not captured by benchmarks like HumanEval. The
authors introduce a public benchmark called DSeval based on these findings which
demonstrates this gap and allows future research to measure and narrow it.
strength_weakness: strength_weakness: Some of the terminology is not aligned with the widely used meaning of the
concepts, which makes the paper harder to read. Most significantly, the paper
talks about APIs like sum, max, or reshape. These are usually considered
functions and/or deep learning operators , and not APIs. APIs are usually
considered to be larger, like the API of a library such as PyTorch, or the API
of OpenAI. Also, PyTorch is usually thought of as a deep learning library, not a
data science library.



Based on this I would probably not call the benchmark DSeval. Also, the
parameter constraints are probably also a salient feature of the benchmark and
the work, so I would call it something along the lines of DLParamEval, or
DLParamConstraintEval. Naming the benchmark is of course the authors's choice,
so this is just a suggestion and won't influence my score.

The paper doesn't distinguish between GPT-4 and GPT-4-Turbo. The authors usually
write about GPT-4 while GPT-4-Turbo is used in the evaluations.

Some smaller remarks:
- I believe that the last three lines of the input are actually part of the
  output in Figure 2a.
- The explanation of subfigures could be added to the caption of Figure 2
- full program, all parameters, and individual parameters could be explained
  briefly also near line 66
questions: questions: In Figure 5, why does the accuracy drop much more significantly for Linear than
for the other functions/constructors? Is this only true for DeepSeekCoder-33B?
limitations: limitations: There is no section on limitations but I don't believe that one would be
necessary as the whole work is about the limitations of LLMs.
paper_summary: paper_summary: This paper investigates a problem, precisely the title, can LLMs implicitly learn numeric parameter constraints in data science APIs? To investigate this problem, this paper constructs a benchmark, DSEVAL, which contains a series of API calling based code completion tasks. Then, this paper evaluates several LLMs to derive the conclusion: LLMs can implicitly learn the constraint pattern during training, but lack genuine comprehension of the underlying constraints.
strength_weakness: strength_weakness: - I am not sure whether this paper should be submitted to Dataset & Benchmark track, since this paper focuses on evaluating LLMs’ underlying capabilities for satisfying parameter constraints when calling data science APIs, rather than proposing any novel algorithmic advances, analysis, or applications. 

- This paper should provide some naïve solutions as the starting point for this problem. However, this paper just provides a comprehensive evaluation on existing LLMs, without proposing any possible solution for this problem.

- I think one natural question is that whether LLMs can solve this problem by using ReAct [1] prompt strategy, i.e., each time LLMs call a data science API, LLMs are prompted to first output a thought about the constraint for this API, and then generate the parameter. I am curious about the results about this setting.

[1] ReAct: Synergizing reasoning and acting in language models, ICLR 2023.
questions: questions: See weakness.
limitations: limitations: I donot find any discussion about limitation in this paper.
paper_summary: paper_summary: In data science library APIs, there is a numerical parameter constraint between input data and parameters. This paper presents an empirical study on whether large language models (LLMs) can implicitly learn numerical constraints in data science library APIs. The findings indicate that LLMs demonstrate the ability to memorize common patterns in the use of popular data science APIs, and their performance significantly declines when the rank or dimensions of the input data increase, suggesting a general lack of true understanding of the API's numerical constraints. Additionally, the paper reports several other findings. It also introduces a dataset named DSEVAL, designed to evaluate LLMs' capabilities in understanding numerical API constraints for some popular data science libraries.
strength_weakness: strength_weakness: 1. While the problem of numerical parameter constraints in the data science APIs discussed in this paper exists, it may not be particularly significant and appears to be easily resolvable. This raises concerns about the potential for further research.
2. Some of the experimental details of the paper are unclear and require additional elaboration.
questions: questions: 1. I believe that by incorporating API documentation into fine-tuning or including it within prompts, LLMs might better understand numerical parameter constraints, which could potentially address the issue you're exploring.

2. API parameters can vary across different versions of a library. Could these variations impact the results of your study?

3. Why do you choose to conduct the study with 28 APIs, whereas only 12 APIs are used in the DSEVAL dataset? What accounts for this difference?

4. The specific LLM used during input creation is not mentioned. Could you clarify which LLM is employed?

5. In Figure 5, why do different sets of APIs get affected as the input rank and dimensionality increase? Furthermore, how do rank and dimension affect the complexity of these APIs?
limitations: limitations: The authors have not dedicated a separate section to discussing the limitations; however, they consider the limitations to be associated with the scope of their study. I consider this sufficient.

Comments: title: Thanks for your review and great suggestions!
Thanks for your review and great suggestions
comment: Thanks again for your great suggestions to categorize and analyze the common error patterns that help us strengthen the paper! We would like to share with you our new interesting findings regarding other common mistakes made by LLMs. In additional to 1) struggling with uncommon input tensors; and 2) predicting common parameter values blindly, we found that 3) **LLMs pay attention to the wrong tokens/irrelevant parameters**: LLMs can learn spurious correlations and pay attention to the wrong context tokens. For example, open-source LLMs struggle with the simple equality constraint `in_features=input.shape[-1]` in `torch.nn.Linear` because the attention weights are focused on the irrelevant parameters.

**Attention analysis:** To give an example, for the DeepSeek-1.3B model and the API `torch.nn.Linear` with difficulty of rank=5, the input tensor has 5 dimensions and is presented in context like `x = torch.randn(5, 4, 2, 14, 11)`, and the model is asked to predict a parameter `in_features`, with the constraint being `in_features==input.shape[-1]`. For this simple task, the LLM only achieves 6% accuracy. To investigate why, we compute the attention weight of each input token (maximum of all attention heads across all layers) to locate the most significant token among the relevant ones (e.g., `5, 4, 2, 14, 11` in the previous example). Next, we map it to a specific dimension of the input tensor (i.e., `0, 1, 2, 3, 4`). Here’s the result:

Out of the 200 tests, here is the number of times that the predicted value matches each input dimension: `{0: 30, 1: 113, 2: 40, 3: 19, 4: 12}`. The correct answer should be `4`, the last dimension of a rank-5 tensor. Note that the sum of this is not 200 because the predicted value can match more than one input dimensions if they are identical.

We observe that the predicted value almost always (98%) matches with the context token that has the highest attention weight. This indicates that the LLM does learn to always copy a specific dimension from the input tensor. As for the detailed break-down: (1) When the attention is paid to the wrong context token/parameter value (92%), it leads to incorrect results; (2) For 1% of the times, it pays attention to the right dimension; (3) Interestingly, for 5% it copies from the wrong position but that specific value just happens to be correct. 

We will incorporate your suggestion of including comprehensive error pattern categorization, the attention visualization results, and other remarks in your review in our next revision. Thanks again for your support and valuable comments!
Thanks again for your great suggestions (especially pointing out the interesting results on `torch.nn.Linear`) that helps us strengthen the paper! We would like to share our new interesting findings regarding the common mistakes made by LLMs. In additional to 1) struggling with uncommon input tensors; and 2) predicting common parameter values blindly, we found that 3) **LLMs pay attention to the wrong tokens/irrelevant parameters**: LLMs can learn spurious correlations and pay attention to the wrong context tokens. For example, open-source LLMs struggle with the simple equality constraint `in_features=input.shape[-1]` in `torch.nn.Linear` because the attention weights are focused on the irrelevant parameters.

**Attention analysis:** To give an example, for the DeepSeek-1.3B model and the API `torch.nn.Linear` with difficulty of rank=5, the input tensor has 5 dimensions and is presented in context like `x = torch.randn(5, 4, 2, 14, 11)`, and the model is asked to predict a parameter `in_features`, with the constraint being `in_features==input.shape[-1]`. For this simple task, the LLM only achieves 6% accuracy. To investigate why, we compute the attention weight of each input token (maximum of all attention heads across all layers) to locate the most significant token among the relevant ones (e.g., `5, 4, 2, 14, 11` in the previous example). Next, we map it to a specific dimension of the input tensor (i.e., `0, 1, 2, 3, 4`). Here’s the result:

Out of the 200 tests, here is the number of times that the predicted value matches each input dimension: `{0: 30, 1: 113, 2: 40, 3: 19, 4: 12}`. The correct answer should be `4`, the last dimension of a rank-5 tensor. Note that the sum of this is not 200 because the predicted value can match more than one input dimensions if they are identical.

We observe that the predicted value almost always (98%) matches with the context token that has the highest attention weight. This indicates that the LLM does learn to always copy a specific dimension from the input tensor. As for the detailed break-down: (1) When the attention is paid to the wrong context token/parameter value (92%), it leads to incorrect results; (2) For 1% of the times, it pays attention to the right dimension; (3) Interestingly, for 5% it copies from the wrong position but that specific value just happens to be correct. 

We will incorporate your suggested `torch.nn.Linear` analysis, the attention visualization results, and other remarks in your review in our next revision. Thanks again for your support and valuable comments!
To Area Chair: Thanks for noticing and posting the comment!

To Reviewer qmKc:

Thanks for reading our rebuttal and for your reply!

> My main concern is the number of PIS and complexity of the problem. Compared with similar benchmarks in such kind, the evaluation is small and not comprehensive. 


We kindly point out here that DSeval is the **first** benchmark targeting the validity of DS API parameter constraints, consisting of **19600** different problems that span **fine-grained** settings and difficulty levels for 12 APIs, and we evaluated across 8 state-of-the-art open/close source code LLMs. For comparison, DS-1000 [29], which is the closest benchmark we found in the DS code generation domain, contains 1000 problems. Would you kindly share examples of “similar benchmarks in such kind” you have in mind?

Please allow us to further clarify our criteria for dataset construction. 

Firstly, our chosen APIs are the core APIs commonly used by users. While there are a large number of APIs in DS libraries, the commonly used ones (e.g., Conv2d) are not that many. For example, the widely-used benchmark DS-1000 [29] only covers 16 PyTorch APIs and 59 NumPy APIs (after excluding data construction APIs like “np.ones”).

Additionally, not all the APIs have numerical parameter constraints. For example, NNSmith [32] supports 73 core operators that are commonly used in DL programs, and only 22 of them have numerical constraints that fall in the scope of our study, and we’ve included *all 22* of them (plus 6 additional ones) in our first two settings (full program generation and all parameter generation).

Furthermore, some APIs have similar constraints or same constraint types. For example, numpy.max and numpy.min have the same constraints, whereas torch.nn.Conv2d and torch.nn.Conv1d have very similar constraints. Therefore, in our DSeval benchmark, we select 12 representative APIs to keep the experiments at an affordable scale, while still ***covering all major constraint categories*** in Table 1. We believe this is a representative and **comprehensive (in terms of constraint types)** set of APIs and constraints used to evaluate LLMs’ capabilities. 

> The rebuttal mentions that this can be extended without real evidence.

Thanks for asking this follow-up question. We apologize that we didn't include concrete evidence in our initial response, as it wasn't specifically requested at that time. Please find the supporting evidence below:

We can extend the experiment to new APIs with little engineering effort. For example, we’ve already added torch.nn.Linear into our benchmark during the rebuttal as suggested by reviewer bNWw, and obtained interesting results. Building upon the original framework, we only need to add 17 lines of code for the newly supported API. We didn’t include more APIs due to computing budget constraints, but we are happy to extend the study to all 28 APIs and even more if the reviewer thinks it is crucial. 

As we also mentioned in our previous response, the reason that we can easily support the addition of new APIs is because our framework uses symbolic constraint solving techniques to both generate and validate them. New APIs can be added by simply encoding the numeric constraints (written as basic z3 formulas). Similarly, we can also add more complex chained API sequences using our framework by again symbolically modeling each API to build the symbolic computation graph and construct a program. Please note that we built our framework upon NNSmith [32], and thus such graph-level modeling is intrinsically supported and only requires minimal modification to support our use case.

More concretely, **to verify the correctness of the logic of a full program, we just need to symbolically build the API graph, propagate the tensor shapes, and verify the validity of each API.** To give a more concrete example, let us consider two APIs with *symbolic shapes and symbolic parameters*:

- F: Input shape [x,y], parameter a; Output shape [x, a]; Constraint: a<y
- G: Input shape [x,y], parameter b; Output shape [x, y-b]; Constraint: y>b

Chaining F(G(input)) with symbolic input [u,v]:
1. G([u,v]) -> [u,v-b]; Constraint: v>b
2. F([u,v-b]) -> [u,a]; Constraint: a<v-b

Note that we have already symbolically modeled the shape transfer rules and validity constraints of each individual API. Therefore, the whole process uses symbolic expressions, allowing seamless propagation and validity checking through the API chain without modifying individual APIs.

Please let us know if our response has addressed your concern, and we are more than happy to provide further evidence if the reviewer is willing to specify which aspects they’d like us to elaborate on or what particular evidence they’re seeking.
We are very sad to hear that the reviewer believed our effort spent during multiple responses and added experimental results were not sufficient. We have made every effort possible to address all the main concerns raised, and hope the reviewer can also look at the other positive reviews and reconsider. Below are our answers:

>I do not think the raised concerns are clear misunderstandings. I mention the reference [1] to demonstrate why I think this paper does not discover exciting findings for me. I didn’t claim that this paper and [1] investigate similar problems.

If the reviewer believes that [1] and our work do not investigate similar problems then how does the finding of [1] affect the contribution of our work? Just like our previous replies, we again point out that scientific discovery is not based on whether something is “exciting” or can be easily “predicted”. The prediction (i.e., hypothesis) to be investigated needs to be rigorously shown and proven through experiments. We believe that our work is an extensive evaluation of the implicit ability of LLMs to satisfy valid numeric constraints in DS programs, which is an extremely important problem in the era of deep learning. 

> I didn’t request additional experimental results during author-reviewer discussion. Also, the concern about initial solutions is already included in my initial review (See Weakness 2).

You did. The original review did not mention anything about fine-tuning or pre-training, even including Weakness 2. In fact the original review only requested a new experiment on ReAct in Weakness 3: *“I think one natural question is that whether LLMs can solve this problem by using ReAct [1] prompt strategy, i.e., each time LLMs call a data science API, LLMs are prompted to first output a thought about the constraint for this API, and then generate the parameter. I am curious about the results about this setting.”*. We did add this ReAct experiment and demonstrate interesting results, but then the reviewer completely ignored our new ReAct results, and asked for new experiments on fine-tuning/pre-training, e.g., in the new [comment on Aug 11](https://openreview.net/forum?id=LfC5rujSTk&noteId=rrQOuQhfQa): *“The original manuscript only provides CoT as the initial solution, while providing ReAct and documentation-augmentation during the rebuttal. All these three methods are performed during inference stage. How about pretraining/finetuning stage?”*.
From Area Chair:  This reply is from reviewer qmKc.  I believe it was accidentally posted without the "visible to authors" box checked.  :-)

---
Thank you for the rebuttal. I would keep my ratings after reading the rebuttal and some thinking. My main concern is the number of PIS and complexity of the problem. Compared with similar benchmarks in such kind, the evaluation is small and not comprehensive. The rebuttal mentions that this can be extend without real evidence.
Thanks for your further reply. I want to clarify some key points.

**1. I do not think the raised concerns are clear misunderstandings.**

I mention the reference [1] to demonstrate why I think this paper does not discover exciting findings for me. I didn’t claim that this paper and [1] investigate similar problems.

[1] Case-Based or Rule-Based: How Do Transformers Do the Math? ICML 2024.

**2. I didn’t request additional experimental results during author-reviewer discussion. Also, the concern about initial solutions is already included in my initial review (See Weakness 2).**

The mentioned experiments are the reasons why I think this paper does not provide sufficient insights about potential reasons for the phenomenon.

Although I acknowledge that the authors have made efforts during the rebuttal period, **from my perspective**, this paper does not meet the high standards expected at NeurIPS and my concerns about the contribution remain unresolved, so I will maintain my original score.
Thanks for reading our response and sharing your thoughts!

>The PyTorch documentation also suggests that APIs are not single functions. If you open it using the link you provided, you can immediately see that the "Python API" refers to the collection of all of the functions, classes, etc.; it's not the case that each function is an API.

We acknowledge that there is a difference of opinion regarding the definition of an API in terms of DS libraries including PyTorch. Here we refer the reviewer to prior work [a, b, 32] that also considers each individual function/class/etc. as an API. We will definitely make our definition clear in the next revision of our paper. 

**References**

[a] DocTer: Documentation-Guided Fuzzing for Testing Deep
Learning API Functions. (ISSTA 2023)

[b] Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source. (ICSE 2022)
>If Figure 6h refers to the CoT results, I think that the legends are wrong? Or could you tell me which line refers to CoT?

The dashed lines with triangular markers in Figure 6h refer to the CoT prompting approach when using the instruction-tuned LLMs (as described in line 296-297). We included the complete CoT prompt used by us for all models in Appendix. The solid lines with circular markers refer to the base LLMs (not the instruction-tuned variants), allowing us to compare and contrast their performances. 

>I think documentation augmentation can be a strong strategy. But from Figure 14, GPT-4 turbo w/ doc performs poorly compared with GPT-4 turbo. This is quite surprising for me. Could you please provide the prompt for GPT-4 turbo w/ doc?

Great observation! We would like to point out that adding documentation does not always decrease performance. For example, in the `torch.nn.Fold`, adding documentation is able to improve performance of GPT-4-Turbo across all difficulty levels (Figure 14 (e)). However, like the reviewer pointed out, there are also similar cases where adding documentation decreases performance. For example the GPT-4-Turbo drops in performance when given the documentation for `torch.nn.Conv2d` (Figure 14 (c)).



Please note that the success rate of adding documentation can vary depending on the specific model, API studied, as well as the quality of the documentation. For example, although modern LLMs are more and more powerful, they are still far from perfect and cannot always effectively leverage all the information provided. Such “surprising” findings can hopefully also inspire various future work for further improving the ***implicit*** reasoning capability of LLMs.

Here is the complete prompt for GPT-4-Turbo for `torch.nn.Fold` with documentation:

System Prompt:
```
You are an expert Python programmer and are good at writing correct PyTorch code.
```

Input Prompt:
````
Please refer to the given API documentation and complete the Python program.
Documentation for the torch.nn.Fold API:
{documentation_omitted_due_to_space}

Please complete the program by filling in the correct API parameter(s). You should keep the exact same program unchanged, just fill in the missing code part.
```python
import torch
x = torch.randn(90, 96, 126)
m = torch.nn.Fold(output_size=(15, 10), kernel_size=<insert code here>, dilation=1, padding=0, stride=1)
```
````

In the above example, we omit the exact document due to the reply word limit. We provide the raw documentation of each API (obtained from the source code docstring). An example can be found here: https://pytorch.org/docs/stable/_modules/torch/nn/modules/fold.html#Fold

>Moreover, could you tell me the difference between gpt4-turbo and gpt4-turbo-inst.? I think that gpt4-turbo from OpenAI is already the instruction-following version.

Sorry for the confusion, you are right that GPT-4-Turbo is already the instruction-following version. We use *-Inst. to differentiate the generation setting: infilling (GPT-4-Turbo) and free-form generation (GPT-4-Turbo-Inst.) used in our study. For GPT-4-Turbo, we use our infill-specific prompt (See Appendix F for an example) to ask it to **only fill in** the missing code without adding any additional text. This setup allows us to compare against other infilling LLMs in the same setting. On the other hand, for GPT-4-Turbo-Inst., we allow it to generate additional text (such as CoT or ReAct reasoning steps). Please note that GPT-4-Turbo-Inst. is only used in our CoT experiments in the original paper. Thanks again for the question, and we will definitely have better naming for the next revision to avoid such confusion.
> (3) LLMs pay attention to the wrong tokens/irrelevant parameters. Could you please tell me which part of your paper can lead to this finding? I do not find any attention map visualization in this paper. Please correct me if I am wrong.

We apologize for the confusion. The attention analysis is part of the additional in-depth error analysis we conducted during the rebuttal period following the suggestion of Reviewer xaud.

Since we couldn’t provide figures in the comments, we explain the results in text below. We will add the attention visualization as well as more in-depth discussion in our revision.

To give an example, for the DeepSeek-1.3B model and the API `torch.nn.Linear` with difficulty of rank=5, the input tensor has 5 dimensions and is presented in context like `x = torch.randn(5, 4, 2, 14, 11)`, and the model is asked to predict a parameter `in_features`, with the constraint being `in_features==input.shape[-1]`. For this simple task, the LLM only achieves 6% accuracy. To investigate why, we compute the attention weight of each input token (maximum of all attention heads across all layers) to locate the most significant token among the relevant ones (e.g., `5, 4, 2, 14, 11` in the previous example). Next, we map it to a specific dimension of the input tensor (i.e., `0, 1, 2, 3, 4`). Here’s the result:

Out of the 200 tests, here is the number of times that the predicted value matches each input dimension: `{0: 30, 1: 113, 2: 40, 3: 19, 4: 12}`. The correct answer should be `4`, the last dimension of a rank-5 tensor. Note that the sum of this is not 200 because the predicted value can match more than one input dimensions if they are identical.

We observe that the predicted value almost always (98%) matches with the context token that has the highest attention weight. This indicates that the LLM does learn to always copy a specific dimension from the input tensor. As for the detailed break-down: (1) When the attention is paid to the wrong context token/parameter value (92%), it leads to incorrect results; (2) For 1% of the times, it pays attention to the right dimension; (3) Interestingly, for 5% it copies from the wrong position but that specific value just happens to be correct. Please kindly let us know if any further analysis is needed, and we are happy to add that in the next revision.


>The original manuscript only provides CoT as the initial solution, while providing ReAct and documentation-augmentation during the rebuttal. All these three methods are performed during inference stage. How about pretraining/finetuning stage? For example, I think providing the experiments of pretraining/finetuning LLMs with the designed uncommon inputs can support the assumed reasons.

Again, we want to stress that our goal is to evaluate the ***implicit*** reasoning capability of LLMs in solving numeric constraints (as indicated by our title as well as introduction paragraphs). We totally agree with the reviewer that having training experiments can potentially reinforce and unlock additional insights. However, this is ***outside the scope of our study***. Additionally,  the reviewer only asked for the additional ReAct prompting experiment for the rebuttal period, which we did. Due to the time limit and resource cost of the reviewer-author-discussion period we are unable to perform any pre-training or fine-tuning experiments. 

Furthermore, prior work [b, c, d] with similar contributions (i.e., discovering important findings/limitations for LLMs) also do not include any pre-training or even fine-tuning experiments. We would also like to point out that fine-tuning and pre-training with task-specific synthetic data on small-sized LLMs do not always lead to the same conclusion when done on large state-of-the-art LLMs trained with a mixture of different real-world data.


**References**

[b] Statistical Knowledge Assessment for Large Language Models. https://openreview.net/forum?id=pNtG6NAmx0 NeurIPS 2023 (poster).

[c] Towards Understanding Factual Knowledge of Large Language Models. https://openreview.net/forum?id=9OevMUdods ICLR 2024 (poster). 

[d] KoLA: Carefully Benchmarking World Knowledge of Large Language Models. https://openreview.net/forum?id=AqN23oqraW ICLR 2024 (poster).
We truly appreciate the prompt response! Meanwhile, we noticed some misunderstandings and additional experiment requests which were not mentioned in the original review (so we did not have a chance to address in the rebuttal period). We are happy to respond to them in detail below.

>The mentioned surprising findings do not seem that surprising for me, which is also the main reason why I think that this paper investigates an interesting yet less significant problem – the key finding in this paper (i.e., LLMs can implicitly learn simple constraints but fail for complex constraints during pre-training) is predictable. ... These two findings can be indicated by [1]. 

Please note that in our previous response, we referred to [1] as Hu, Yi, et al.’s ICML 2024 or [Hu, Yi, et al.] and have already discussed the fundamental difference. We will make sure to discuss this related work in our revised manuscript.

Again, we would like to point out that our work and [1] study two completely different problem domains:

1. First, different from the **explicit** problems in [1] (e.g., `1+2=?`), in our study, the API constraints are **not** directly specified in the problem. Instead the LLM should **implicitly** learn the constraints through pre-training on large amounts of open-source DS code examples. This makes the fundamental problem setup different between our work and [1]. 

2. Second, our problems are also completely different from the synthetic problems (e.g., chicken and rabbit problem) used in [1] that are rarely seen during pre-training. Instead, there are more than 2,400,000/480,000 open-source projects using NumPy/PyTorch, meaning there exists a massive amount of DS program examples in the pre-training dataset in an exact or very similar format as in our benchmark. This makes our study’s starting point completely different from prior work [1]. 

3. Third, we would like to point out that there are other works similar to [1] which show completely different findings. For example [1] shows that scratchpad fine-tuning underperforms compared to direct fine-tuning, while another work [a] demonstrates that fine-tuning with scratchpad (especially with few-shot examples) can significantly improve length generalization in the coding domain. 

Furthermore, we kindly argue that the value of scientific progress and discovery is not measured by the predictability of the final conclusion. In our work, we perform a rigorous study on evaluating the implicit ability of LLMs to satisfy valid numeric constraints in DS programs, which is an extremely important problem in the era of deep learning. We believe that even if some of the main conclusions are predictable, we are the first one to demonstrate this for the important domain of DS code generation. Additionally, our detailed analysis of each LLM’s performance on different problem settings and difficulties, along with our insights, can provide concrete guidelines for improving code LLMs and inspire future work.

**References**

[a] Exploring Length Generalization in Large Language Models. https://openreview.net/forum?id=zSkYVeX7bC4 NeurIPS 2022 (poster).
Thank you for your answers!

It is indeed an interesting result, thank you for obtaining it!

Regarding your second answer, I don't believe that Data Science is a subfield of
Machine Learning but that's probably a matter of opinion so it won't influence
my score.

The PyTorch documentation also suggests that APIs are not single functions. If
you open it using the link you provided, you can immediately see that the
"Python API" refers to the collection of all of the functions, classes, etc.;
it's not the case that each function is an API.

Similarly, if you go to the "Functional higher level API", for example, you can
see that the first sentence reads: "This section contains the higher level API
for the autograd that builds on the basic API above and allows you to compute
jacobians, hessians, etc.". So the API is the collection of the functions on
this page.
A quick reminder: Did the author's response adequately address your questions and concerns? Can you please write a brief response to let them know that you read it?
Thanks for your further clarification. However, I still have the following main concerns.

**1. The mentioned surprising findings do not seem that surprising for me, which is also the main reason why I think that this paper investigates an interesting yet less significant problem – the key finding in this paper (i.e., LLMs can implicitly learn simple constraints but fail for complex constraints during pre-training) is predictable.**

> (1) LLMs struggle with uncommon input tensor. (2) LLMs tend to predict common parameter values blindly.

These two findings can be indicated by [1]. LLMs perform case-based reasoning instead of rule-based reasoning: LLMs struggle with uncommon input tensor due to lack of similar case, and LLMs tend to generate common parameter values due to similar case in pretraining.

[1] Case-Based or Rule-Based: How Do Transformers Do the Math? ICML 2024.

> (3) LLMs pay attention to the wrong tokens/irrelevant parameters.

Could you please tell me which part of your paper can lead to this finding? I do not find any attention map visualization in this paper. Please correct me if I am wrong.

**2. The assumed potential reasons for these findings should be supported by experiments.**

> LLMs struggle with uncommon input tensors: We found that across many APIs and constraints, LLMs struggle when provided with uncommon input tensor ranks (i.e., rank > 4) or uncommon shapes (e.g., x = torch.rand(9, 30, 23, 4)). The reason is that LLMs are mostly trained with data that contains very common shapes or ranks. As such, LLMs can easily make mistakes on uncommon inputs.

For example, I think providing the experiments of pretraining/finetuning LLMs with the designed uncommon inputs can support the assumed reasons.

**3. The original manuscript only provides CoT as the initial solution, while providing ReAct and documentation-augmentation during the rebuttal. All these three methods are performed during inference stage. How about pretraining/finetuning stage?**

- If Figure 6h refers to the CoT results, I think that the legends are wrong? Or could you tell me which line refers to CoT?

- I think documentation augmentation can be a strong strategy. But from Figure 14, GPT-4 turbo w/ doc performs poor compared with GPT-4 turbo. This is quite surprising for me. Could you please provide the prompt for GPT-4 turbo w/ doc? Moreover, could you tell me the difference between gpt4-turbo and gpt4-turbo-inst.? I think that gpt4-turbo from OpenAI is already the instruction-following version.
> From my perspective, the answer is predictable:  LLMs can implicitly learn simple constraints but fail for complex constraints during pre-training

**Despite the seemingly obvious answer, our result reveals multiple surprising findings**. For example, Figure 13 (see attached PDF in global response) shows that for the simple equality constraint: `in_features=input.shape[-1]` in `torch.nn.Linear`, open-source LLMs cannot generalize to out-of-distribution problems (i.e., tensor rank > 4) and have less than 20% accuracy. In contrast, GPT-4-Turbo is able to maintain its performance. One potential reason is that LLMs are mostly trained with data with very common shapes or ranks (e.g., rank=3, 4), leading them to always copy the 3rd/4th shape dimension, while the true rule is to copy the last dimension. As such, LLMs can easily make mistakes on uncommon inputs. For example, when rank=6, they still copy the 3rd or 4th dimension. We also examined the attention weights and found that when LLMs make mistakes for `torch.nn.Linear`, it is usually because they are assigning larger attention weights to incorrect positions, and therefore copying values from a wrong parameter. We believe there can be lots of interesting follow-up works from our fine-grained benchmark and analysis.

> (a) the potential reasons for this phenomenon

Please also note that we did conduct in-depth analyses based on different problem settings, constraint types (e.g., equality, inequality, arithmetic, and set-related), and across LLMs, and have interesting findings (highlighted in Section 4.1, 4.2, and 4.3). For example, we found that there is a large difference between open source LLMs and state-of-the-art LLM’s performance, especially on complex constraints. Such difference in performance has not been observed previously on standard coding benchmarks (e.g., HumanEval) where the performance gap between open and close source LLMs are considerably smaller. We also have additional interesting findings and analyzed potential reasons regarding the common mistakes made by LLMs, such as:

1. **LLMs struggle with uncommon input tensors**: We found that across many APIs and constraints, LLMs struggle when provided with uncommon input tensor ranks (i.e., rank > 4) or uncommon shapes (e.g., `x = torch.rand(9, 30, 23, 4)`). The reason is that LLMs are mostly trained with data that contains very common shapes or ranks. As such, LLMs can easily make mistakes on uncommon inputs.

2. **LLMs tend to predict common parameter values blindly**:  We also observe that LLMs tend to generate common parameter values (e.g., 0, 1, powers of 2) which often turn out to be incorrect. This is again because LLMs are trained with pre-training code that frequently contains such parameter patterns and thus are likely to predict them even given a different input context.

3. **LLMs pay attention to the wrong tokens/irrelevant parameters**: LLMs can learn spurious correlations and pay attention to the wrong context tokens. For example, open-source LLMs struggle with the simple equality constraint `in_features=input.shape[-1]` in `torch.nn.Linear` because the attention weights are focused on the irrelevant parameters.

> (b) some possible initial solutions for this issue during pretraining / finetuning / inference stage

Please note that we have explored possible initial solutions such as prompting techniques like CoT during the inference stage. We also added the new ReAct and documentation-augmented experiments as suggested by reviewers.

We also want to point out that the exact reason for investigating this problem is because the model has been exposed to a large amount of similar data to the problems in our studied settings during the pre-training stage. This led to our goal of studying the ***implicit*** ability for LLMs to satisfy numerical constraints across different constraint types and difficulty levels without any additional fine-tuning or prompting.

Additionally, we believe that before exploring additional new techniques to improve performance, building an extensive and high quality benchmark is extremely important. To enable future research and evaluate potential solutions, we constructed a detailed benchmark -- DSeval. We believe this is an essential and crucial contribution, allowing new techniques to be developed and tested in this important domain of generating correct and valid DS code.
Thanks for reviewing our response and providing further feedback. We would like to politely point out that in our initial rebuttal response, we addressed all the reviewer’s initial questions and concerns. This included adding the requested experimental results demonstrating that even advanced prompting techniques like ReAct, cannot handle complex constraints. Please let us know which specific responses in the initial rebuttal that the reviewer still has issues with. Additionally, we also respectfully disagree with the reviewer’s new comments. Please see our detailed responses below:

>This paper investigates an interesting yet less significant problem

**We believe that evaluating the LLMs’ ability to generate correct and valid DS programs is extremely important**. Data science applications, and by extension DS libraries/APIs, are used daily by developers to process and analyze large amounts of data to build ML systems and make decisions. As such, DS APIs have penetrated almost every corner of modern society in the era of deep learning, including autonomous driving software [9, 27, 45], financial systems [18, 4], and coding assistants [44, 36]. LLMs are being widely used to aid in and generate programs, especially in the important domain of DS code. A recent study [e] has shown that GPT-4 can achieve performance on par with humans on various data analysis tasks. Additional work [f] has demonstrated improved performance when using LLMs to assist data analysts. Furthermore, there has been active research like [g] and the development of automatic tools like Data-Copilot [h] that use LLMs to automatically solve data science tasks. Due to this wide adoption, it is critical to check whether LLMs can implicitly learn the numeric constraints in data science APIs. While fine-grained analysis has been done in the math reasoning domain (e.g., Hu, Yi, et al.’s ICML 2024), none of the prior work has focused on the DS code generation domain. In this work, we advocate for a thorough examination of the assumption -- LLMs can implicitly learn the correct DS API parameter constraints, relied on but not tested by many prior works [46, 21, 16].

Besides practical importance, we would also emphasize the notable distinctions in our approach and findings compared to previous work. **Our problems with DS API parameters are fundamentally different from the synthetic datasets or toy problems studied in prior work like Hu, Yi, et al.’s ICML 2024 work or [a,b,c,d], which we believe provides unique insights and values**. Firstly, our settings do not involve unnatural synthetic problems like linear regression function learning in [Hu, Yi, et al.] that are rarely seen during pre-training. Instead, there are more than 2,400,000/480,000 open-source projects using NumPy/PyTorch [i, j], meaning there exists a massive amount of DS API parameter examples in the pre-training dataset in an exact or very similar format as in our benchmark. Secondly, prior benchmarks typically present problems explicitly (e.g., `1+2=?`), while in our problem the constraints are implicit. This provides a unique testbed and poses an important theoretical question: Given a large set of examples $(\mathbf{X_1},\mathbf{X_2},\mathbf{X_3}, \ldots, \mathbf{X_N})$ where each assignment $\mathbf{X_i}=(x_i^1,x_i^2,...x_i^m)$ satisfies a certain constraint $\phi$ in first-order logic, can a model ***pre-trained*** on large mixture of data including these examples (i.e., $\\{\mathbf{X_i}\\}_{i=1}^N$) implicitly learn $\phi$ and generalize to predict $x_i^j$ when $x_i^{-j}$ is out of distribution?

**References**

[e] Is GPT-4 a Good Data Analyst? https://arxiv.org/abs/2305.15038 (2023).

[f] How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz
Study? https://arxiv.org/abs/2309.10108 (CHI 2024).

[g] Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering. https://arxiv.org/abs/2305.03403 (NeurIPS 2023).

[h] Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow. https://arxiv.org/abs/2306.07209 (2023).

[i] https://github.com/pytorch/pytorch/network/dependents 

[j] https://github.com/numpy/numpy/network/dependents
Thank the authors for providing the detailed feedback for my concerns. 

I have read the provided references but I think this paper fails in providing sufficient insights in evaluation, especially compared with [1, a-d].

This paper investigates an interesting yet less significant problem, i.e., whether LLMs implicitly learn the numeric constraints in data science APIs. From my perspective, the answer is predictable: LLMs can implicitly learn simple constraints but fail for complex constraints during pre-training. This paper just verifies this point by (1) constructing a benchmark; and (2) testing the LLMs. However, I expect more interesting and significant insights such as **(a) the potential reasons for this phenomenon**; **(b) some possible initial solutions for this issue during pretraining / finetuning / inference stage**. I think exploring these directions may bring much more contribution to the community.

Considering the points above, I do not think this paper provides enough contribution for a top conference like NeurIPS and I decide to keep my score unchanged.

[1] Case-Based or Rule-Based: How Do Transformers Do the Math? ICML 2024.