LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language

Authors: james  requeima
john  bronskill
dami  choi
richard e turner
david  duvenaud
  𝑥
  𝑦
⟨𝑡⟩  𝑥
⟨𝑡⟩𝑥  𝑛
𝑥  𝑛
𝑦  𝑛
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-05 23:46:16

Abstract: Machine learning practitioners often face significant challenges in formally integrating their prior knowledge and beliefs into predictive models, limiting the potential for nuanced and context-aware analyses. Moreover, the expertise needed to integrate this prior knowledge into probabilistic modeling typically limits the application of these models to specialists. Our goal is to build a regression model that can process numerical data and make probabilistic predictions at arbitrary locations, guided by natural language text which describes a user's prior knowledge. Large Language Models (LLMs) provide a useful starting point for designing such a tool since they 1) provide an interface where users can incorporate expert insights in natural language and 2) provide an opportunity for leveraging latent problem-relevant knowledge encoded in LLMs that users may not have themselves. We start by exploring strategies for eliciting explicit, coherent numerical predictive distributions from LLMs. We examine these joint predictive distributions, which we call LLM Processes, over arbitrarily-many quantities in settings such as forecasting, multi-dimensional regression, black-box optimization, and image modeling. We investigate the practical details of prompting to elicit coherent predictive distributions, and demonstrate their effectiveness at regression. Finally, we demonstrate the ability to usefully incorporate text into numerical predictions, improving predictive performance and giving quantitative structure that reflects qualitative descriptions. This lets us begin to explore the rich, grounded hypothesis space that LLMs implicitly encode.

Main paper: 
 introduction: 
 Incorporating prior knowledge into predictive models is highly challenging which can restrict the scope for detailed, context-sensitive analysis. In addition, the skill required to incorporate this prior knowledge into probabilistic modelling can restrict the use of these models to experts. In this work, our objective is to develop a probabilistic prediction model that facilitates user interaction through straightforward, natural language. For this purpose, we explore strategies for eliciting explicit, coherent numerical predictive distributions from LLMs.Why go to so much effort to elicit predictions from a slow, expensive, and sometimes inconsistent model like an LLM? We expect their hypothesis class to be both rich, and grounded in exactly the kinds of high-level side information that we currently struggle to communicate to our numerical models. For instance, knowing that prices rarely go below zero, that certain kinds of sensors can saturate at particular values, or that trends almost always eventually level off, are easy to express in natural language, but not straightforward to incorporate into a model without getting lost in difficult-to-specify details about aspects of the domain that aren't well understood. To summarize, we want to develop such a model because it would allow users to 1) provide prior, potentially expert, information to the model about the problem setting in plain-language rather than attempting to capture this information in closed form priors (e.g. Gaussian Process kernels) and 2) it would allow users to access problem-relevant latent knowledge encoded in LLMs that users may not have themselves.LLMs have recently been shown to be able to condition on the particular task being solved, leveraging contextual information to make better predictions or decisions (Stanton et al., 2022; Choi et al., 2022) . They have also been shown to competitively predict time series based only on a text tokenization of numerical data (Gruver et al., 2023) . In this work, we further push in both these directions; 1) using LLMs for numerical prediction tasks going beyond onedimensional time series forecasting to multi-dimensional regression and density estimation and 2) exploring the ability of these models to condition on both numerical data and rich, unstructured text to improve these predictions. In this paper we make the following contributions:• We define LLM Processes (LLMPs) using methods we develop for eliciting numerical predictive distributions from LLMs. 1 LLMPs go beyond one-dimensional time series forecasting to multi-dimensional regression and density estimation. We propose two approaches for defining this joint predictive distribution over a collection of query points and evaluate their compatibility in principle with the consistency axioms necessary to specify a valid statistical process.• We develop effective prompting practices for eliciting joint numerical predictions. We investigate various methods for conditioning LLMs on numerical data, including prompt formatting, ordering, and scaling. We characterize which schemes perform best on a set of synthetic tasks.• We show that LLMPs are competitive and flexible regressors even on messy data. Through an extensive set of synthetic and real world experiments, including image reconstruction and black-box function optimization, we evaluate the zero-shot regression and forecasting performance of LLMPs. We demonstrate that LLMPs have well-calibrated uncertainty and are competitive with Gaussian Processes (GPs), LLMTime (Gruver et al., 2023) , and Optuna (Akiba et al., 2019) . We show that LLMPs use in-context learning to automatically leverage information from related datasets, can easily handle missing datapoints, perform image reconstruction, and output multimodal predictive distributions.• Lastly, we demonstrate the ability to usefully incorporate problem-relevant information provided through unstructured text into numerical predictions, visualized in Figure 1 , resulting in quantitative structure that 1 Source code available at: https://github.com/ requeima/llm_processes reflects qualitative descriptions. Other additions such as labelling features using text and specifying units allow LLMPs to make use of usually-ignored side information. 
llm processes: defining a stochastic process that can condition on text: 
 Our goal for this section is to use an LLM to elicit joint predictive distributions over arbitrary sized target sets that we can guide and modify using natural language. Please see Appendix D for a discussion on work related to our method. Formally, given a set of input and output observations D train = {(x i , y i )} M i=1 and some text, T , we would like to elicit the predictive distribution defined by an LLM at a collection of targets {(x * j , y * j )} N j=1 denoted p LLM (y * 1 , . . . , y * N | x * 1 , . . . , x * N , D train , T ). Rejection sampling from an LLM allows us to access what we may interpret as the LLM's predictive distribution and gain insights into the model's inductive biases; sampling from the LLM's categorical distribution over text tokens while ignoring non-numerical tokens yields numerical samples from the LLM. The process of sampling from an LLM is depicted in Figure 2 and Algorithm 1 with sample prompts in Appendix C. Since an accurate sampling-based empirical distribution incurs a high computational cost, next we define an approach to elicit continuous likelihoods from an LLM.Continuous Marginal Likelihoods From an LLM. We approximate a continuous density over our target values by discretizing the space using bins with arbitrarily fine precision, similar to the method used in Gruver et al. (2023) . Crucially, this hierarchical approach allows us to compute the probability of a bin with width 10 -n . For example, if n = 1 then Pr{y ∈ [1.0, 1.1)} = p(1)p(.|1)p(0|1.) because '1.0' is a prefix for all y ∈ [1.0, 1.1) . We can convert probability mass to probability density by assuming a uniform distribution within each bin, and dividing the mass by the Unlike (Gruver et al., 2023) , we do not rescale the values to remove decimal places. We hypothesize that such scaling removes prior information communicated to the LLM via the scale of the problem. We examine the effect of scaling values in Section 3. We also differ from (Gruver et al., 2023) by including a terminal token after every value in our prompt -for example, given a terminal token ⟨t⟩, we represent 12 as 12⟨t⟩. Including a terminal token prevents numbers of varying orders of magnitude to share the same prefix -i.e. p(1)p(2|1)p(⟨t⟩|12) no longer includes the probability of numbers in [120, 130) , [1200, 1300) , etc.Note that this approach does not guarantee that P (12⟨t⟩) yields the mass assigned by the LLM to values in the bin [12, 13) but we empirically observed that our predictive distribution closely matches the sampling distribution to our satisfaction. See Section H.1 for details and comparisons.Defining an LLM Process. Thus far we have established a procedure defining the predictive distribution at a single target location, p LLM (y * n | x * n , D train , T ). We now outline two methods which we call independent marginal (I-LLMP) and autoregressive (A-LLMP) predictions, for defining the joint predictive distribution over a collection of target points: We note that Equation (1) satisfies the Kolmogorov Extension Theorem (Oksendal, 2013) therefore defining valid stochastic process (see Appendix A.3) . However, it assumes conditional independence given the training set and model weights and the stochastistity represented by the model is via independent marginals. Equation ( 2) takes inspiration from the autoregressive structure of the LLMs predictive distribution and should yield much richer predictive distri-butions as we are now able to model dependencies between output variables. However, this definition is no longer guaranteed to give us a valid stochastic process as the predictive distribution is now target order dependent and will likely fail the Kolmogorov exchangability condition. We investigate both of these questions in Section 3.Connection to Neural processes Neural Processes (NPs) (Garnelo et al., 2018) are a class of meta-learning models parametrized by neural networks and trained to learn a map from training (context) sets to predictive distributions,p θ (y * 1 , . . . , y * N | x * 1 , . . . , x * N , D train ).The definitions in Equations 1 and 2 take inspiration from the joint distributions defined by Conditional NPs (Garnelo et al., 2018) as independent marginals conditioned on the training/context set and Autoregressive NPs (Bruinsma et al., 2023) utilizing the chain rule of probability, respectively. Through this lens, LLMPs can be viewed as examples of NPs. However, NPs are directly trained to output this predictive distribution where as LLMPs are repurposing pretrained LLMs.Missing Data. We highlight that, through the flexibility of the LLM prompt, we do not have to draw a distinction between which variables, or variable dimensions are to be modelled or conditioned and can easily handle missing values. Suppose we have a collection of variables {x 1 , . . . , x n } and {y 1 , . . . , y m } (or more), some subset of which we would like to regress on (including x and y-values) and the remainder we wish to condition on. To do so using an LLMP, we simply construct the training prompt such that the variables we would like to regress on occur at the end of the prompt and are blank (generated) when sampling from the LLMP. If any values are missing they can simply be removed from the prompt. 
llmp configuration: 
 In all of the experiments, we use six different open source LLMs: Mixtral 8×7B, Mixtral-8×7B-Instruct (Jiang et al., 2024) , Llama-2 7B, Llama-2 70B (Touvron et al., 2023) , Llama-3 8B, and Llama-3 70B (AI@Meta, 2024). Our primary metrics are negative log probabilities of the model Prompt Engineering. We perform a set of experiments for determining the best LLMP prompt configuration. We use the Sigmoid, Quadratic, and Linear+Cosine functions with 10, 20 and 75 training points, respectively (see Appendix E.1) with I-LLMP using the Mixtral-8×7B LLM.• Prompt Formatting Two separators are required to achieve the best performance. One to separate the x and y values within a pair and another to separate the x, y pairs. Figure 3 (left) demonstrates that , \n is the best option in terms of performance and token efficiency.• Prompt Ordering Figure 3 (middle) shows that ordering the training points by distance to the current target point is best, outperforming both random and sequential ordering. We posit that ordering by distance provides a hint to the LLM to weigh the contribution of closer training points to the current target point to a greater degree.• Prompt y-Scaling Figure 3 (right) shows that performance degrades as the range of the y components of the training points increases and when incorporating negative values. This is due to the fact that when the range is wider, the LLM must accurately generate more numerical digits and potentially a negative sign when predicting f (x * ).• top-p and Temperature Figure H.9 shows that performance is surprisingly insensitive to varying the LLM nucleus sampling parameter top-p (Holtzman et al., 2020) and LLM softmax temperature.Autoregressive vs Independent Marginal Predictions. In this series of experiments we examine two questions: first, does the autoregressive defininiton of the joint predictive likelihood (A-LLMP) in Equation ( 2) improve performance versus the independent marginal definition of Equation ( 1) (I-LLMP). Second, "how close" is A-LLMP to a stochastic process in terms of performance variability across query orderings. We first look at log-likelihoods and MAE for A-LLMP and I-LLMP using the random and distance training point orderings discussed earlier. Results can be seen in Figure 4 (left). Similar to our findings earlier, ordering the training values according to distance to target has a large effect, improving performance for both I-LLMP and A-LLMP. Unsurprisingly, the richer joint distribution given by A-LLMP gives us better predictive performance.We next examine the variability in performance of A-LLMP when different autoregressive target orderings are used to get a sense of how far our method is from a stochastic process (which would be permutation invariant in the target points). The results of using ten sets of randomly ordered target points compared to I-LLMP and the ground truth log-likelihood of the test sample under the generative distribution are presented in Figure 4 (center). Note that the training data is distance sorted in all cases. We also present the result when ordering target points according to distance to the closest training point, from smallest to largest. We make three key observations: first, log-likelihood performance of all A-LLMP orderings is better than I-LLMP. Second, the variance of random orderings is small on the scale of the log-likelihood of the generative model. And third, distance ordering the targets gives better or at least competitive performance with a random ordering. These results present practitioners a choice: do you care more about using a valid statistical process or obtaining good predictive performance? If it is the latter, you would be better served using A-LLMP. 
evaluating llmp performance: 
 In this series of experiments, we evaluate the performance of LLMPs in a wide variety of settings. Additional details and results for all experiments in this section can be found in Appendix I.1D Synthetic Data Experiments. To show that LLMPs are a viable regression model with well-calibrated uncertainties, we benchmark in Table 1 our casting task using the weather dataset (described in Appendix E.2). Additional plots with missing training data are in Appendix I.3. We posit that A-LLMP betters LLMTime due to the fact that 1) A-LLMP naturally handles irregularly spaced x and y data whereas LLMTime uses only regularly spaced y information requiring imputation with NaN values where data is missing; and 2) A-LLMP performs no scaling on y values in contrast to LLMTime that scales data to eliminate the use of decimals and normalize the range of the data and as a result removes information that the LLM can potentially leverage.In the next three experiments we showcase the ability of LLMPs to handle multi-dimensional data.Image Reconstruction. As a 2-dimensional input experiment, Figure 6 shows reconstruction results from images drawn from the Fashion-MNIST dataset (Xiao et al., 2017) . We convert pixel data into prompt data points by forming a series of (row, column, pixel value) tuples. Additional results and details are in Appendix I.4. Using 20% train pixels, the basic form is captured and at 50%, the reconstruction is accurate despite the sharp pixel intensity transitions.Black-Box Function Optimization. Black-box optimization involves minimizing or maximizing a function where there is only access to the output of a function for a specified input. We benchmark the ability of LLMPs to perform maximization on six commonly used multi-dimensional functions. We compare our results using Llama-2-7B to Optuna (Akiba et al., 2019) , a commercial hyperparameter optimization framework. Results and implementation details are in Appendix I.5. In all cases, LLMPs obtain as good or better approximation to the true maximum value in a fewer number of trials.Simultaneous Temperature, Rainfall, and Wind Speed Regression. To examine how well an LLMP can model multi-dimensional outputs, we compare LLMP regression to a multi-output GP on the weather dataset described in Appendix E.2. Figure 7 shows the results for the Llama-3-8B LLM (top) and a 3 output RBF kernel GP with trained hyperparameters (bottom). The LLM is similar to and in most cases better than the GP in terms of MAE and NLL.In  
conditioning llmps on textual information: 
 One of the most exciting directions of LLMPs is the potential to incorporate prior information about problems via text. Now that we can examine functional predictive distributions of LLMs, we can begin to explore their rich prior over functions by conditioning on both text and numerical data. In this section we present two experiments with details and additional experiments presented in Appendix J.Scenario-conditional Predictions. In this experiment, we examine the influence of text providing information about various synthetic problem settings on the predictive distribution of an LLMPs. In all of the following examples, we provide the same two synthetic training points to the LLMP but change the prompting text that comes before the training data. We then use A-LLMP with Llama-3-70B to forecast trajectories 50 steps ahead. We begin by examining the predictive distribution with no prompt (Figure 10a ). We prompt the LLMP to generate daily temperature measurements in degrees Celsius from Montreal in January (Figure 10b ) and May (Figure 10c ), and monthly precipitation values from San Diego, CA (Figure 10d ) and Singapore (Figure 10e ). Figure 1 Shows the results of prompting the LLMP to generate (left) a stock price financial time series (centre) for a company that eventually goes out of business and (right) for a company whose price goes to zero on day 30.Indeed, the LLMP modifies the predictive distribution accordingly relative to the no prompt predictions. We highlight the following observations: first, for prompts b) and c), the model moves about half of its predictive mass below zero for temperatures beginning in January and above zero for the May temperatures. Second, the LLMP is able to recall actual historical trends for average monthly precipitation for Singapore and San Diego to condition on prompts d) and e). Despite getting the trend correct, we note that the median prediction in d) seems to be biased toward the training values and not reflective of the actual monthly median. Last, for stock price simulations, the model places all of its density on positive numbers since it is modelling prices. It is able to produce realistic trajectories and decreases them in expectation when prompted that the company goes out of business. The model is able to condition on the fact that the price goes to zero on day 30 which correctly interprets the meaning of the x-values as days starting from 0, that the y-axis is the price and the phrase "price goes to zero" corresponds to a y-value of zero.Labelling Features Using Text. In the following example, we examine the performance of a Mixtral-8x7B Instruct I-LLMP on predicting American housing prices. The dataset (Larcher, 2023) contains 39980 housing prices and various variables around housing and demographics for the top 50 American cities by population. Note that this dataset was generated on 12/09/2023, however it contains data from the 2020 US Census and the 2022 American Community Survey (ACS) so we cannot guarantee that models did not see data within this dataset during training.For each prediction task, we show the I-LLMP 10 randomly selected training examples from the dataset and predict on 20 randomly selected test examples. In the prompt, before the numerical value (price) we provide a string which encodes the datapoint index/features that the model can use.For our first experiment we examine the behaviour of the LLMP when more features are added to the prompt. We experiment with five ways of indexing the training and test points; For case (1), we provide latitude and longitude of the house as numerical values (eg. 32.74831, -97.21828) converted to strings similar to our method in previous experiments. For the remaining 4 cases, we provide additional labeled features, adding more features for each case with the prompt for case ( 5) containing all labelled features, illus- Note that the LLMP is able to take advantage of the additional features provided to improve predictive performance.To see examine the effect of adding text labels to the features, we ran another set of experiments on 10 new random datasets providing the LLMP with either labeled or unlabelled numerical features. 9 (right). Note that the LLMP is not able to use the raw feature values to improve performance from only 10 training examples, but is able to do so with labelled features suggesting that LLM is able to utilize the latent relationship between the feature and the price once the feature is identified. We found that the Mixtral-8×7B Instruct model had the best performance on this task and was able to utilize text information better (results for other models in Appendix J.2). 
discussion, limitations, and societal impact: 
 Discussion. We defined LLMPs for eliciting numerical predictive distributions from LLMs and when used as a zero-shot muti-dimensional regression model are competitive with GPs. Excitingly, we demonstrated the ability to condition on text to improve predictions and probe the LLMs' hypothesis space. An interesting extension would be to condition on other modalities in addition to text.Limitations. Along with the flexibility of LLMs, LLMPs inherit their drawbacks. Maximum context sizes limit the size of tasks we can apply this method to and the amount of textual information we can condition on. LLMPs are also significantly more computationally expensive compared to Gaussian Processes and standard regression methods. In this section we elaborate on the explanations and definitions in Section 2. Our goal is to use an LLM to elicit joint predictive distribution over arbitrary sized target sets that we can guide and modify using plain language. Formally, given a set of observations D train = {(x i , y i )} M i=1 and some text, T , we would like to elicit the predictive distribution defined by an LLM at a collection of targets {(x * j , y * j )} N j=1 denoted p LLM (y * 1 , . . . , y * N | x * 1 , . . . , x * N , D train , T ). To achieve the goal, we can can keep in mind two interpretations of what we mean by a predictive distribution defined by an LLM. First, we can interpret the LLM as maintaining having a predictive distribution over numerical values, which we can probe by sampling from the LLM. This interpretation is beneficial if we believe that the LLM has learned useful prior information that we would like to access via its beliefs about these numerical values and for our goal of guiding the predictive distribution using text. The other interpretation is more empirical: we simply use the LLM as a tool to define a valid predictive distribution and evaluate how well this definition performs on test cases. Our approach is a combination of the two philosophies -we will propose a method defining a predictive distribution that is valid and performs well on test cases, but closely matches what we think of as the LLM's underlying distribution.As discussed in Section 2, we use a method similar to the one proposed by Gruver et al. (2023) ; we approximate the continuous density by discretizing the space using bins with arbitrarily fine precision. Let's assume a fixed number of decimal places n, and that LLMs generate one digit at a time2 . The key idea is that each new digit can be viewed as being generated from a categorical distribution with the probabilities p given by a softmax over numerical tokens. Crucially, this hierarchical approach allows us to compute the probability of a bin with width 10 -n . For example, if n = 1 then Pr{y ∈ [1.0, 1.1)} = p(1)p(.|1)p(0|1.) because '1.0' is a prefix for all y ∈ [1.0, 1.1) . We can convert probability mass to probability density by assuming a uniform distribution within each bin, and dividing the mass by the bin width. A visualization of this construction can be viewed in Appendix H.1.The method in (Gruver et al., 2023) has two main shortcomings for our purposes: first, the authors propose to scale all y ∈ D train to eliminate decimals from their numerical representation. For example, for a precision of 2 decimal places, the numbers 0.123, 1.23, 12.3, and 123.0 will be transformed to 12, 123, 1230, and 12300 respectively. Scaling removes prior information communicated to the LLM via the scale of the problem. For example, it is likely that the LLM has encountered financial data with decimal places. Potentially, it also makes it more difficult to communicate prior information about the problem to the LLM via text.Second, probabilities of all sequences of integers given by an LLM contain the mass of all values that also start with that sequence. We can think of this as the problem of not knowing when the LLM intends to terminate a value. For example, if y = 12, Pr{y ∈ [12, 13)} ̸ = p(1)p(2|1) since p(1)p(2|1) includes the probability of all numbers with '12' as a prefix -this includes [12, 13) but also [120, 130) , [1200, 1300) and so on.We follow Gruver et al. (2023) and discretize the continuous space with bins of width 10 -n , computing the probabilities for each bin using the hierarchical softmax approach. However, different from their approach we 1) keep values at their original scale, and 2) include a terminal token after every value -for example, given a terminal token ⟨t⟩, we represent 12 as 12⟨t⟩ and 120 as 120⟨t⟩. Including a terminal token prevents numbers of varying orders of magnitude from sharing the same prefix -i.e. p(1)p(2|1)p(⟨t⟩|12) no longer includes the probability of numbers in [120, 130), [1200, 1300) , and so on. After we compute the mass of a bin via hierarchical softmax, we divide the mass by the bin width 10 -n to get an estimate of the density value. This procedure defines a valid predictive distribution over y-values, and we call this elicitation method 'logit-based' since we derive probabilities from the logits directly instead of sampling. Pseudocode can be found in Algorithm 2.It must be noted that this approach does not guarantee that P (12⟨t⟩) yields the mass assigned by the LLM to values in the bin [12, 13) . However, we note that our method defines a valid predictive distribution and we empirically observed that our predictive distribution closely matches the sampling distribution to our satisfaction (see Appendix H.1).So far we have established a procedure for defining the predictive distribution at a single target location, p LLM (y * n | x * n , D train , T ). We now discuss how to define the joint predictive distribution over a collection target points. In particular, we would like to define a stochastic process via its finite-dimensional marginal distributions ρ x1,...,x N defined over locations x 1 , . . . , x N . The Kolmogorov Extension Theorem (Oksendal, 2013) states that such a collection defines a stochastic process if it satisfies 1. Exchangeability: Given any permutation π of the integers {1, . . . , N }ρ x1,...,x N (y 1 , y N ) = ρ x π(1) ,...,x π(N ) (y π(1) , y π(N ) ) 2. Consistency: if 1 ≤ M ≤ N then ρ x1,...,x M (y 1 , . . . , y M ) = ρ x π(1) ,...,x π(N ) (y π(1) ,y π(N ) ) dy M +1 . . . dy NIn Equation ( 1) we define a collection of joint distributions by defining a factorized distribution over target locations x * 1 , . . . , x * N :p I-LLMP (y * 1 , . . . , y * N | x * 1 , . . . , x * N , D train , T ) = N n=1 p LLM (y * n , | x * n , D train , T )where p LLM (y * n , | x * n , D train , T ) is defined above. This definition satisfies the Kolmogorov Extension Theorem and so it defines a valid stochastic process. However, it assumes conditional independence given the training set and model weights and, conditional on these variables, the stochastistity represented by the model is via independent marginals. Taking inspiration from the autoregressive structure of the LLMs predictive distribution, we can write the joint distribution according to the product rule:p A-LLMP (y * 1 , . . . , y * N | x * 1 , . . . , x * N , D train , T ) = N n=1 p LLM (y * n | y * 1 , . . . , y * n-1 , x * 1 , . . . , x * n , D train , T )Where, the previous target location is autoregressively added to the conditioning data via the LLM prompt. This should yield much richer predictive distributions as we are now able to model dependencies between output variables. However, this definition is no longer guaranteed to give us a valid stochastic process as the predictive distribution is now target order dependent and most likely will fail the Kolmogorov exchangability condition. We investigate these questions in Section 3. Sequential:"T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩D *x " "T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩E *x " "T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩F *x " "T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩G *x " Random:"T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩D *x " "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩E *x " "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩F *x " "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩G *x " Distance:"T ⟨t⟩C x , C y ⟨t⟩B x , B y ⟨t⟩A x , A y ⟨t⟩D *x " "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩E *x " "T ⟨t⟩A x , A y ⟨t⟩C x , C y ⟨t⟩B x , B y ⟨t⟩F *x " "T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩G *x "Sequential:"T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩D * x " "T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩D * x , D * y ⟨t⟩E * x " "T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩D * x , D * y ⟨t⟩E * x , E * y ⟨t⟩F * x " "T ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩C x , C y ⟨t⟩D * x , D * y ⟨t⟩E * x , E * y ⟨t⟩F * x , F * y ⟨t⟩G * x " Random: "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩D * x " "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩D * x , D * y ⟨t⟩E * x " "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩D * x , D * y ⟨t⟩E * x , E * y ⟨t⟩F * x " "T ⟨t⟩C x , C y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩D * x , D * y ⟨t⟩E * x , E * y ⟨t⟩F * x , F * y ⟨t⟩G * x " Distance: "T ⟨t⟩C x , C y ⟨t⟩B x , B y ⟨t⟩A x , A y ⟨t⟩D * x " "T ⟨t⟩C x , C y ⟨t⟩D * x , D * y ⟨t⟩A x , A y ⟨t⟩B x , B y ⟨t⟩E * x " "T ⟨t⟩D * x , D * y ⟨t⟩A x , A y ⟨t⟩E * x , E * y ⟨t⟩C x , C y ⟨t⟩B x , B y ⟨t⟩F * x " "T ⟨t⟩D * x , D * y ⟨t⟩A x , A y ⟨t⟩E * x , E * y ⟨t⟩B x , B y ⟨t⟩F * x , F * y ⟨t⟩C x , C y ⟨t⟩G * x "In this section, we discuss work related to eliciting distributions from LLMs including forecasting, regression, and in-context learning among others. We also discuss related work on LLM hyperparameter optimization, eliciting priors from LLMs, eliciting distributions from humans, and finally neural processes.The most closely related work to ours is LLMTime (Gruver et al., 2023) . LLMTime is capable of zeroshot extrapolation of one-dimensional time series data at a level comparable to trained purpose-built approaches. In addition, they develop a method for eliciting marginal probability distribution functions from LLM posteriors over functions, which we build on. They also begin to investigate the effect of conditioning on text. In contrast, we focus on (i) interpolation with multi-dimensional inputs and outputs; (ii) eliciting joint distributions over functions, not just marginals; and (iii) exploring the ability of models to condition simultaneously on both numerical data and text. More recently, TimesFM (Das et al., 2023) , a foundation model for one-dimensional zero-shot times series forecasting was introduced. However, TimesFM does not support interpolation or higher dimensional data and does not consider distributions. PromptCast (Xue & Salim, 2023) performs zero-shot time series forecasting by combining numerical data and text in a question answer format. Our approach for combining problem specific text along with numerical data differs in that it handles both interpolation and extrapolation and does not rely on a question-answer format. Hegselmann et al. (2023) utilize LLMs to do zero-shot and few-shot classification on tabular data that compares favorably to standard ML approaches.LLM Regression Pesut (2022) do some initial investigations into the use of LLMs as regressors on 1D synthetic functions.Our work greatly expands on these early investigations. Vacareanu et al. ( 2024) is concurrent work that shows that LLMs are capable linear and non-linear regressors. However, their work does not condition on any textual information, compute log probabilities, compare to Gaussian Processes, investigate the the effect of prompt formatting, or employ auto-regressive sampling.In ). These models are parameterized using a neural network and there have been various proposals for different architectures using attention (Kim et al., 2019) , transformers (Nguyen & Grover, 2022), Gaussian Process output layers (Markou et al., 2021) , and diffusion models (Dutordoir et al., 2023) . The definitions of the joint distributions in equations 1 and 2 take inspiration from the joint distributions defined by Conditional Neural Processes (Garnelo et al., 2018) as independent marginals conditioned on the training/context set and Autoregressive Neural Processes (Bruinsma et al., 2023) utilizing the chain rule of probability, respectively. Through this lens, LLMPs can be viewed as examples of Neural Processes. LLMPs differ from standard NPs in two main ways: (i) Training objective: Neural Processes are meta-trained using maximum likelihood to optimize p(y * |x * , D train ) directly. LLMPs have a very indirect training procedure -they are trained to be language models i.e. autoregressive token predictors. One of the contributions of this paper is the demonstration that, despite this, they can perform zero-shot probabilistic regression. (ii) Architecture: NPs have an output layer that parametrizes the predictive distribution over targets directly. Since LLMPs are repurposing language models for regression, we need to define the mapping from distributions over language tokens to distributions over target variables. We note that LLMs themselves can be viewed as AR-CNPS (Bruinsma et al., 2023) with a fixed, predefined target ordering.This section provides details on the various datasets used in the experimentsWe use the 12 synthetic function datasets (Linear, Exponential, Sigmoid, Log, Sine, Beat Inference, Linear + Cosine, Linear × Sine, Gaussian Wave. Sinc, Quadratic, X × Sine) from Gruver et al. (2023) each of which consists of 200 discrete points. We construct 7 datasets each with 10 random seeds for each function with a subset of 5, 10, 15, 20, 25, 50, and 75 randomly training points sampled from the original 200 points. We add Gaussian noise with µ = 0 and σ = 0.05 to the training points and then round the values to 2 decimal places. Unless otherwise stated, we use 40 equally spaced target points to sample at.The dataset was queried from OpenWeather (2024) and consists of daily high temperature, precipitation, and wind speed readings for 86 consecutive days commencing on December 12, 2023. The data was recorded after the release dates of the Llama-2 and Mixtral-8x7B LLM release dates to avoid any data leakage into the LLM datasets.For the "Comparison to LLMTime" experiment, We used the first 50 readings of the temperature data for training data and ask LLMTime and LLMPs to predict/forecast the final 36 values. The authors of LLMTime suggest the method can handle missing values by inputting NaN values in their place. Since LLMPs can work with irregularly spaced and missing data, we also compare the methods with a reduced number of randomly spaced training points.For the "Simultaneous Temperature, Rainfall, and Wind Speed Regression" experiment we used 30 randomly chosen training points within the first 76 points, leaving the last 10 for extrapolation.It is likely that LLMs used in our experiments have been exposed during training to some of the real-world data that we use in our experiments which would give it an advantage against other models. However, we feel confident that the LLMs tested were not simply recalling memorized data -note that in all cases the LLMPs produces a full distribution and not just a deterministic value -and we have taken steps in our experiments to mitigate this issue. When synthetic functions or Fashion MNIST data (Xiao et al., 2017) is used, we have altered the original data via subsampling, rescaling and in some cases adding noise to the datapoints. Any data used from the internet was altered from its original form when given to the model. Some datasets (in particular the Weather Dataset described in Appendix E.2), were explicitly chosen to be recorded after the release dates of the LLMs that they were evaluated on.PyTorch is used as the basis for all of the experiments, with the exception of the Gaussian Processes baselines that are implemented using the GPyTorch package (Gardner et al., 2018) .The experiments using the Mixtral 8×7B, Mixtral-8×7B-Instruct (Jiang et al., 2024 ), Llama-2 70B (Touvron et al., 2023) , and Llama-3 70B (AI@Meta, 2024) LLMs were run on two NVidia A100 GPUs with 80 GB of memory. The experiments using the Llama-2 7B (Touvron et al., 2023) and Llama-3 8B (AI@Meta, 2024) LLMs were run on one NVidia 3090 GPU with 24 GB of memory. The total compute used in the paper exceeded 600 GPU hours.No training was done in our LLM experiments, we simply input the prompt to the LLM and ran it forward to get a prediction for a particular target point.Processing times varied as a function of:• The GPU used.• The length of the prompt.• The number of target points queried.• The number of tokens required to be generated for a particular target point.• The number of samples taken at each target point.1D Synthetic Data Experiments: • LLM: Mixtral-8×-7B • GPU: 2 × Nvidia A100,We first investigate whether our logit-based method of eliciting distributions (Appendix A.2) match the sampling distribution of the LLM. In order to estimate the true distribution, we obtain 1000 samples from the LLM at each target location, and fit a histogram using the same bins as our logit-based method. Figures H.2 to H.4 show that our method yields a distribution that is visually similar to the one obtained by sampling. Overall, LLMPs tested are robust to the prompt format. The results indicate that two separators are required to achieve the best performance. One to separate the x and y values within a pair and another to separate the x, y pairs. The , format uses a comma to separate within a pair and nothing to separate the pairs and it has the worst results. The x y format uses letter prefixes to separate values and pairs with improved metrics. Trading off token efficiency and performance, , \n is the best option as it uses only one comma to delimit x and y and \n to delimit x, y pairs. However, given that some regions use a comma as a decimal place, we use , \n prompt format in our experiments as it comparable performance and only uses one additional space per pair. The ( , ) and x= , y= \n formats are more human readable, but the extra tokens do not improve performance. We consider the effect of three different orderings of the training data D train in the prompt:• Sequential: (x i , y i ), ∈ D train are ordered sequentially from smallest to largest x i , regardless of the location of the target point.• Random: (x i , y i ), ∈ D train are randomly ordered. Distance ordering consistently yields the best results overall. We posit that distance ordering is effective as it provides a hint to the LLM to weigh the contribution of closer points to the current target point to a greater degree. Unless otherwise noted, we use distance ordering for our experiments. In this experiment, we examine the effect of the magnitude and sign of the y-values of the task given to the LLM when no other contextual information is provided. We take the same three synthetic examples but scale the y-values to be in the ranges [0, 1], [-1, 1], [0, 10] and [-1000, 1000] .Figure H.7 shows NLL and MAE for various prompt y-scaling and each LLM. Table H . 4 shows the tabular results. The raw values given to the LLM are scaled meaning the observation noise is scaled accordingly. We have scaled the likelihoods and MAE values to compensate for the difference in range. According to the evaluation metrics we observe that performance degrades with increased range and incorporating negative values also hurts MAE. This is due to the fact that when the range is wider, the LLM must accurately generate more numerical digits and potentially a negative sign when predicting f (x * ). Predictive distributions given by the Mixtral-8×7B LLM on scaled Linear + Cos with 75 observations. This example exhibited one of the largest variation in metrics as a result of scaling. Despite this, all predictive distributions look reasonable..9 shows how MAE varies with LLM top-p and temperature. Table H .5 shows the tabular version of the results.Surprisingly, all LLM's are insensitive to temperature and top-p with respect to MAE.Though not evident from these MAE results, we sometimes observed that using a top-p of 1.0 can result in some extreme values in samples. However, we consider temperature = 1.0, and top-p = 1.0 closest to the default distribution given by the LLM. Since it had competitive performance with the other options, we use these settings to compute log-likelihoods in our experiments which allows us to examine the default characteristics of the LLM's predictive distribution. To verify that LLMPs are able to produce non-Gaussian, multimodal predictive distributions we sampled training data from the following synthetic, bimodal generative distribution: Black box optimization involves minimizing or maximizing a function where there is only access to the output of a function for a specified input. It is often used to optimize functions that are expensive to evaluate and the goal is to find the minimum or maximum value with the fewest number of calls to the function (often referred to as trials). To acquire the location of the next point to observe, we sample the LLM using Thompson sampling (Thompson, 1933; 1935) . Details are in Algorithm 3. We benchmark the ability of an LLM to perform black box maximization on six commonly used functions implemented in (Hoffman & Shahriari, 2015) , including Gramacy (Gramacy & Lee, 2012) , Branin, Bohachevsky, Goldstein, and Hartmann3. We compare our results using Llama-2-7B to Optuna (Akiba et al., 2019) , a commercial hyperparameter optimization framework. We run both methods for 100 trials and record the trial at which the the best approximation to the maximum occurs. The results are shown in Table I .7. In all cases, we obtain as good or better approximation to the true maximum value in a fewer number of trials. Note that Optuna will perform 100 trials in a few seconds while the LLM approach can take up to 2 Nvidia A100 GPU hours. However, the results show that the log likelihood of LLMPs is capable of accurately portraying regression uncertainty. For each location, we use the Mixtral-8×7B A-LLMP to forecast 32 months of average precipitation values given the previous four month observations taken from a random historical three-year period between 1913-2017 (conditional on data availability).It is then provided with 1-12 examples of random three year periods of historical values from the same location in-context. An example prompts for 0, 1 (1976-1978) and 2 (1976-1978, 1949-1951) The models we evaluate are trained with tokenization schemes that tokenize each digit in a number separately.Gruver et al. (2023) include a space between each digit for tokenizers that do not tokenize each digit separately. 
 
 Appendix:  


Decision: The paper studies the use of LLMs and in-context learning for regression problems. The authors look into prompt engineering exploiting both numerical examples and their textual explanation for eliciting coherent predictive distributions. They evaluate a variety of regression tasks such as for-casting and time series prediction, multi-dimensional regression, and more. On many 1D synthetic tasks, LLMP demonstrates matching or better performance when compared to GPs with a RBF kernel. Their approach outperforms the most related baseline of LLMTime in terms of both NLL and MAE in predictions on a Weather time series benchmark. Their experiments also outperform a GP (with RBF kernel) on a multidimensional input/output task of simultaneous temperature, rainfall and wind speed regression task. Reviewers found the problem statement is novel and interesting, and the results are ample and promising. The paper can be improved by making experimental part clearer, and add analysis of runtime

Reviews: review: paper_summary: paper_summary: This paper investigates the regression problem in large language models via in-context learning. They evaluate a variety of regression tasks such as for-casting and time series prediction, multi-dimensional regression, and more. They look into prompt engineering exploiting both numerical examples and their textual explanation for eliciting coherent predictive distributions.
strength_weakness: strength_weakness: -some parts of the paper were unclear to me, especially the experimetnal part. 
-Though the related works are covered well in terms of citation the actual resutls are not compared to the previous ones. see below my questions.
questions: questions: If I correctly understand the experimetnal part reports resutls on both training and only prompting. In this case make these two paradigms more clear in the organization of the experimental results. 

The paper that is cited [21] : their experiments should be comparable to this work, why the results are not compared? while you apply more technical variations here what is the main message and your new contribution compared to them? can you put their results and yours side by side and compare them? 


[21] From words to numbers ....
limitations: limitations: NA
paper_summary: paper_summary: This paper proposes using LLMs to model joint distributions over numerical outputs while conditioning on potentially multiple covariates per data point. This is achieved through tokenizing a series of input and output pairs and then decoding corresponding outputs for another input. Outputs for numerical values are achieved by treating the number as a sequence of digits, each of which are represented with a token. The method is empirically validated with additional experiments showcasing capabilities of conditioning on additional contextual information, such as text describing the process.
strength_weakness: strength_weakness: The main weakness in the approach that I potentially see is the runtime. From what I could tell, there is no runtime results in either the main paper or the appendix. For processing a sequence of input-output pairs, what is the general runtime exhibited, and how does it change as the sequence grows?
questions: questions: See weaknesses.
limitations: limitations: The authors extensively discussed the limitations and societal impact of the model.
paper_summary: paper_summary: This paper focuses on the setting where the goal is to use LLMs to make numerical predictions, as these models can be grounded by the provision of side-information (given in text) as well as other information that may be learned during their pretraining. 

They propose LLM Processes, an approach to apply LLMs to numerical tasks, such as density estimation and multivariate time series tasks. In computing their densities over continuous valued numerical inputs, they employ binning as is done in LLMTime, although they do not rescale to remove decimal points. They also add an additional terminal special token (<t>).
In selecting their input prompt formats, they use separators between each x, y and (x,y) pair, order points by distance to the current point, scale the y values to be closer to [0, 1] and not incorporate -1 values. 

On many 1D synthetic tasks, LLMP demonstrates matching or better performance when compared to GPs with a RBF kernel. Their approach outperforms the most related baseline of LLMTime in terms of both NLL and MAE in predictions on a Weather time series benchmark. Their experiments also outperform a GP (with RBF kernel) on a multidimensional input/output task of simultaneous temperature, rainfall and wind speed regression task.

They also provide experiments looking at the ability of LLMP to handle textual information (alongside numerical inputs); they provide descriptions of the numerical features, which cannot be used in baseline forecasting approaches that do not understand text.
strength_weakness: strength_weakness: Slightly unclear in the evaluation; please see questions below.
questions: questions: 1. It’s not immediately clear to me what the takeaways from Figure 9 are. What are the axes for each of the subplots? 
2. My current understanding is that incorporating some additional textual information should meaningfully change the predictive distribution, which seems to occur. Is there any way to measure how correct this change in distribution is?
3. For instance, in Figure 9d, it seems that the median from 5 random samples seems to be roughly constant (around 2.5-3), which does not seem to match the ground truth values in 9f. Is this case just a failure of the LLM Process?
limitations: limitations: Yes, limitations are adequately addressed.

Comments: title: Reviewer Response
comment: Thank you for addressing my question and adding the new resutls. I have raised my score.
Thank you for the response to my question, this does indeed directly communicate the type of information I wanted to see regarding this issue. 

As far as the paper, I do still feel strongly that this is innovative work with a great deal of experimental results backing the empirical findings. As such, I maintain my original score.
Thank you for the clarifications about the evaluation and axes meaning in the figures. I'll maintain my stance of a weak accept; I think there are useful comparisons to prior work and ablations that illuminate some of both the abilities and failures of the application of LLMs to numerical prediction.