LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
Authors: James Requeima
John F Bronskill
Dami Choi
Richard E. Turner
David Duvenaud
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Probabilistic Regression
In-context Learning
Primary area: probabilistic_methods
TL;DR: We use LLMs to build a regression model that can process numerical data and make probabilistic predictions at arbitrary locations, guided by natural language text which describes a user's prior knowledge.

Abstract: Machine learning practitioners often face significant challenges in formally integrating their prior knowledge and beliefs into predictive models, limiting the potential for nuanced and context-aware analyses. Moreover, the expertise needed to integrate this prior knowledge into probabilistic modeling typically limits the application of these models to specialists. Our goal is to build a regression model that can process numerical data and make probabilistic predictions at arbitrary locations, guided by natural language text which describes a user's prior knowledge. Large Language Models (LLMs) provide a useful starting point for designing such a tool since they 1) provide an interface where users can incorporate expert insights in natural language and 2) provide an opportunity for leveraging latent problem-relevant knowledge encoded in LLMs that users may not have themselves. We start by exploring strategies for eliciting explicit, coherent numerical predictive distributions from LLMs. We examine these joint predictive distributions, which we call LLM Processes, over arbitrarily-many quantities in settings such as forecasting, multi-dimensional regression, black-box optimization, and image modeling. We investigate the practical details of prompting to elicit coherent predictive distributions, and demonstrate their effectiveness at regression. Finally, we demonstrate the ability to usefully incorporate text into numerical predictions, improving predictive performance and giving quantitative structure that reflects qualitative descriptions. This lets us begin to explore the rich, grounded hypothesis space that LLMs implicitly encode.

LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
James Requeima" University of Toronto Vector Institute requeimaBcs. toronto. .du
John Bronskill University of Cambridge jfb540cam. ac
Dami Choi
University of Toronto choidami@cs toronto. edu
Richard E. Turner University of Cambridge The Alan Turing Institute ret260cam. ac .k
David Duvenaud
University Toronto Vector Institute duvenaud0cs. toronto. edu
Abstract
Machine learning practitioners often face significant challenges in formally in- tegrating their prior knowledge and beliefs into predictive models, limiting the potential for nuanced and context-aware analyses. Moreover, the expertise needed to integrate this prior knowledge into probabilistic modeling typically limits the application of these models specialists. Our goal is to build a regression model that can process numerical data and make probabilistic predictions at arbitrary locations, guided by natural language text which describes a user's prior knowl- edge. Large Language Models (LLMs) provide a useful starting point for designing such tool since they 1) provide an interface where users can incorporate expert insights in natural language and 2) provide an opportunity for leveraging latent problem-relee relevant knowledge encoded in LLMSs that users may not have themselves We start by exploring strategies for eliciting explicit, coherent numerical predic- tive distributions from LLMs. We examine these joint predictive distributions, which we call LLM Processes, over arbitrarily-man quantities in settings such as forecasting, calli-ddmeenionall regression, black- -ox optimization, and image modeling. We investigate the practical details of prompting to elicit coherent predictive distributions, and demonstrate their effectiveness at regression. Finally, demonstrate the ability to usefully incorporate text into numerical predictions, improving predictive performance and giving quantitative structure that reflects qualitative descriptions. This lets us begin to explore the rich, grounded hypothesis space that LLMs implicitly encode.
1 Introduction
Incorporating prior knowledge into predictive models is highly challenging which restrict the scope for detailed, context-sensitive analysis. In addition, the skill required to incorporate this prior knowledge into probabilistic modelling can restrict the of these models experts. In this work, our objective is to develop probabilistic prediction model that facilitates user interaction through straightforward, natural language. For this purpose, explore strategies for eliciting explicit, coherent numerical predictive distributions from LLMs.
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Data
Text Information
Data
Text Information "...the company goes out of business."
Data
Text Information
...a financial time series."
...on day 30."
LLMP
LLMP
LLMP
Figure 1: Predictive distributions from an LLMP conditioned on both data and text information. The -enth-percentiles from 50 samples are visualized in faded blue and the median is presented in dark blue with five random samples shown in various colours.
Why go to so much effort to elicit predictions from slow, expensive, and sometimes inconsistent model like an LLM? We expect their hypothesis class to be both rich, and grounded in exactly the kinds of high-level side information that we currently struggle to communicate to our numerical models. For instance, knowing that prices rarely go below zero, that certain kinds of sensors can saturate at particular values, or that trends almost always eventually level off, easy to express in natural language, but not straightforward to incorporate into a model without getting lost in difficult-to- to-specify details about aspects of the domain that aren't 't well understood. To summarize, we want to develop such a model because it would allow users to 1) provide prior, potentially expert, information the model about the problem setting in plain-language rather than attempting capture this information in closed form priors (e.g. Gaussian Process kernels) and 2) it would allow users to access reeleantrelevan latent knowledge encoded in LLMs that users may not have themselves.
LLMs have recently been shown to be able to condition on the particular task being solved, leveraging contextual information to make better predictions or decisions M. They have also been shown to competitively predict time series based only on a text tokenization of numerical data [2]. In this work, further push in both these directions: 1) using LLMs for numerical prediction tasks going beyond one-dimensional time series forecasting to multi celti-dimensional regression and density estimation and 2) exploring the ability of these models to condition on both numerical data and rich, unstructured text to improve these predictions. In this paper we make the following contributions:
We define LLM Processes (LLMPs) using methods we develop for eliciting numerical predic- tive distributions from LLMs LLMPs go beyond one-dimensional time series forecasting to multi-dimensional regression and density estimation. We propose two approaches for defining this joint predictive distribution over collection of query points and evaluate their compatibility in principle with the consistency axioms necessary to specify a valid statistical process.
We develop effective prompting practices for eliciting joint numerical predictions. We inves- tigate various methods for conditioning LLMs on numerical data, including prompt formatting, ordering, and scaling. We characterize which schemes perform best on a set of synthetic tasks.
We show that LLMPs are competitive and flexible regressors even messy data. Through an extensive set of synthetic and real world experiments, including image reconstruction and black-box function optimization, we evaluate the shot regression and forecasting performance of LLMPs. We demonstrate that LLMPs have well well-calibrated uncertainty and are competitive with Gaussian Processes (GPs), LLMTime (2). and Optuna 33. We show that LLMPs use in in-context learning to automatically leverage information from related datasets, can easily handle missing datapoints, perform image reconstruction, and output multimodal predictive distributions.
Lastly, we demonstrate the ability to usefully incorporate problem-relevant information pro- vided through unstructured text into numerical predictions, visualized in Figure [ resulting in quantitative structure that reflects qualitative descriptions. Other additions such as labelling features using text and specifying units allow LLMPss to make use of usually-ignored side information.
Source code available at: https://ggitii http///roouuuee////nnoooessaaa

---

y
Y: Problam Related Text
X 2/2
(0): Terminal Token
Independent
samples each target point Y
X2. Y2
X3
"T() 22l
LLM
X %
Xa. Ya
x 1/
7()) 2005)
X
Autoregressive
20
Prompts
Autoregressive: Append previous output to prompt
Medien Uncertainty
"T(%% 21
Observed Point
Locations
"T(tx, ya ("x10))
Figure 2: Sampling from an LLM using either independent marginal or autoregressive sampling.
2
LLM Processes: Defining Stochastic Process That Can Condition on Text
Our goal for this section is to use an LLM to elicit joint predictive distributions over arbitrary sized target sets that can guide and modify using natural language. Formally, given set of input and output observations Dirain = {(T,, 36) )i and some text, T, we would like to elicit the predictive distribution defined by an LLM at collection of targets {(a3,19)) 15) denoted PLLM(yi.... . -i,,..
,N. Diain: T).
Rejection sampling from an LLM allows us to access what we may interpret as the LLM's predictive distribution and gain insights into the model's inductive biases; sampling from the LLM's categorical distribution over text tokens while ignoring non-numerical tokens yields numerical samples from the LLM. The process of sampling from an L.MM depicted in Figure and A Algorithm T Sample prompts are in Appendix C Since an accurate sampling-base empirical distribution incurs a high computational cost, next we define an approach to elicit continuous likelihoods from an L.L..
Continuous Marginal Likelihoods From an LLM. We approximate continuous density over our target values by discretizing the space using bins with arbitrarily fine precision, similar to the method used in Gruver et al. 22) Crucially, this hierarchical approach allows us to compute the probability of a bin with width 10-" For example, if
then Pr(y E [1.0, 1.1)} p(1)p((11)) (ppp(111p(00)) (O]1) because *1.0' is prefix for all [1.0, 1.1). We can convert probability mass to probability density by assuming a uniform distribution within each bin, and dividing the mass by the bin width. A
visualization of this construction is in Figures G.2 to G.4
we do not rescale the values remove decimal places. We hypothesize that such scaling Unlike 2
removes prior information communicated to the LLM via the scale of the problem. We examine the effect of scaling values Section 3 We also differ from by including terminal token after every value in our prompt - for example, given terminal token (t), we represent 12 as 12(t). Including terminal token prevents numbers of varying orders of magnitude to share the same prefix i.e. p(1 p(1)p (2)((())))(()))) (((o))(((((())))(( 12) no longer includes the probability of numbers in [120, 130), [1200, 1300), etc.
Note that this approach does not guarantee that P(12(t)) yields the mass assigned by the LLM to values in the bin [12, 13) but we empirically observed that our predictive distribution closely matches the sampling distribution our satisfaction. See Section G.T for more details and comparison.
Defining an LLM Process. Thus far we have established a procedure defining the predictive distribution at single target location, PLLM x; Duains T'). We now outline two methods which we call independent marginal (I-LLMP) and autoregressive (A-LLMP) predictions, for defining the joint predictive distribution over collection of target points:
N
PI-LMM((i .vi.. .... Viv 12i.... .... NN Dirainy T)
II
(V 2",, Durain: T)
(1)
n=1
NN II
PA v.v 2i..... x.y, D Draain T)
P vi..... Vn- 1n-1:900 21. ....... Th D Durain T)
(2)
We note that Equation (I) satisfies the Kolmogorov Extension Theorem 44] therefore defining valid stochastic process (see Appendix A.3)). However, it assumes conditional independence given the training set and model weights and the stochastistity represented by the model is via independent
marginals. Equation 2 takes inspiration from the autoregressive structure of the L.MS predictive distribution and should yield much richer predictive distributions we are now able to model
3

---

dependencies between output variables. However, this definition is no longer guaranteed to give us valid stochastic process as the predictive distribution is now target order dependent and will likely fail the Kolmogorov exchangability condition. We investigate both of these questions in Section 3
Connection to Neural processes Neural Processes (NPs) are class of meta-learning models parametrized by neural networks and trained to learn a map from training (context) sets to predictive distributions, pe (yi...... V/N i .i...... NN Dirin).. D The definitions in Equations] and 2 take inspira- tion from the joint distributions defined by Conditional NPs (5] as independent marginals conditioned on the training/context set and Autoregressive NPs 6 utilizing the chain rule of probability, respec- tively. Through this lens. LLMPs can be viewed as examples of NPs. However, NPs are directly trained output this predictive distribution where as LLMPss are repurposing pretrained LLMs.
Multi-dimensional Density Estimation and Handling Missing Data. We highlight that, through the flexibility of the LLM prompt, we do not have draw distinction between which variables, or variable dimensions are to be modelled or conditioned and can easily handle missing values. Suppose we have a collection of variables {-1,.... In and {y 3m 3 (or more), some subset of which we would like to regress on (including I and y-values) and the remainder we wish to condition on. To do so using an LLMP, we simply construct the training prompt such that the variables we would like to regress on occur at the end of the prompt and are blank (generated) when sampling from the LLMP. If any values are missing they can simply be removed from the prompt.
3 LLMP Configuration
Experiment Details. In all of the experiments in Sections to 5 we use six different open source LLMs: Mixtral 8 7B, Mixtral-8 x IB-Intrucct 77.. Llama-2 7B, Llama-2 70B (8) Llama-3 8B, and Llama-3 70B 99. Note that we never modify the LLM parameters via training or fine-tuning, we use only prompting. Our primary metrics are negative log probabilities (NLL) of the model evaluated at the true function values fx²") averaged over the target locations and Mean Absolute Error (MAE) between the predictive median and the true function value. Unless otherwise stated, samples from the L.MM at each target location x and compute the median and the 95% confidence interval of the sample distribution. Details of the datasets given in Appendix D Since the LLMs used in our experiments have undisclosed training sets, we address the steps taken to mitigate the issue of data-lcakage in Appendix B Additional implementation details and processing times are in Appendix F
use 50
Promet Formstting
Trshing Dara Ordering
Prompt y Scaling
NILL
1422
0.300
Figure 3: NLL and MAE for various prompt formats ordered from the most to least token efficient
(left), training data orderings (middle), and prompt y-scaling (right) using the Mixtral-8 7B LLM. The height of each bar is the mean of 10 random seeds that determine the training point locations. The vertical black lines indicate the standard error. In the Prompt Formatting legend (left), the two '_' characters indicate the positions of the and y values and 'n represents new line terminal token.
Prompt Engineering. We perform set of experiments for determining the best L.MP prompt configuration. We use the Sigmoid, Quadratic, and Linear+Cosine functions with 10, 20 and 75 training points, respectively (see Appendix D.II with I-LLMP using the Mixtral-8x7B LLM.
Prompt Formatting Two separators are required to achieve the best performance. One to separate the x: and y values within a pair and another to separate the r, pairs. Figure 3 (left) demonstrates that that__.nn is the best option in terms of performance and token efficiency.
Prompt Ordering Figure (middle) shows that ordering the training points by distance to the current target point is best, outperforming both random and sequential ordering. We posit that ordering by distance provides a hint to the L.M to weigh the contribution of closer training points to the current target point to greater degree.

---

A-UMP KLLMP Rriiing Cata Ordering
Sigmeid-10
Quadralin200
Ln-
20
13-
poy:: Lngits
15
L78
L708
Ma
L700
Goarate 2n
Bandom-H-Le
Randum A4LL90
Indepandert
Sansd
Distance HUMP
Distance AALLMP
Ground Tun
Sandom Test
Figure 4: Autoregressive Experiments. Left: NLL and MAE for A-LLMP and I-LLMP using different prompt orderings using the Mixtral-8x7B LLM. The height of each bar is the mean of random seeds that determine the training point locations. The black lines indicate the standard error. Center: log-liilihood results of using various test set orderings with Llama-2-7B. Llama-2-70B and Mixtral-8x A A-LLMP. The orange X indicates I-LLMP, the purple circles used distance ordered test points, and the blue whiskers are the mean and standard error of 10 randomly sampled test orderings. The red dashed line shows the log-likelihood of the test set under the generative process. Right: Heatmap visualization of the Llama-3-70B A-LLMP predictive distribution conditioned on data from bimodal generative process. Black dots are training points.
Prompt J-Scaling Figure 3 (right) shows that performance degrades as the range of the compo- nents of the training points increases and when incorporating negative values. This due the fact that when the range is wider, the LLM must accurately generate more numerical digits and potentially negative sign when predicting I [xx).
top-p and Temperature Figure G.9 shows that performance is surprisingly insensitive to varying the LLM nucleus sampling parameter top-p [10 and LLM softmax temperature.
Autoregressive Independent Marginal Predictions. Here examine two questions: first, does the autoregressive defininiton of the joint predictive likelihood (A-LLMP) in Equation (2) improve performance versus the independent marginal definition of Equation ( (I-LLMP). Second, "how close" is A-LLMP to a stochastic process in terms of performance variability across query orderings.
We first look at log- liglliiclshoods and MAE for A-LLMP and I-LLMP using the random and distance training point orderings discussed earlier. Results can be seen in Figure (left). Similar to our findings earlier, ordering the training values according to distance to target has a large effect, improving performance for both I-LLMP and A-LLMP. Unsurprisingly, the richer joint distribution given by A-LLMP gives us better predictive performance.
We next examine the variability performance of A-LLMP when different autoregressive target orderings are used to get a sense of how far our method is from stochastic process (which would be permutation invariant in the target points). The results of using ten sets of randomly ordered target points compared to I-L. and the ground truth log-likelihood of the test sample under the generative distribution are presented in Figure 4 (center). Note that the training data is distance sorted in all cases. We also present the result when ordering target points according to distance to the closest training point, from smallest to largest. We make three key observations: first, log-likelihood performance of all A-LLMP orderings is better than I-LLMP. Second, the variance of random orderings is small on the scale of the log-likelihood of the generative model. And third, distance ordering the targets gives better at least competitive performance with random ordering. These results present practitioners a choice: do you care more about using a valid statistical process or obtaining good predictive performance? If it is the latter, you would be better served using A-LLMP.
4 Evaluating LLMP Performance on Numerical Data
In this section, we evaluate the performance of LLMPs on purely numerical data in a wide variety of settings. Additional details and results for experiments in this section can be found in Appendix H
1D Synthetic Data Experiments. To show that LLMPs are viable regression model with well- calibrated uncertainties, we benchmark in Table O our A-LLMP method against GP on the Function Dataset (Appendix I.II. The GP uses an RBF kernel with optimized length scale and noise. The Mixtral-8 A-LLMP achieves the lowest negative log-likelihoods averaged over 7 function sizes and seeds on 10 out of of the functions and equal or better MAE on 8 of the functions. Visualiza- tions of the predictive distributions and plots of MAE and A-LLMP are shown in Appendix H.I
5

---

Table 1: Mean and standard error of MAE and NLL averaged over the seven training set sizes and seeds each function for Mixtral-8x A-LLMP and a GP with an RBF kernel.
Functio
Metric
Beat
Eap
Gan Ware
Lincar
Lis Cos
Lis Sinc
Ldg
Quadralic
Signaid
Sinc
Sine
Nx Sine
0898263
0.37+885
GP
1.00aa
-1:4530.22
40400.22
4.44
12.641142
MAA NEL.
0,311881
0.410001 -D081000
0.091001
IUSJIUM
0,448:00
0:911004
0.27:0:2
12,46:137
4.761688
-1.5600M
-2.88.1938
-1,001
-2.2048.02
-L.SI08
-33818.84
-1.060L
0.1418.11
3.9010.23
To verify that LLMPs are able to produce -on-Gausiinn, multimodal predictive distributions we sampled training data from synthetic, multimodal generative distribution (experimental details in Appendix H.2). The Llama-3-70B LLMP predictive distribution is visualized in Figure (right)
Comparison to LLMTime. Figure 5 demonstrates that A-LLMP yields superior results in terms of MAE and NLL when compared LLMTime using 1llma-2-77B on forecasting task using the weather dataset (described in Appendix D.2.. Additional plots with missing training data are in Appendix H.3 We posit that A-LLMP betters LLMTime due to the fact that 1) A-LLMP naturally handles irregularly spaced and data whereas L.MTTmme uses only regularly spaced y information requiring imputation with NaN values where data is missing; and 2) A-LLMP performs no scaling on y values in contrast to LLMTime that scales data to climinate the use of decimals and normalize the range of the data and as a result removes information that the LLM can potentially leverage.
pants
ALLMP Own
ALUPP Veatin
LUMITNI
LLMIene Medhan
W.ffres
ALUMA Toriig Navis-5o Hot-3
ALL-O
LLV Tralning Norts Mat-1 RLL-1 829
335-
MunMan' Mack
2,75
225
Figure 5: Comparison of A-LLMP and LLMTime the weather dataset. Left: Plot using all 50 training points. Right: Plot of MAE and NLL versus the amount of training data removed. A-LLMP has lower MAE and NLL and the margin over LLMTime increases more training data is removed.
Comparison to From Words to Numbers. We compare our I-LLMP method to the approach in [ on their Original #1 dataset. The experimental set-up is follows: There are 100 trials with each trial consisting of 50 training points and a single target point. The training and target points for each trial are randomly generated using the function described in [ We use the code from their paper to generate the data and evaluate their approach and compare it to ours using identical numerical data. We the Llama-2-77 2-BB LLM for both methods to ensure fair comparison. I-LLMP achieved lower MAE on 78 of the 100 trials when compared to their method. When the errors are averaged over the 100 trials, the 1-L.MM average error was 0.836 and theirs was 3.737. These results indicate that our I.MPP approach is clearly superior. This is due to the facts that (i) we sort the training points according distance to the eurrent target point when creating the prompt whereas they do not, and (ii) we form a distributional estimate for the predicted point and then take the median sample value as the best estimate, whereas they generate single point estimate.
In the next three experiments we showcase the ability of LLMPs to handle multi-dimensional data.
Image Reconstruction As 2-dimensional input experiment, Figure 6 shows reconstruction results from images drawn from the Fashion- MNIST dataset [12] We convert pixel data into prompt data points by forming a series of (row, column, pixel value) tuples. Additional results and details are in Appendix H.4 Using 20% train pixels, the basic form is captured and at 50%, the reconstruction accurate despite the sharp pixel intensity transitions.
True
Train
20% Result
Train
Result
20%
Train 50%
Figure 6: Fashion-MNIST Mixtral image reconstruction results. The blue pixels indicate unobserved.

---

Black-Box Function Optimization Black-box optimization involves minimizing maximizing function where there is only access to the output of function for specified input. We benchmark the ability of LLMPs to perform maximization on six commonly used multi multi-dimensional functions. We compare our results using Llama-2-7B to Optuna [3, commercial hyperparameter optimization framework. Results and implementation details are in Appendix H.5 In all cases, LLMPs obtain as good or better approximation to the true maximum value a fewer number of trials.
Simultaneous Temperature, Rainfall, and Wind Speed Regression To examine how well an LLMP can model multi-dimensione outputs, we compare LLMP regression
multi-output GP on the weather dataset described in Appendix D.2 Figure shows the results for the Llama- 3-BB LLM (top) and 3 output RBF kernel GP with trained hyperparameters (bottom). The LLM similar and in most cases better than the GP in terms of MAE and NLL.
We Cerlarec
Moze
U-ma-3-78 MAE-1 NiL-S
(nme)
Usme 3-78 MAC
MLL 203
Lime- 7B MAE NLL-5
Pretonatoo (rrm) Tree pitatin
Wind Spenddpeeddn Speniiig Wand Speed
MAE-1-424
MAD 522 NJL-8
since December
Figure 7: Results for simultaneously predicting temperature, precipitation, and wind speed using the Llama-3-77 3-7B LLM (top) and a 3 output RBF kernel GP with trained hyperparameters (bottom).
In-context Learning Using Related Data Examples. In this experiment, investigate LLMPS' ability to learn from similar examples in-context to predict average monthly precipitation across 13 Canadian locations
13] one from each province and territory. For each location, we use the Mixtral-8 8 x 7B A-LLMP to forecast 32 months of average precipitation values given the previous four month observations taken from a random historical three-year period between 1913-2017 (conditional on data availability). It is then provided with 1-12 examples of random three year periods of historical values from the same location in-context. Results shown in Figure g and experimental details in Appendix H.6 Conditioning the LLMP on historical examples improves performance saturating after 4 years, and degrading slightly thereafter. Generally, the L.MPP is able use the examples to pick up on seasonal trends from history. We note that some locations do not have obvious or strong seasonal patterns but examples still help performance these cases (see Appendix (.6).
Conditioning LLMPs on Textual Information
One of the most exciting directions of LLMPs is the potential to incorporate prior information about problems via text. Now that we can examine functional predictive distributions of LLMs, we can begin to explore their rich prior over functions by conditioning on both text and numerical data. In this section we present two experiments with details and additional experiments presented Appendix [
Scenario-condition Predictions. In this experiment, we examine the influence of text providing information about various synthetic problem settings on the predictive distribution of an LLMPs. In
MAF 30.66531 NILL 2.49844
MAF 25. 05.11906 NIL 1.98546
MAF 12.25406 NIL 1 1.45454
Treppiatoo leet
pncppaation.oo leet
ineed
3.0
22
15
Northi
Morthc
Norths
Number of Ecamplee
(a) examples
(b) example
(c) examples
(d) Examples vs NLL
Figure 8: (Left three plots) Visualizations of the predictions given by the Mixtral-8 7B LLMP for Ranfurly, Alberta. Blue and black circles are training and test points, respectively. Red circles are median predictions and shaded areas indicate tenth meth-eercentiles over 30 samples. (Right) NLL vs number examples. Error bars show standard error over 13 locations.
7

---

(a) No prompt
(b) Montreal daily temp. in Jan.
(c) Montreal daily temp. in May
Cen Orage
Accupt
23
(d) Monthly precip. in Singapore
(c) Monthly precip. San Diego
(f) Actual monthly averages
Figure 9: a)-e) predictive distributions from A-LLMP using Llama-3-70B under various scenario prompts. Black points are two training points given to the LLM process, the same values for each scenario. The tenth-percentiles from 50 samples presented in dark blue with five random samples shown in various colours. Figure f) shows the actual average monthly rainfall for Singapore from 1991-2020 [4] and San Diego from 2000-2024 15
visualized in faded blue and the median is
all of the following examples, we provide the same two synthetic training points to the LLMP but change the prompting text that comes before the training data. We then use A-LLMP with Llama-3- 70B to forecast trajectories 50 steps ahead. We begin by examining the predictive distribution with no prompt (Figure Dap. We prompt the LLMP to generate daily temperature measurements in degrees Celsius from Montreal in January [Figure[9b] and May (Figure 9c). and monthly precipitation values from San Diego, CA (Figure odh and Singapore (Figure De). Figure Shows the results of prompting the LLMP to generate (leff) a stock price financial time series (centre) for a company that eventually goes out of business and (right) for company whose price goes zero on day 30.
Indeed, the LLMP modifics the predictive distribution accordingly relative to the no prompt predic- tions. We highlight the following observations: first, for prompts b) and c), the model moves about half of its predictive mass below zero for temperatures beginning in inJa January and above zero for the May temperatures. Second, the LLMP is able to recall actual historical trends for average monthly precipitation for Singapore and San Diego to condition on prompts d) and e). Despite getting the trend correct, we note that the median prediction in d) seems be biased toward the training values and not reflective of the actual monthly median.
Last, for stock price simulations, the model places all of its density positive numbers since it is modelling prices. It is able to produce realistic trajectories and decreases them in expectation when prompted that the company goes out of business. The model is able to condition on the fact that the price goes to zero on day 30 which correctly interprets the meaning of the x- æ-values as days starting from O, that the y-axis is the price and the phrase "price goes to zero" corresponds to a y-value of O.
Labelling Features Using Text. In the following example, we examine the performance of Mixtral-8x7B Instruct I-LLMP on predicting American housing prices. The dataset [6] contains 39980 housing prices and various variables around housing and demographies for the top 50 American cities by population. Note that this dataset was generated on 12/09/2023, however it contains data from the 2020 US Census and the 2022 American Community Survey (ACS) so we cannot guarantee that models did not see data within this dataset during training.
For each prediction task, we show the I-LLMP 10 randomly selected training examples from the dataset and predict on 20 randomly selected test examples. In the prompt, before the numerical value (price) we provide a string which encodes the datapoint index/features that the model can use. For our first experiment we examine the behaviour of the LLMP when more features are added to the prompt. We experiment with five ways of indexing the training and test points; For case (1), we provide latitude and longitude of the house as numerical values (eg. 32.74831, -97.21828) converted to strings similar to our method in previous experiments. For the remaining 4 cases, we provide additional labeled features, adding more features for each case with the prompt for case (5) containing
8

---

22
Toe Price
4.2
6.m
3.0-
Piice (UST)
St faurane
5,50
2213 House Sxample
Figure 10: Results of a Mixtral-8x7B Instruct I-LLMP predicting US housing prices. Left: Predictions for 10 randomly selected houses using index style 1) and 5). Xs are mean predictions using 30 samples from the LLMP and error bars indicate 2 standard deviations. Centre and right: Average MAE and NLL performance of the LLMP over 10 experiments with error bars representing the standard error for experiments from Section
all labelled features, illustrated with the following example: (2) Location: Fort Worth, Texas, Latitude: 32.74831, Longitude: -97.21828, (3) Zip Code: 761 12, Median Household Income: 71452.0, (4) Zip Code Population: 42404 people, Zip Code Density: 1445.0 people per square mile, (5) Living Space: 1620 square feet, Number of Bedrooms: 3, Number of Bathrooms: 2.
This procedure is repeated 10 times to compute statistics. Results presented in Figure I0 (left, centre). Note that the LLMP is able to take advantage of the additional features provided to improve predictive performance. To see examine the effect of adding text labels to the features, we ran another set of experiments on 10 new random datasets providing the LLMP with either labeled or unlabelled numerical features. The following example feature strings: (i) "30.45738, -97.75516" (ii) "Location: Austin, Texas, Latitude: 30.45738, Longitude: -97.75516" (iii) "30,45738,
-97.75516, 78729, 107830.0, 30907, 1216.1, 1349, 3" (iv) "Location: Austin, Texas, Latitude:
30.45738, Longitude: -97.75516, Zip Code: 78729, Median Household Income: 107830.0, Zip
Code Population: 30907 people, Zip Code Density: 1216.1 people per square mile, Living Space: 1349 square feet, Number of Bedrooms: 3, Number of Bathrooms: 2". Results of this experiment are presented in Figure (right). Note that the LLMP not able to use the raw feature values to improve performance from only 10 training examples, but is able to do so with labelled features suggesting that LLM able to utilize the latent relationship between the feature and the price once the feature is identified. We found that the Mixtral-8 x 7B Instruct model had the best performance on this task and was able utilize text information better (results for other models in Appendix (.2))
Related Work
In this section, we discuss work related to eliciting distributions from LLMs including forecasting, regression, in-context learning, and nearal processes among others.
LLM Forecasting The most closely related work to ours is LLMTime 2 LLMTime is capable of zero-shot extrapolation of one-dimensional time series data at a level comparable to trained purpose- built approaches. In addition, they develop a method for eliciting marginal probability distribution functions from L.M posteriors over functions, which we build on. They also begin to investigate the effect of conditioning on text. In contrast, we focus on (i) interpolation with multi- multi-dimensionll inputs and outputs; (ii) eliciting joint distributions over functions, not just marginals; and (iii) exploring the ability of models to condition simultaneously on both numerical data and text. More recently, TimesFM 117. foundation model for one-dimensional zero-shot times series forecasting was introduced. However, TimesFM does not support interpolation or higher dimensional data and does not consider distributions. PromptCast [18] performs zero-shot time series forecasting by combining numerical data and text in question answer format. Our approach for combining problem specific text along with numerical data differs in that it handles both interpolation and extrapolation and does not rely few-shot classification on tabular data that compares favorably to standard ML approaches.
dnesstion-asswrr format. Hegselmann et al. [19 utilize LLMs to do zero-shot and
LLM Regression Pesut 20 do some initial investigations into the use of LLMs as regressors on ID synthetic functions. Our work greatly expands on these early investigations. Vacareanu et al. [ is concurrent work that shows that L.Mss are capable linear and non-linear regressors. However, their work does not condition on any textual information, compute log probabilities, compare to Gaussian Processes, investigate the effect of prompt formatting, or employ auto-regressive sampling.

---

In-context learning (ICL) in LLMs Xie et al. [21] point out that ICL can be seen as being equivalent to Bayesian inference latent variable model. More recently, [22 explain in-context learning in LLMs as kernel regression. Garg et al. [3 train transformers to do in-context learning on various function classes including linear (up to 50 dimensions), decision trees, and two-layer ReI.U networks. Coda-Forno et al. (24) demonstrate that LLMs are capable of meta-in-context learning and that performance on I-D linear regression and two-armed bandit tasks improves with multiple examples. TabPFN [25 is a trained transformer that is able to do tabular classification given in-context examples.
LLM Hyperparameter Optimization Zhang et al. 26 and Liu et al. use LLMs to perform hyperparameter optimization, showing that LLMs can condition on mixture of textual data as numerical observations effectively optimize hyperparameters in machine learning models.
Eliciting priors from LLMs Binz and Schulz [88 fine-tune LLMs on data from psychological experiments achieve accurate representations of human behavior. Choi et al. [ show how using an L.MM to assess the importance of features or the causal relationship between variables that can improve performance on tasks. Lipkin et al. [29] find that LLMs can derive human-like distributions over the interpretations of complex pragmatic utterances.
Eliciting distributions from humans Schulz et al. [30] look at compositional inductive biases in function learning, showing humans have compositional structure in their priors on functions. [31 catalogue standard strategies for eliciting distributions from expert humans.
Neural processes Neural Processes are a class of meta-learning models trained to learn a map from training (context) sets to predictive distributions, (o .i,...., Ii.: Darin). These models are parameterized using neural network and there have been various proposals for different architectures using attention [32) transformers (33] Gaussian Process output layers [34], and
diffusion models [35]. The definitions the joint distributions in equations [ and 2 take inspiration from the joint distributions defined by Conditional Neural Processes (5) as independent marginals conditioned on the training/context set and Autoregressive Neural Processes [6) utilizing the chain rule of probability, respectively. Through this lens, LLMPs be viewed examples of Neural Processes. LLMPs differ from standard NPs in two main ways: (i) Training objective: Neural Processes meta-trained using maximum likelihood to optimize p(y" Duain) directly. LLMPs have a very indirect training procedure they are trained to be language models i.e. autoregressive token predictors. One of the contributions of this paper is the demonstration that, despite this, they can perform zero-shot probabilistic regression. (ii) Architecture: NPs have an output layer that parametrizes the predictive distribution over targets directly. Since LLMPs are repurposing language models for regression, we need to define the mapping from distributions over language tokens to distributions over target variables. We note that LLMs themselves can be viewed as AR-CNPS with a fixed, predefined target ordering.
7 Discussion, Limitations, and Societal Impact
Below we discuss our findings, the limitations and societal impact of the work presented. Further discussion on these issues can be found in Appendix !
Discussion We defined L.MPss for eliciting numerical predictive distributions from LLMs and when used as a zero-shot muti- muti-dimensional regression model are competitive with GPs. Excitingly, we demonstrated the ability to condition on text to improve predictions and probe the LLMs' hypothesis space. An interesting extension would be to condition on other modalities in addition to text.
Linitations Along with the flexibility of LLMs, LLMP's inherit their drawbacks. Maximum context sizes limit the size of tasks we can apply this method to and the amount of textual information we can condition LLMPss are also significantly more computationally expensive compared to Gaussian Processes and standard regression methods. All of experiments were performed on readily available open source L.MMs that are smaller and generally less capable compared proprietary L.Mss
Societal Impact Our work has demonstrated new and useful zero-shot approach for generating probabilistic predictions using plain language augment numerical data. It has the potential to allow practitioners from fields such as medical research and climate modelling to more easily access probabilistic modelling and machine learning. Like all machine learning technology, there is potential for abuse, and possible consequences from incorrect predictions made with LLMPs. Also, we do not know the biases in the underlying LLMs used and what effect they may have LLMPs output.
10

---

Acknowledgments and Disclosure of Funding
James Requeima and David Duvenaud acknowledge funding from the Data Sciences Institute at the University of Toronto and the Vector Institute. Dami Choi was supported by the Open Phil AI Fellowship. John Bronskill is supported by EPSRC grant EP/T0053861L, Richard E. Turner is supported by Google, Amazon, ARM, Improbable, EPSRC grant EP/T005386/1, and the EPSRC Probabilistic AI Hub (ProbAI, EP/Y028783/1).
We thank Anna Vaughan for help with the weather datasets and discussions. We also thank Will Teb- butt, Matthew Ashman, Stratis Markou, and Aristeidis Panos for helpful comments and suggestions.
References
[1] Kristy Choi, Chris Cundy, Sanjari Srivastava, and Stefano Ermon. Lmpriors: Pre-trained language models as task-specific priors. arXiv preprint 1oXiv:22001/53300 2022.
[2] Nate Gruver, Mare Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:23100078820, 2023.
[3] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference knowledge discovery & data mining, pages 2623-2631, 2019.
[4] Bernt Oksendal. Stochastic differential equations: introduction with applications. Springer Science Business Media, 2013.
[5] Marta Garelo,, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference on machine learning, pages 1704-1713. PMLR, 2018.
[6] Wessel P Bruinsma, Stratis Markou, James Requiema, Andrew YK Foong, Tom R Andersson, Anna Vaughan, Anthony Buonomo, J Scott Hosking, and Richard E Turner. Autoregressive conditional neural processes. arXiv preprint 3rX1v:2303.14468. 2023.
[7] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0440 2024.
[8] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXIv:2307.009288, 2023.
[9] AI@ Meta. Llama 3 model card. 2024. URL httpp::/ggitubbcommmeeee/ -laametttalllalalaamaa 1aa.ellaaaaaammmm 1lama3/ #lob/main/MODD CARD md
[10] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
[11] Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu. From words to numbers: Your large language model is secretly a capable regressor when given in-context examples. arXiv preprint arXiv:2404. 07544, 2024.
[12] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
[13] Environment and Climate Change daily adjusted total precipitation. climate-
Canada. Online,
Monthly
total of
2024.
URL
https
lity/
Accessed:
April 2024, Last updated: 2017-08-09.

---

[14] University of East Anglia Climatic Research Unit. Observed Historical Cli-
mate Data for Singapore. World Bank Climate Knowledge Portal, 2024. URL
worldbank. haaaaaa..rrr/oooor/riiiiiiaooooo
-
Accessed: 2024-05-06.
[15] Climate Data. National Weather Service, 2024. URL httpp::/ wgw.ew/th.rr...........
C1 Climate? ? Accessed: 2024-05-06.
[16] Jeremy Larcher. American house prices, 2023,
URL https: wwwge/.
7162651
[17]
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou, A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310. 100:28880.008,, 2023.
[18] Hao Xue and Flora D Salim. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions Knowledge and Data Engineering, 2023.
[19] Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. Tabllm: Few-shot classification of tabular data with large language models, In International Conference on Artificial Intelligence and Statistics, pages 4549-5581. PMLR, 2023.
[20] Lovre Pesut. Who models the models that model models? an exploration of gpt-3's in-context model fitting ability. URL https://www. alignmentforum, wwt//osss/2RFFFaankkRRAAAXXX// models-the-modele thatddll--aa---mdl---ooeoee modelsss-- -ntexxlooration-.) 2022.
[21] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.0200 2021.
[22] Chi Han, Ziqi Wang, Han Zhao, and Heng Ji. Explaining emergent in-context learning as kernel regression. arXiv preprint arXiv:2305. 12766, 2023.
[23] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30595 2022.
[24] Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matthew Botvinick, Jane X Wang, and Eric Schulz. Meta-in-context learning in large language models. arXiv preprint arXiv:2305.12907, 2023.
[25] Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. TabPFN: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2023. URL /ttpp::/ /ttp::///openrviee net//oorum7dd.cppPcccc...
[26] Michael Zhang, Nishkrit Desai, Juhan Bae, Jonathan Lorraine, and Jimmy Ba. Using large language models for hyperparameter optimization. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
[27] Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. arXiv preprint Stl::2002.0392/) 2024.
[28] Marcel Binz and Eric Schulz. Turning large language models into cognitive models. arXiv preprint arXiv:2306.039/ 2023.
[29] Benjamin Lipkin, Lionel Wong, Gabriel Grand, and Joshua B Tenenbaum. Evaluating statistical language models pragmatic reasoners. arXiv preprint arXiv:2305.01020 2023.
[30] Eric Schulz, Joshua B Tenenbaum, David Duvenaud, Maarten Speekenbrink, and Samuel J Gershman. Compositional inductive biases in function learning. Cognitive psychology, 99: 44-79, 2017.
[31] Bogdan Grigore, Jaime Peters, Christopher Hyde, and Ken Stein. Methods to elicit probability distributions from experts: systematic review of reported practice in health technology assessment. Pharmacoeconomics, 31:991-1003, 2013.
12

---

[32] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.05760 055611 2019.
[33] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. arXiv preprint arXiv:2207.041799 2022.
[34] Stratis Markou, James Requeima, Wessel Bruinsma, and Richard Turner. Efficient gaussian neural processes for regression. arXiv preprint arXiv:2108.00677 100.0666,, 2021.
[35] Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. In International Conference on Machine Learning, pages 8990-9012. PMLR, 2023.
[36] Weather Weather API, 2024. URL http:ppp//popoesseeemmpoooor/o/p api 2024-03-07.
Accessed:
[37] Jacob R Gardner, Geoff Pleiss, David Bindel, Kilian Q Weinberger, and Andrew Gordon Wilson. Gpytorch: Blackbox matrix- matrix gaussian process inference with gpu acceleration. In Advances in Neural Information Processing Systems, 2018.
[38] William Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):285-294, 1933.
[39] William Thompson. On the theory of apportionment. American Journal of Mathematics, 57 (2):4504456, 1935.
[40] Matthew W. Hoffman and Bobak Shahriari. benchfunk. hstt::///gihuub..co//mm/offffaa benchfunk 2015.
[41] Robert Gramacy and Herbert KH Lee. Cases for the nugget modeling computer experiments. Statistics and Computing, 22:713-722, 2012.
13

---

A LLM Processes: Defining a Stochastic Process That Can Condition on Text
In this section we elaborate on the explanations and definitions in Section 2 Our goal is to use an LLM to elicit joint predictive distribution over arbitrary sized target sets that can guide and modify using plain language. Formally, given a set of observations Duain = {z;, y:) } V and some text, T, would like to elicit the predictive distribution defined by an LLM at collection of targets ((rj, 55 }))) denoted pLIM (yi,.... ...... ViN -f,..., ,,,. Duain 7). To achieve the goal, we can can keep mind two interpretations of what we mean by predictive distribution defined by an LLM. First, we can interpret the L.M as maintaining having a predictive distribution over numerical values, which we can probe by sampling from the L.M.. This interpretation is beneficial if we believe that the LLM has learned useful prior information that we would like to access via its belief's about these numerical values and for our goal of guiding the predictive distribution using text. The other interpretation is more empirical: we simply use the LLM as a tool to define a valid predictive distribution and evaluate how well this definition performs on test cases. Our approach is a combination of the two philosophies - we will propose a method defining a predictive distribution that is valid and performs well on test cases, but closely matches what we think of as the LLM's underlying distribution.
A.1 Continuous Marginal Likelihoods From an LLM
As discussed in Section 2
we use a method similar to the one proposed by Gruver et al. 12 we :
approximate the continuous density by discretizing the space using bins with arbitrarily fine precision. Let's assume a fixed number of decimal places n, and that LLMs generate one digit at a time) 3 The
key idea is that each new digit can be viewed as being generated from a categorical distribution with the probabilities p given by softmax over numerical tokens. Crucially, this hierarchical approach allows us to compute the probability of bin with width 10-", For example, if n = then Pr(y E [1.0, 1.1)} = p(1)p(.I (1.()(((1ppp) (0))( because "1.0' is prefix for all [1.0, 1.1). We can convert probability mass to probability density by assuming a uniform distribution within each bin, and dividing the mass by the bin width. A visualization of this construction can be viewed in Appendix G.I
The method in 2 has two main shortcomings for our purposes: first, the authors propose to scale all y E Dunin to climinate decimals from their numerical representation. For example, for a precision of 2 decimal places, the numbers 0. 0.133, 1.23, 12.3, and 123.0 will be transformed to 12, 123, 1230, and 12300 respectively. Scaling removes prior information communicated the LLM via the scale of the problem. For example, it is likely that the LLM has encountered financial data with decimal places. Potentially, also makes it more difficult to communicate prior information about the problem the LLM via text.
Second, probabilities of all sequences of integers given by an LLM contain the mass of all values that also start with that sequence. We can think of this as the problem of not knowing when the LLM intends to terminate a value. For example, y 12, Pr(y E [12, 13)} * p(1) since p(1))((21)) includes the probability of all numbers with '12' as prefix- this includes [12, 13) but also [120, 130), [1200, 1300) and so on.
A.2
The LLM Process Method
We follow Gruver al. (2 and discretize the continuous space with bins of width 10-", computing the probabilities for each bin using the hicrarchical softmax approach. However, different from their approach we 1) keep values at their original scale, and 2) include a terminal token after every value for example, given a terminal token (t), we represent 12 as 12(t) and 120 as 120(t). Including a terminal token prevents numbers of varying orders of magnitude from sharing the same prefix - i.e. p(1) (())((2())p))))) (1) no longer includes the probability of numbers in [120, 130), [1200, 1300), and so on. After we compute the mass of a bin via hierarchical softmax, we divide the mass by the bin width 10-" to get an estimate of the density value. This procedure defines a valid predictive distribution over y-values, and we call this elicitation method 'logit-based" since we derive probabilities from the logits directly instead of sampling. Pseudocode can be found in Algorithm 2
'The models we evaluate are trained with tokenization schemes that tokenize each digit in number separately. Gruver al. [2] include space between each digit for tokenizers that do not tokenize each digit separately.
14

---

It must be noted that this approach does not guarantee that P(12(t)) yields the mass assigned by the LLM to values the bin [12, 13). However, we note that our method defines valid predictive distribution and we empirically observed that our predictive distribution closely matches the sampling distribution to our satisfaction (see Appendix G.
A.3 Defining an LLM Process
So far we have established a procedure for defining the predictive distribution at single target location, PLLM z Duain: T). We now discuss how define the joint predictive distribution over collection target points. In particular, we would like to define a stochastic process via its finite- dimensional marginal distributions Psi......x defined over locations æ ...... TN- The Kolmogorov Extension Theorem 4
states that such a collection defines stochastic process if it satisfies
Exchangeability Given any permutation π of the integers (1,...,NY N)
Prima (311 UN) Pa (1m(1)) Ma(N))
Consistency: if S M N then
Pe (31, ........ yM) = P P..c)) (Wr(1)-b((N)) dym++1.... .yy
In Equation ① we define a collection of joint distributions by defining a factorized distribution over target locations 2i, -
PLLLMP (YiMMP VN ii...., Iyy, Derain: T) = II PLAM xig Derains T)
where ( 2n Dirain T) is defined above.
This definition satisfies the Kolmogorov Extension Theorem and so it defines a valid stochastic process. However, assumes conditional independence given the training set and model weights and, conditional on these variables, the stochastistity represented by the model is via independent marginals. Taking inspiration from the autoregressive structure of the LLMs predictive distribution, we can write the joint distribution according to the product rule:
PA Viv 2i. IN Draim T)
II ( Vi..... Vū- 21 2 D Drain T)
Where, the previous target location is autoregressively added to the conditioning data via the LLM prompt. This should yield much richer predictive distributions we now able to model depen- dencies between output variables. However, this definition is no longer guaranteed to give us valid stochastic process as the predictive distribution is now target order dependent and most likely will fail the Kolmogorov exchangability condition. We investigate these questions in Section
15

---

B LLM Processes Pseudocode
Algorithm 1 Pscudocode for sampling numbers from an LLM
N < Number of desired samples
samples + []]
while len (en(samples) < N do
out < model. generatteenet))))
if out is number then append(oappen)((ut)
end if end while
Algorithm 2 Pseudocode for computing the log pdf of y
number of digits after decimal point monnum_idxe +- tokens & tokenize([")) 'O,, 'I',..., '9', ' (t) 'D)
full__ext -- prompt str(y)
y_idxs + indices of the tokens that correspond to y in fexl
logits model(full_teey))
logiitt + logits[y idxs]
y logits[nonnum idxs] -00
logpmf +- (loss = [ogits[[=-]] targets str(y)[l:])ssum( > Mass of bin that
includes y
y_logpdf <- logpmf n log 10
b Convert mass to continuous likelihood
16

---

C
Sample Prompts
Figure C.T depicts three observed training points and four target locations. Below are sample prompts for various configurations discussed in the paper. 7 refers to problem related text.
y
Observed Point
Target Locstions
Bx By
Cx Cy
Axi Ay
X
D
Ex
G
Figure C.I: Three observed training points and four target locations which serve the basis for the example prompts.
Independent Marginal Prompts
Sequential:
"T(t)A,, A,, (,, C, AT((),,, (,, C,
"((t) (t (t, (,,t),,, (t) (t)G)
Random:
A,1, A,)
B2, (1,
Distance:
"T()C,,, C,(t),,, B,tt,, "T(t)C,, "T,t)A, C. "T(I) (1) C,
17

---

Autoregressive Prompts
Sequential:
",( Br G,,,,,C,,G,,,,, "T(t)As; CIDDD
",( C,U,,, E,
A A, (,,1))),,, FOPE
Random:
"TWO,,, A,0,,,,,
Distance:
"T(I)C2, C, By(1) ADDD
"T(JC,,C,/1)D "
E,)C, "T(I)D" (1(1) Ay Eg F,00,,,,
18

---

D Dataset Details
This section provides details on the various datasets used in the experiments
D.1 Function Dataset
We use the 12 synthetic function datasets (Linear, Exponential, Sigmoid, Log, Sine, Beat Inference, Linear Cosine, Linear Sine, Gaussian Wave. Sinc, Quadratic, X Sine) from Gruver et al. 2 each of which consists of 200 discrete points. We construct 7 datasets each with 10 random seeds for each function with subset of 5, 10, 15, 20, 25, 50, and 75 randomly training points sampled from the original 200 points, We add Gaussian noise with μr = 0 and a 0.05 to the training points and then round the values to 2 decimal places. Unless otherwise stated, use 40 equally spaced target points to sample at.
D.2 Weather Dataset
The dataset was queried from OpenWeathed (36] and consists of daily high temperature, precipitation, and wind speed readings for 86 consecutive days from London, UK commencing on December 12, 2023. The data was recorded after the release dates of the Llama-2 and Mixtral-8x7B LLM release dates to avoid any data leakage into the LLM datasets.
For the "Comparison to LLMTime" experiment, We used the first 50 readings of the temperature data for training data and ask LLMTime and LLMPs to predict/forecast the final 36 values. The authors of LLMTime suggest the method can handle missing values by inputting NaN values in their place. Since LLMPss can work with irregularly spaced and missing data, we also compare the methods with reduced number of randomly spaced training points.
For the "Simultaneous Temperature, Rainfall, and Wind Speed Regression" experiment we used 30 randomly chosen training points within the first 76 points, leaving the last 10 for extrapolation.
E
Data Leakage
likely that LLMs used in our experiments have been exposed during training to some of the real-world data that use in experiments which would give it advantage against other models. However, we feel confident that the LLMs tested were not simply recalling memorized data that in all cases the LLMPs produces full distribution and not just deterministic value and we have taken steps in our experiments to mitigate this issue. When synthetic functions or Fashion MNIST data [12] is used, have altered the original data via subsampling. rescaling and in some cases adding noise to the datapoints. Any data used from the internet was altered from its original form when given the model. Some datasets (in particular the Weather Dataset described in Appendix D.2), were explicitly chosen to be recorded after the release dates of the LLMs that they were evaluated on.
19

---

Additional Implementation Details
PyTorch is used as the basis for all of the experiments, with the exception of the Gaussian Processes baselines that are implemented using the GPyTorch package [37
The experiments using the Mixtral 8 7B, Mixtral-8x 7B- Instruct 77), Llama-2 70B 8 and Llama-3 70B 19] LLMs were run on two NVidia A 100 GPUs with 80 GB of memory. The experiments using the Llama-2 7B B and Llama-3 8B 9 LLMs
NVidia 3090 GPU with 24 GB of memory. The total compute used in the paper exceeded 600 GPU hours.
No training was done in our LLM experiments, simply imput the prompt the LLM and ran it forward to get a prediction for particular target point.
F.1
Processing Times
Processing times vary as function of:
The GPU used.
The length of the prompt.
The number of target points queried.
The number of tokens required to be generated for particular target point.
The number of samples taken at each target point.
Whether independent or autoregressive sampling is used.
Example experiment processing times:
Basic Scenario: Table F.T indicates that the longer the prompt, the longer the computation time for each target point. For independent sampling (I-LLMP), the prompt length is constant and is only a function of the number of training points as each target point is processed independently, For autoregressive sampling (A-LLMP), the prompt length is a function of both the number of training points and the number of target points since each target point is appended to the prompt as is sampled.
Table F.1: Times to load the LLM into GPU memory, for the LLM generate all samples at all target points, and to compute the probability distribution over the true target points. All runs used the Llama-2-7B LLM and were executed on an NVIDIA 3090 GPU with 24GB of memory with a batch size of 10. All times are in seconds.
Function
Model
Load (s)
Sample (s)
Compute Likelihood (s)
Quadratic- Training Points, 40 Target Points
I-LLMP
Quadratic- Training Points, 40 Target Points
A-LLMP
170
3
Quadratic- Training Points, 40 Target Points
I-LMMP
259
4
Quadratic Training Points, 40 Target Points
A-L.MMP
354
7
ID Synthetic Data Experiments:
LLM: Mixtral-8xx-7B
GPU: x Nvidia A100, 80 GB
Parameters: A-LLMP, 40 target points, 50 samples, log probabilities
Tasks: functions x seeds x sizes
Approximate Time: 19.6 hours
20

---

Black Box Optimization:
LLM: Llama-2
GPU: Nvidia A100, 80 GB
Parameters: I-LLMP, 500 target points, sample
Tasks: 6 functions, 100 trials
Approximate Time: 20 hours
Fashion MNIST Image Reconstruction:
LLM: Mixtral-8x--7
GPU: 2 x Nvidia A100, 80 GB
Parameters: I-LLMP, 400 target points, samples
Tasks: images 2 sizes
Approximate Time: 15 hours
Simultaneous Temperature, Rainfall, and Wind Speed Regression
LLM: Llama-3 8B
GPU: x Nvidia 3090, 24 GB
Parameters: A-LLMP, 40 target points, 50 samples
Tasks: functions, 100 trials
Approximate Time: minutes
21

---

G Additional Configuration Results
G.1 Comparing Sampling and Logit Based Distributions
We first investigate whether our logit-based method of eliciting distributions (Appendix A.2) match the sampling distribution of the LLM. order estimate the true distribution, we obtain 1000 samples from the LLM at each target location, and fit histogram using the same bins as our logit- based method. Figures G.2 to G.4 show that our method yields distribution that visually similar to the one obtained by sampling.
8:0
8:20
110
plys: Sampling
plys: Sampling
10
10a2
10-2
pAy:: Logits
10
10
plys:
10a
10a
(a) Llama-7B
(b) Mixtral 8
Figure G.2: Visualization of the predictive densities estimated via sampling (middle) and model logits (bottom) for the Sigmoid function with 10 training points (shown in white). Cross section histograms (top) presented 50, 100 and 150.
Logit
Ya Camping
Logha
Va Sampiig
6.00
2.00
202
alyi Sampling
10-1
10-1
10-2
elyl:
10:a
10-3
pl9i Loghts
10
10
ply::
10
10
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
(a) Llama-7B
Mixtral 8x
Figure G.3: Visualization of the predictive densities estimated via sampling (middle) and model
logits (bottom) for the Quadratic function with 20 training points (shown in white). Cross section histograms (top) are presented r 50, 100 and 150.
22

---

surpping
Va
Surpling
6.00
1.00
plys: Samp ing
10-1
10
10-2
10
d09: Logit
10-2
10-2
10
10"*
25
75
100
125
150
175
200
25
50
75
1000
125
150
175
200
Llama-7B
(b) Mixtral 8x
Figure G.4: Visualization the predictive densities estimated via sampling (middle) and model logits (bottom) for the Linear Cosine function with 75 training points (shown in white). Cross section histograms (top) are presented x 50, 100 and 150.
23

---

G.2 Additional Prompt Format Results
Figure G.5 shows NLL and MAE for various prompt formats and 3 LLMs. Tables G.2 and G.3
show
the tabular versions of prompt formatting results.
Overall. LLMPss tested are robust to the prompt format. The results indicate that two separators are required to achieve the best performance. One to separate the x and values within a pair and another to separate the 2; y pairs. The - format uses
separate within a pair and nothing to separate the pairs and has the worst results. The x_y_ format uses letter prefixes to separate values and pairs with improved metries. Trading off token efficiency and performance, _._in An the best option as it uses only one comma to delimit and and \n to delimit 2, V pairs. However, given comma as decimal place, we - _n prompt format our experiments as it comparable performance and only uses one additional space per pair. The (.) and x= y= formats are more human readable, but the extra tokens do not improve performance.
that some regions
Sigmost
Pramst formaa
-0.5
MAG
0.55
X V 1p
5.08
Uama-2-78 Lama-2-708 Witral-Be78
Uama-2244 2-78
Llama-2-708 Nixtral-kx78
Jama-2-78
Uama-2-700 2-788 Mistral-ex78
Figure G.5: NLL and MAE for various prompt formats and each LLM. The height of each bar the mean of 10 random seeds that determine the locations of the observed points. The small black lines the top of each bar indicates the standard error. The two characters in the legend indicate the positions the z and y values. \n indicates the newline character. From left to right, the prompts are ordered from the most to least token efficient.
Table G.2: NLL for various prompt formats and each LLM. Each entry is the mean and standard еrror of 10 random seeds that determine the locations of the observed points. From left to right, the prompts are ordered from the most least token efficient. The number below each function indicates the number of observed points.
Function
LLM
.._
C)
y=
Sigmoid
Lluma-2-7B -0.963-10.056
-0.788+00072
-1.140+0.051
-1.194400.055
-1.192=0.008
-1.116=00005
Llama-2-70B
-0.9561100033
-0,897400100
-1.33540.053
-1.329::0.066
-1.231=0.054
-1.293-0.072
Mixtral-xX7B
-0.861++00067
-0.940-10.09
-1.135+00057
-1.276:10.066
.1.34820.02
-1.306=0.067
Quadratic
Llama-2-7B
-0.882400006
-0.824100009
-1.269100032
-1.266:10.032
-1.293-0.029
-1226320.023
20
Llama-2-70B
-0.980:10.035
-1. -1.207:1.0022
-1.48210.034
-1.48910.037
-1.44520.032
-1.54020.032
Mixtral-xx7B
-0.9764+0.028
-1.179440.044
-1.371+:0.033
-1.401
-1.450=0.039
-1.45920.039
Linear
Lluma-2-7B
-0.362-40.02
-0,445+0.0022
-0.64540.0299
-0.32440.034
.0.613+0.028
.0.676-00.033
Cosine 75
Llama-2- 2-70B
-0.386110.012
0.611110.077
0.67911..02
0.673+10.024
0.718:20.029
0.769=0.030
Mixtral-8x7B
-0.368110.013
0.600-10.029
-0.88510.038
-0.78810.036
-0.723-0.031
0.78220.030
24

---

Table G.3: Mean Average Error (MAE) for various prompt formats and each LLM. Each entry is the mean and standard eπror of 10 random seeds that determine the locations of the observed points. From left to right, the prompts are ordered from the most to least token efficient. The number below each function indicates the number of observed points.
Function
LLM
x_y_
G.
x=_,y_la
Sigmoid
Llama-2-7B
0.062:10.004
0.069-10.006
0.056110.005
0.061=0.004
0.060:10.004
0.053:10.00
Llama-2-70B
0.070+00.08
0.060++00006
0.047++0.005
0.049=0.004
0.054++0.005
0.047+00.005
Mixtral-8x7B
0.07110.006
0.058:10.005
0.052:10.005
0.047=0.005
0.046:10.003
0.045::0.004
Quadratic
Llama-2-7B
0.005=00.055
0.070=+0.004
0.062++0.004
0.059=0.004
0.051==0.002
0.056+0.002
20
Llama-2-70B 0.06611.000
0.055:20.003
0.044::0.002
0.046=0.003
0.050=10.003
0.040=1.0002
Mixtral-8X7B
0.065+0.003
0.051++0.033
0.047::0.002
0.049=0.003
0.048:+0.003
0.045=000033
Linear +
Llama-2-7B
0.122::00044
0.112:10002
0.093:10.004
0.097=0.005
0.088:10.00 004
0.085 0.000
Llama-2-70B
0.110::0.003
0.087-10.004
0.074:10002
0.074=0.003
0.074:10.003
0.074
75
Mixtral-8x7B
0.119:10.03
0.09210.005
0.079:10004
0.080=0.004
0.083110000
0.075:0.004
25

---

G.3
Additional Prompt Ordering Results
We consider the effect of three different orderings of the training data Dusin the prompt:
Sequential: (z,, Va), Duain are ordered sequentially from smallest to largest 2,, regardless the location of the target point.
Random; (2), 36);
Dinin are randomly ordered.
Distance: For the prediction target point x², the training points (24, Mb), E Dinin ordered from largest to smallest distance to the query point x* i.e. l Dil2 such that the training points closer to x² appear later in the prompt.
Figure G.6 shows NLL and MAE for various prompt orderings and each LLM. Table G.4 shows the tabular version of the results.
Distance ordering consistently yields the best results overall. We posit that distance ordering effective as it provides hint to the LLM to weigh the contribution of closer points to the current target point to greater degree. Unless otherwise noted, we use distance ordering for our experiments.
Sigmaid Diserved
Quadiatic- 20 Obarreed
Linear Cosine Observed
-05
64
random sequentibl
-1.0
6.2
6.1
U1ma-2278 Llame-2-706
Uluma-2-78
Ulmaa--708
Llaa-2275
Liame-2-78
Figure G.6: NLL and MAE for various prompt orderings and each LLM. The height of each bar is the mean random seeds that determine the locations of the observed points. The small black lines the top of each bar indicates the standard error.
Table G.4: Mean Average Error (MAE) and NLL for various prompt orderings and each LLM. Each entry is the mean and standard error 10 random seeds that determine the locations of the observed points. The number below each function indicates the number of observed points.
Distance
Random
Sequential
Function
LLM
MAE 1
NLL
MAE
NLL
MAE 1
NLL
Sigmoid
1lama-2-78
0.060-10.004
0.093-40.017
0.977400.03
0.150+0.016
10
Llama-2-708 2-78
0.049110.004
-1.32910.056
0.05110.000
-1.307-10.066
0.0061
0.782.10.085
0.050-+0.005
-1.276+000066
0.060+0.006
-1.240+-0.077
0.073+40.016
0.707-+0.116
Quadratic
Llama 2 7B
0.063.10.044
1.266
0.146.10.00
0.731.10.034
0.22410.012
0.14710.019
20
Llama-2-700 70B
0.046-+0.003
-1.490+6.0037
0.099-10.009
-1.013-40.00 055
0.122-00011
-0,368+00.035
Mixtral-8x7B
0.049:00003
-1.401#0.033
0.095=0.011
-1.066
0.24620.016
-0.11740.053
Linear
Llama-2-7B
0.092++0.003
-0.632-t0.00
0.205++0.000
-0.086-t0.00
0213+00.004
0,445=0.002
Cosine 75
Llama-2-70B
0.074:0.003
-0.673+00.024
0. 0.189:10.000
-0.058+1.00
0.178:10.00 0.004
0.361=00018
Mixtral-xx7B
0.08010.0044
-0.778110.036
0.204.1.0004
.0.11410.007
0.15410..006
0.410=0.034
26

---

G.4 Additional Prompt y-Scaling Results
In this experiment, we examine the effect of the magnitude and sign of the y-values of the task given to the LLM when no other contextual information is provided. We take the same three synthetic examples but scale the y-values to be in the ranges [0, 1]. [-1, 1], [0, 10) and 1-000 1000].
Figure G.7 shows NLL. and MAE for various prompt y-scaling and each I.L.M Table G.5 shows the tabular results. The values given the LLM are scaled meaning the observation noise scaled accordingly. We have scaled the likelihoods and MAE values to compensate for the difference in range. According the evaluation metrics we observe that performance degrades with increased range and incorporating negative values also hurts MAE. This is due to the fact that when the range is wider, the LLM must accurately generate more numerical digits and potentially negative sign when predicting fxx")
Sigmols 10 Obsenved
Quadrati: 20 Observed
Lirear-Codne- 75 Observed
Frmp: Scans
[0,
2,5
10.
[-1000, 1000]
NAR
0..5
3.05
Lame-2-78 Liame-2-708 Mistral8x75
Ulmaa2-78
Laama-2770B Matrrl-B788
Llama-2-88
Uame-2-706
Mintral-8x.8
Figure G.7: NLL and MAE for various prompt y-scalings and each LLM. The height each bar is the mean of 10 random seeds that determine the locations of the observed points. The small black lines at the top each bar indicates the standard error.
Table G,5: MAE and NLL for various y-scaling ranges and three LLMs. Each entry is the mean and standard error of 10 random seeds that determine the locations of the observed points. The number below each function indicates the number of observed points.
[0]]]
[-II]]
(0.10)
[-1000 1000)
Fanctiio
N
NL..
NLL.
MAR
NL..
Signold
11700-2-708
0.066
0.005
1,246
0.005
0.005
Mistrd-Ss
0,50 42 0.004
0.120 +f. 0.66
0.061 42 0.007
1,343 4f 0.66
0.051 4/ 0.005
2.502
0.05
0.064 -/- 0.006
7.668 0,212
Qualmiic
Liama-2-78
BJ61 -2 0.004
DUSG -i 0066
0J065 -1 0.004
1.322 -/ 0048
0.06 0.005
0.788 ei- +1-00061
0,00 -4- 0.003
2.524 41-2-4--000 0.00441
2-708
0.334 5009
0184 4t 11KE3
1,376 =/ 0.47
IL003
0,669 0.03
0185 IL0033
1,874 0.063
Misard-8e78
0.003
1.883
0.300
2.708
0.276
1,453
01002
2,245
01003
5,70M
Mistrd-8t78
0.556 0.003
0.193 +1. 0.066
0.055 44 0.003
1.199 4/ 0.55
0.060 0.003
1.900 4f-0.066
0.060 0.002
5,036 0.006
However, observing the plots in Figure G.8 of the predictive distribution each scale, the model gives reasonable predictions regardless of scale. no scenario context is provided via text to the LLM, rescaling task values be approximately between 0 and improves performance our experiments. However, in general we use unscaled data so that we can examine the prior beliefs learned by the LLM about tasks communicated through the raw values.
27

---

wwwm
1-1000. 10000
Figure G.8:
Predictive distributions given by the Mixtral-8x7B LLM on scaled Linear + Cos with 75 observations. This example exhibited one of the largest variation in metrics as result of scaling. Despite this, all predictive distributions look reasonable.
28

---

G.5 top-p and temperature results
shows how MAE varies with LLM top-p and temperature. Table G.6 shows the tabular version of the results.
Figure G.9
Surprisingly, all LLM's insensitive to temperature and top-p with respect to MAE.
Though not evident from these MAE results, we sometimes observed that using a top-p of 1.0 can result in some extreme values in samples. However, we consider temperature 1.0, and top-p 1.0 closest to the default distribution given by the LLM. Since had competitive performance with other options, use these settings to compute log-like logliikelhhoods in our experiments which allows us to examine the default characteristies of the LLM's predictive distribution.
Observed
Quadrstic
Linssrn-osine
Topo and
0.0
05.10
05. 10
07. 03
19) 05
07.07
10) 07
Tomk:
mistrd
Nano-78
Roma-768
misrrsl
9ama-19 18
Hamm:
misra
07. 69
07.10
10, 09
13
Figure G.9: MAE (lower better) for various temperature and top-p settings and each LLM. All LLM's are relatively insensitive temperature and top p with respect to MAE.
Table G.6: MAE (lower is better) for various top-p and temperature settings and all LLMs.
Teresnstacm0
Torgeraacc=01
Tosperauc
Taspetatrc
Fundtion
11.38
p=1.0
p=1.0
pr3.7
prtl3
Signest
01029
00134
0.3444
0,425
0.0425
L
0.065
0.3445
00439
0.0429
0.0459
08429
0/6117
0.0407
03459
00395
0.0409
0.3422
00432
0,629
0.041
0641
Squar
0.081
0.853
0/961
0.986
00838
0995)
0.85%
0083
0/858
0.0611
0/911
1,208
0.001
0.991
0.003
1077
0 10001
01015
0 1887
11005
0.9075
01065
186
0 1.3066
0.009
0.071
0.108
0.115
01891
110885
00996
0.879
01889
00864
010671
0.0
015059
0100
0.048
L-M8
0.061
0.3445
00713
0.000
0.060
6054:
0,0255
0.055
0.652
0072)
0,0511
0.2155
00709
0,008
0.033
0.835
29

---

G.6 Additional Autoregressive Sampling Results
Figure Ghoos NLL and MAE of random and distance training point orderings for A-LLMP and I-LLMP and each LLM. Table G.7 shows the tabular results.
Sigmosd- 30 Deserved
Quetratic 20 Diserved
Cosine
Deserved
Random Distance LLMP
-10-
0.1
Usma-2278 Uama-2-70B Mistrai-BX77 8x78
Usma-2-78 Unma-2-768 Mistral-2778
Lamb-2-78
Laam--2-700
Mixtral-x:78
Figure G.10: NLL and MAE for various prompt y-scalings and each LLM. The height of each bar is the mean of random seeds that determine the locations of the observed points. The small black lines the top of each bar indicates the standard error.
Table G.7: Mean Average Error (MAB) and Negative Log Likelihood (NLL) for autoregressive and marginal sampling with two different prompt orderings and three LLMs.
Random IND-LLMP
IND-LUMP
Random AUTO-LLMP
AUTO-LIMP
Function
MAE
MAE
NLL.
Sigmoid
Liuma
0.125-00:05
0,829100011
1025510,070
0,06200009
1363300022
0007200016
1.900-0051
Liama 700
0.061:00008
1.30320008
01641130077
1,25720,016
0000
2.252100334
007010.010
2.1621:0019
Mistral
0.7/3200088
1,08210000
DER010.00
-11555.0,012
0009L0011
299.10023
0055
2.217200122
Quadratic
Liuma
0.156100.110
0,769100044
B.06210.006
1.347_00,022
0.196200122
L:84100000
0004100007
L795500099
1-900
0.051:0.00
-1,10010000 099
D.06510 001
116344.0.018
010810004
1.991000044
0051100000
1.924400015
Mistral-Sr 8x28
0.000:00008
-1221120.103
0.051-19.000 005
-1.514:0,008
0/04400013
9.00000054
0049100005
-.970:00013
Linear
Liama 2-28
0.203-00:01
0,026.100000
0.0901...00
0.6188.0.001
0,20900005
0.11610001
0.102.00005
0,79910002
Costne
Lama-227000
0.122:0.015
00.10420.443
D.D D63
-4.645:0.001
0,17580011
-0.405:10.00
00/2300005
0:908:00055
MistralSs288
0.215:0,033
-0,030::00200
0.068:200007
-0 7666:00066
0:220:00055
-0.11:30.059
0050:300005
00:91:00001
30

---

G.7
Additional Autoregressive Process Results
Figure G.
shows the MAE results for the autoregressive process experiments. Figures G.12
and G. G.13S0
show the Avg log p(y) and MAE for 10 different orderings of the query points.
Sigmoid-10
Quadratic-20
Lin+ Cos- 75
0.15
0.06
0.06
0.10
MAE
0.04
0.04
0.05
0.02
0.02
0.00
0.00
0.00
L7B
L70B
Mix
L7B
L70B
Mix
L7B
L70B
Mix
Independent
Ground Truth
Sorted Test
Random Test
Figure G.11: Autoregressive process MAE results.
10
0.5
seed
Avg tog ptV1
5end
3.5
0.4
5tcd
Sted
50
Uama-2-78
Unmor 2-108
Mittral 9x78
0.0
Llama-2-78
Usme 2-708
Mixtral-Ex78
5.0
Llama-2-78 2.78
Uame 2-708
Mixtral-8s 8x78
Seed
S2cd 10
Figure G.12: Avg log p(u) for the 10 seeds for each LLM for the autoregressive process experiment.
Sigmoid 10 Observed
Qusdratic 20 Deserved
Linear MosareCosine-- -5 Oiserved
Randen Seed
seed
0.04
Seed
0.92
0.05
Seedd
seed
0.08
0.02
0.01
Need
Seed
Uamrr -788
U200--2-708
Mixtral-kx18
Liams Uammn-2-78
Laama 2-708
Wistral-h78
Uama-2-78
L10m922-700
Mistrul
Soud
Figure G.13: MAE for the 10 seeds for each LLM for autoregressive process experiment.
31

---

Additional LLMP Performance Details and Results
H.1 Additional Comparison to Gaussian Processes (GP) Results
Figures H. 14 to H.25 shows regression results from the Mixtral-8 x LLM and an RBF kernel GP for the 12 different synthetic functions.
Thre Tunction
Cont dence
Median
Thaiting prints
Confidence
Nean
Beat Inference Mixtral MAEmO. 321 NiL-m 952
Beat Inference GP MAE-0.423 NLL=1.00
0.5
0.0
Beat Inference Mixtral MAE-O 345 0.779
Beat Inference MAE-0.430 NIL-S
1.5
1.0
1.0
0.5
0.0
0.5
-0.5
-1.0
Inference Mixtral MAE N.LL
Beat Inference MAE-0.359 NIL 66B
1.0
1.0
-0.5
Inference Mixtral MAE 304
0.648
Beat Inference MAE-0.301 NLL
1.0
0.5
0.0
0.0
-1.0
Inference Mixtral MAE 302 N.L7
Beat Inference MAE 289 NIL 603
3.0
0.5
0.0
0.0
1.0
Inference Mixtral MAE 295 NIL- 0.444
Beat Inference MAE=0.300 NIL 628
1.0
0.5
0.5
0.0
Beat Inference Mixtral MAE NLL- 0.443
Beat Inference MAE-0.280 NIL S69
1.0
1.0
0.5
0.0
-0.5
005
1.0
50
75
120
325
150
175
200
50
75
100
325
150
375
200
Number of Observed Points
Figure H.14: MAE (lower is better) and NLL (lower is better) for the Mixtral-8x7B LLM versus GP as function of the number of observed points for the Beat function. The GP uses an RBF kernel with optimized length seale and noise.
32

---

Trahiing points
Exponential 05 Motraal MAE 0.18 NUL 1.338
Expomential 05 GP MAE-1.462 NLL=2.025
Exponential 10 Motral MAE NLL 1.339
Exponential 10 GP MAE=O 120 NLL- 0.828
Exponential 15 Mixtral MAE=O 068 NIL.m1 400
Exporential 15 GP MAF=0 NLL--1.579
Exponential 20 Mixtral MAE=O 059 NLL.m1 489
Exporential 20 GP MAE=a056 NI1m-14472
Exponential 25 Mixtral MAE=0 058 N11m-1.413
Exporentiall 25 GP MAE=0.077 NLLm-1.572
Exponential 50 Mixtral MAE=0 049 NLLm-1.55 565
Exporeetiial 50 GP MAE=G NLLm-1.688
Exponential 75 Mixtral MAE=O NLLm-1 665
Exporential 75 GP MAE=G.021 NLLm-1.846
50
100
125
150
175
200
350
200
Number of Observed Points
Figure H.15: MAE (lower is better) and NLL (lower is better) for the Mixtral-8x78 LLM versus as function the number of observed points for the Exponential function. The GP uses an RBF kernel with optimized length scale and noise.
33

---

Trahiing points
Gaussian Wave 05 Mixtral MAE- 0.271 NLL-0.191
Gaussian Wave 05 MAE-0 NLL
Gaussian Wave 10 Mixtral MAE- D. 0.2 NLL=0.240
Goussian Wave 10 GP MAE-0 0L4-3.473
1.0
1.0
0.5
0.0-
Gaussian Wave 15 Mixtral MAE 266 NLI=0.101
Gaussian Wave GP MAE-0.257 NILm-0.013
1.5
1.c
-10
Gaussian Wave 20 Mixtral MAE-0.277 NLL=0.011
Gaussian Wave GP MAE 236 NLL-00.10
-05
-1-
-10
Gaussian Wive 25 Mixtral MAA-0..63 NL1m-0.152
Gaussian Wave GP NAE=0.232 NLLm00336
0.5
-10
Gaussian Wive 50 Mixtral MAE-0.14 MLLm-0.55 545
Gaussian Wave GP MAE 145 NLLm00.50
1.0
1.--
0.5
0.0
0.
1.-
-1.0
Gaussian Wive 75 Mixtral MAE-0.141 MLL--0.735
Gaussian Wave GP MAE=0.01 078 NLLm-1.365
0.0
-1.0
50
125
150
175
200
75
100
325
150
375
Number of Obseived Points
Figure H.16: MAE (lower is better) and NLL (lower is better) for the Mixtral-8xx 7B LLM versus as function of the number of observed points for the Gaussian Wave function. The GP uses an RBF kernel with optimized length scale and noise.
34

---

Trahiing points
Unear 05 Mixtral MAE-0.060 NLL. 2.864
Lnner 05 GP MAE-0.501 NLL-0.874
10-
0.5
2-
Unear 10 Mixtral MAE-0.056 NL- 2.549
Linear 10 GP MAE= 048 NLL 1.272
0.5
-1
-05-
-10
Linear 15 Mixtral MAE=O 064 NILm-2.30 2.302
Linear 15 GP MAE=0.00 041 NI1m-1488
15
10-
-05
-10
Linear 20 Mixtral MAE=0.0S) NLLm-2.129
Linear 20 GP MAE=0. 028 -116-16613
1.5
005
Linear 25 Mixtral MAE=0.057 0.057 NL1m-2.004
Linear 25 GP MAE=0.02 029 N1Lm-1.701
-1
Linear 50 Mixtral MAE=0.03 0.33 NLLm-1 903
Linear 5a GP MAEmD. 029 NLLm-1.788 788
1.0
03
0.5
0.0
005
Linear 75 Mixtral MAE=0.03 033 M1Lm-19914
Linear 75 GP MAE=0. NULm-19902
1.0
1.0
0.
0.0
0.0
-10
50
125
150
175
200
75
100
325
150
375
Number of Observed Points
Figure H.17: MAE (lower is better) and NLL (lower is better) for the Mixtral-8 8B LLM versus a GP as a function of the number of observed points for the Linear function. The GP uses an RBF kernel with optimized length scale and noise.
35

---

Trahiing points
Unear+Cosine 05 Mixtral MAE-0.255 NLL=0
Linear-Cosine 05 GP NAE 276 NLL-D.
2.-
1.5
13-
0.5
0.5
Unear+Cosine 10 Midraal MAE 0.228 NLL=0.174
Lnnaar+Cosine 10 GP 237 NLL 0.095
1.5
10-
3.5
Unear-Coosine 15 Mistral 211 ML 087
Lisiar-Cosine 15 GP MAF=0.192 NLLm-0.187
3.0
1.-
3.5
Linear-Cosine 20 Mistral M44 ML 067
LineareCosine Cosine 20 GP MAE=0.151 NLL-00566
3.2
10-
3.5
0.
Linear-Cosine 25 Mixrral MAE-O 207 NLLm-0 118
Lisiar 25 GP MAE=O.137 NLL 846
10-
0.0
Linear-Cosine 50 Mitrral MAE-O 131 NLLm-0 608
Cosine 50 M7 NLLm-1
15
3.2
1.0
0.5
Linear+Cosine 75 Mitrral MAE=O 091 NLL=-0 794
Losine 75 GP MAE NLLm-1633 635
15
2.2
10
0.5
0.
75
100
125
150
175
220
25
100
125
150
175
Number of Observed Points
Figure H.18: MAE (lower is better) and NLL (lower is better) for the Mixtral-88 7B LLM versus as function of the number observed points for the Linear Cosine function. The GP uses an RBF kernel with optimized length scale and noise.
36

---

Trahiing points
Log 05 Muxrrl MAE-0.059 NLL 2.14
Log 05 GP NAE-0.490 NLL 927
-4
Log 10 Miatral MAF-O 050 NLlm-1 898
10 GP MAF=0.056 MiLm-1.194
0-
Log 15 Mistral MAF-O 050 NLl 955
15 GP MAF-0.057 MLL=-1.466
Log 20 Mistral MAF=O 042 N-Lm-1 894
20 GP MAF-0.039 MLLm-1,574
Log 25 Mistral MAE=O 041 NLLm-1 940
25 GP MAE+0.039 MLLm-1665
Log 50 Mistral MAE=O 034 N-Lm-1 853
5o GP MAE+0.033 MLLm-1.735
-15
-20
-2.0
25
Log 75 Mistral MAE=O 030 NLLm-1 688
75 GP MAE+0.020 MLLm-1.892
0.
0.0
0.-
-0.5
-10
50
125
150
175
200
75
100
150
375
Number of Observed Points
Figure H.19: MAE (lower is better) and NLL (lower is better) for the Mixtral-8 x 7B LLM versus as function of the number observed points for the Log function. The GP uses an RBF kernel with optimized length scale and noise.
37

---

Modin
Trahiing points
Quadratic 05 Mixtral MAE-0.224 NLL 2.263
Quadratic GP MAE-0.250 MLL=0.219
15
1.-
0.5
Quadratic 10 Mixtral MAE-0.087 NLL 1.807
Quadratic 10 GP MAE-0.000 IKE-0.000 NLL=-1.400
0.5
Quadratic Mixtral MAEm0 052 NILm-1.941
Quadratic GP MAE=8.092 M.I-0.050
1.5
1.0
0.5
0.5
0.0
Quadratic Mixtral MAE 044 NLL--2.000
000
Quadratic 20 GP MAE=O. 039 NLLm-1 639
0.5
0.0
000
Quadratic Mixtral MAE 041 NLL--2.039
Quadratic 25 GP MAE=O 042 NLLm-1. 726
1.5
1.0
030
0.
0.0
000
Quadratic Mixtral MAE 027 N118-2.118
Quadratic 50 GP MAE=O 021 NLLm-) 856
100
035
050
025
0.0
000
Quadratic Mixtral MAEm0 028 NLLm-2.088 096
Quadratic 75 GP MAE=O.01 015 NLLm-1 969
100
0.5
050
025
0.00
0.00
50 75
100
125
150
175
200
75
100
325
150
175
200
Number of Observed Points
Figure H.20: MAE (lower is better) and NLL (lower is better) for the Mixtral-8x77 LLM versus GP as function of the number of observed points for the Quadratic function. The GP uses an RBF kernel with optimized length scale and noise.
38

---

Trahiing points
Sigmold 05 Mixrral MAE-0.023 NLL 2.365
Sigmold 05 MAE-0.360 NLL-0.472
15-
1.-
Sigmold 20 Mixraal MAE-0.046 NLL=-2.202 2.2
Sigmold 10 GP MAE-0.037 NLL-11499 1.493
1.0
13-
0.
8.-
Sigmoid 15 Mixtral MAE-O 042 NLLm-2.215
Sigmeid GP MAE=0.02 034 N1l--1.657
1.5
0.5
0.0
Sigmoid 20 Mixtral MAE 042 NLLm-2.171
Sigmeid GP MAE=0.024 026 N1.
1.0
1.0
0.0
5.-
Sigmoid 25 Mixrral M41 NLLm-2.248 248
Sigmeid GP MAE=0.022 000.00009...00 N1Lm-1.781
1.5
1.0
10-
0.5
8.-
Sigmoid 50 Mixtral MAE=O 024 NLLm-2 160
Sigmeid GP MAE=0.00 024 N1L4-1.847
050
0.25
0.00
0.0
Sigmoid 75 Matral MAE=O NLLm-2 092
Sigmeid GP MAE=0.015 N1.
100
0.7
050
025
50
75
100
125
150
175
200
50
75
100
150
175
Number of Observed Points
Figure H.21: MAE (lower is better) and NLL (lower is better) for the Mixtral-88 - LLM versus GP as a function of the number observed points for the Sigmoid function. The GP uses an RBF kernel with optimized length scale and noise.
39

---

Modan
Trahiing points
Sinc 05 Mixtral MAE-O 094 NLL 1.354
Sinc 0s GP MAE-O 098 NLL-1.876
1.0
1.00
0.5-
0.0-
0.25
Sinc 10 Mitrral MAE=0. 087 NLL 1.243
Sinc 10 GP MAE-0. 089 NLL-0.030
1.00
0.75
0.75
030
Sinc 15 Mitrral MAF-0 087 106
Sinc 15 GP MAF-O 093 NLL-1.140
0.50
0.25
-0.25
Sinc Mixrral MAE=0 087 NLIm-1.233
Sinc 20 MAF=O 079 N7.
Sinc 25 Mixrral MAE 085 NLLm-1.256
Sinc 25 MAE 079 NLL=0.850
Sinc 50 Mixtral MAE 073 1.0m-14402
Sinc GP MAE=0.063 NLLm-))314
0.50
0.25
1-
Sinc 75 Mixtral MAE=0 027 NLLm-1.616 616
Sinc 75 GP MAE=0.022 NLLm-1.849
3.0
1.0-
0.5
25
50
100 150
175
200
50
100
125
150
175
200
Number of Observed Points
Figure H.22: MAE (lower is better) and NLL (lower is better) for the Mixtral-88 7B LLM versus as function of the number of observed points for the Sine function. The GP uses an RBF kernel with optimized length scale and noise.
40

---

Traniing points
Sine 05 Mixtral N.650.656 MLL--1.11 11.188
Sine 05 GP MAE-O 0.6 NUL-1.077
Sine 10 Mixtral MAE 0.641 NLL 0.820
Sine 10 GP MAE- 0. 0.529 NLL=0.920
Sine Mixtral MAE MLL
Sine GP NLL-0.520
Sine Mixtral MAE MLL 0.564
GP NLL-0
1.
1.0
Sine Mixtral MAE 0.572 0.555
MAE 0.106
Sine Mixtral MAE MLL
50 MAE 0.908
1.0
0.3
0.0
Sine Mixtral MAE
MLL- 0.825
75 MAE NLLm
1.0
0.0
-1.
50 75
125
150
175
200
75
100
325
152
375
Number of Observed Points
Figure H.23: MAE (lower is better) and NLL (lower is better) for the Mixtral-88 7B LLM versus as function of the number of observed points for the Sine function. The GP uses an RBF kernel with optimized length scale and noise.
41

---

Traniing points
X x Sine 05 Mixtral MAE-D 321 NLL 0.660
X Sine 05 GP MAE-D.316 NLL+5,104
X x Sine 10 Mixtral MAE-0 321 NLL 0.567
X Sine 10 GP MAE-D.308 NLL-0.530
0.5
M
Mm
-1.
-10
X x Sine 15 Mixtral MAE=0.315 315 NLL-0.119
X Sine 15 GP MAF=D 307 NLI-0.534
0.5
-10
-10
x Sine 20 Mixtral MAE 314 NLL-0.343
Sine 20 GP MAF=0. 274 NLL-0.304
3.0
0.0
-1.0
1.0
x Sine 25 Mixtral MAE 303 NLL-0.246
Sine 25 MAE 185 NIL
1.0
-1.0
Sine 50 Mixtral MAE 163 N.43-9.423
Sine 50 MAE 121 NLL
0.5
-0.5
-1.0
Sine 75 Mixtral MAE=0.11 N.73-0.732
Sine 75 MAE=O 038 NLLm-14408
1.0
0.5
0.5
-05
1.0
-10
50
75
125
150
175
200
75
100
325
152
375
Number of Observed Points
Figure H.24: MAE (lower is better) and NLL (lower is better) for the Mixtral-88 7B LLM versus GP as function of the number of observed points for the X x Sine function. The GP uses an RBF kerne with optimized length scale and noise.
42

---

Trahiing points
Linear x Sine Mixtral MAE-23. 236 NLL=4.317
Linear Sine 05 GP MAE-20:867 NLL-28.218
-25
-00
-0
Linear Sine 20 Mixtral MAE-18 735 NLL=4.148
Linear Sine 10 GP MAE-17.924 NLL-18.019
53
40-
-0
-0-
Linear Sine Mixtral MAE=18 598 NLL-3.977 13.970
Linear Sine 15 MAE=16 124 NLL-15.754
59
-20
-00
Linear Sine Mixtral MAE=16 527 NLL-3.773
Linear Sine 20 GP MAE=14.31 115 NLL-13.041
-20
-20
-00
Linear Sine Mixtral MAE=16 223 NLLm3.514
Linear Sine 25 MAE=12.801 NLL-11.393
25
25
-20
Linear Sine 50 Mixtral MAE 799 NLL=2.259
Linear Sine 50 GP MAE=9.06 061 NL1=8.357
-20
Linear Sine 75 Mixtral MAE=2. NL1=1.612
Linear Sine 75 GP MAEm4 968 NLLm4, 430
-20
25
59
100
125
130
220
50
75
125
150
175
Number of Observed Points
Figure H.25: MAE (lower is better) and NLL (lower is better) for the Mixtral-8 7B LLM versus as a function of the number of observed points for the Linear x Sine function. The GP uses an RBF kernel with optimized length scale and noise.
43

---

Figure shows plot of NLL and MAE for the Mixtral-8x xB LLM and the RBF kernel GP for 12 for the 12 different synthetic functions.
Sigmoid
Quadratic
Linea+Coosine
0.2
0.2
0.1
0.1
0.1
Liner
Log
Exponential
0.4
1.0
0.2
0.5
0.0-
Sinc
Linear
Gaussian
30
0.075
0.00
10
Sine
X x Sine
Beat Inference
0.6
0.3
0.35
02
0.30
10
20
30
40 60
20
30
40
50
60
70
10
20
30
50
70
Number of Observed Points
Figure H.26: MAE (lower is better) and NLL (lower is better) for the Mixtral-8 x LLM versus GP as a function of the number of observed points for 12 different synthetic functions. Results are averaged over three sets of random samples for the observed points. The GP uses RBF kernel with optimized length scale and noise.
44

---

H.2 Multimodal Predictive Experiment Details
To verify that LLMPs are able to produce non non-Gaune non-Gaussian, multimodal predictive distributions we sampled training data from the following synthetic, bimodal generative distribution:
.05
y + exp = 0.02x + (0.022 0.08) + 0.03cz
(H.1)
Where 61 Bernoulli(p 0.5) and 62 (0, 1). The Llama-3-70B A-LLMP predictive distri- bution using 100 training points is visualized in Figure 4 (right) and using 40 training points is visualized in Figure H.27
4
10-1
ply): Logits
10
2
10
0
10
10-5
-2
40
20
0 X
20
40
Figure H.27: Heatmap visualization of the Llama-3-70B A-LLMP predictive distribution conditioned on data from bimodal generative process. Black dots are the 40 training points.
45

---

H.3 Comparison to LLMTime
Figure H.28 compares A-LLMP in a temperature forecasting scenario to LLMTime. The dataset consists of 86 daily high temperature readings, obtained after the training cut-off for the Llama-2 LLM to avoid data-leakage. We use the first 50 readings for training data and ask the two methods to predict/forecast the final 36 values. The authors of LLMMTmme suggest the method can handle missing values by inputting NaN values their place. Since I.LMPs can work with irregularly spaced and missing data, we also compare the methods with reduced number of irregularly spaced training points. A-LLMP wins out over LLMTime, the log probabilities for A-LLMP are significantly better.
Tue Function
Triningg points
A Conf
ALLINP Miciian
LLMTimme Conf
LLMTIme Mcdian
AL/O-ULMP -LMPP Training 1aints-10 MAE-2.738 NIL=0.321
LMMTine: Traiting Polnts-11 MAE-3.189
Tamperature PG
Temperature (d
LLMP Points=22 MAE-2.55 2.503
LLMiimm:: Training Poina-22 MLL-1652
wmp bnd
Rone oratuee ("C)
155
3.0
2.5-
Training
Training
tomporture P'l
LMM bmd
LMM
AUTO-LLMP: Training Poinsweo MAE-1.92 NiL-0.064
LUMTTim:: Training Poinswd0 MAE=2.092 MiL-1.50
Temeenture (C))
Temperatree (C))
L
AUTO LLMP: Traininn Poins=55 MAE-1 892 NLL-0.co 0.09
LUMTTme: Training Points MAE-1.915 NLL-1429
Tompersare PC)
Tamperstue PC)
15.
Days Decembar 2023
Figure H.28: MAE and NLL. J for A-LLMP versus LLMTime on a dataset of daily temperatures in London, UK recorded after the release date of the LLM with varying number of training points.
The LLM is Llama-2-7B both cases.
46

---

H.4 Additional Image Reconstruction Results and Details
Figure H.29 depicts six image reconstruction results, all drawn from the Fishion-MNIST dataset [12]. The 28 28 pixel images were first scaled to 20 x 20, due to the context size limitations of the open-source LLMs used in our experiments. The pixel data was then converted into prompt data points by forming series of (row, column, pixel value) integer tuples. We then sampled 80 pixel locations (20%) and 200 pixel locations (50%) as observed points for the reconstruction. Each pixel location (400 in all) was used as target point location for independent marginal sampling with the Mixtral-8x 7B LLM.
True
20% Observed
20% Reconstruct
50% Observed
50% Reconstruct
Figure H.29: Image reconstruction results for six images drawn from the Fashion-MNIST dataset
[2]. 1st column: True images. The 2nd and 4th columns are the observed pixels for the regression task and are sampled at 20% and 50% from the true image pixels. The blue pixels indicate unobserved. The 3rd and 5th columns show the reconstructions using the Mixtral-8x7B LLM.
47

---

H.S Black-box Optimization Results and Implementation Details
Black box optimization involves minimizing or maximizing function where there only access to the output of a function for specified imput. It is often used to optimize functions that are expensive evaluate and the goal is to find the minimum or maximum value with the fewest number of calls to the function (often referred to as trials). To acquire the location of the next point to observe, we Details are in Algorithm 3 We benchmark the ability of an LLM to perform black box maximization six commonly used functions implemented in [4]], including Gramacy 41. Branin, Bohachevsky, Goldstein, and Hartmann3. We compare our results using Llama-2-7B to Optuna 3 commercial hyperparameter optimization framework. run both methods for 100 trials and record the trial at which the the best approximation to the maximum occurs. The results are shown in Table H.8 In all cases, we obtain as good or better approximation to the true maximum value in fewer number of trials. Note that Optuna will perform 100 trials in few seconds while the LLM approach can take up to 2 Nvidia A100 GPU hours. However, the results show that the log likelihood of LLMPs is capable of accurately portraying regression uncertainty.
sample the L.LM using Thompson sampling 38 39
Table H.8: Black box optimization results. The number in the Function column indicates the number of dimensions. The Trial column indicates the trial at which the Best estimate of the maximum for each method occurred.
Optuna
Llama-7B
Function
TRUE
Trial
Best
Trial
Best
Sinusoidal (1)
1.879
70
1.879
23
1.879
Gramacy (1)
0.869
48
0.869
29
0.869
Branin (2)
-0.040
85
-0.041
70
-0.040
Bohachevsky (2)
0.000
82
-5.539
49
-1.305
Goldstein (2)
-3.000
35
-4.876
31
-3.101
Hartmann (3)
3.863
86
3.745
53
3,863
48

---

Algorithm Pseudocode for LLM black-box function optimization
Require: f(x): Function to be maximized
Require: Xmin: Minimum bound
Require: Xmaz: Maximum bound on x
Require: T: Number of trials (default 100)
Require: M: Number of target points (default 500)
Require: C: Number of cold start points (default 7)
observed, + []
b. List of observed x values
observed, + []
List observed y points
for trial + to C do ヒ~~ (((mmnn: Xmaz) abserved,..ppeendx) x)
appendd (xx))
end for
for trial < C. 1 do
targets []
List of target points I> List of samples target points
samples 4 []
for + to M do
target, +~ (x(mn Xmax)
targets.append(tureett)
prompt f construct wwwstrrcctppormoppooooooooooo prompt(observed... observedy. target,) b construct a text prompt samples Algorithm I 1)
Use Algorithm T to obtain single sample the
target point end for new New_observed, < targets/argmax((snlees
Thompson sampling
apperdve ((wwoo//wwwwwwweded/r/oo end for max, + max(observed,)) max; observed (aereedd/armmaxxbbereee.... occurs
> Best estimate of maximum value of value of where best estimate of maximum of
49

---

H.6 In-context Experiment Details and Additional Plots
For the in-context learning experiment in Section we investigate LLMPs' ability to learn from similar examples in-context to predict average monthly precipitation across 13 Canadian locations [3] one from each province and territory: Alert, NU, Charlottetown, PE, Comox, BC, Goose, NL, Greenwood, NS, Keylake, SK, Montreal, QC, Ottawa, ON. Ranfurly, AB, Saint John, NB, Thompson, MB, Whitehorse, YK, and Yellowknife, NT. For each location, we use the Mixtral-8 A-LLMP to forecast 32 months of average precipitation values given the previous four month observations taken from random historical three-year period between 1913-2017 (conditional on data availability). It is then provided with 1-12 examples of random three year periods of historical values from the same location in-context. An example prompts for (1976- 1978) and 2 (1976-199 1978, 1949-1951) examples are:
1. "Monthly total of daily adjusted rainfall, mm. \n1976 1978:1n,'
"Monthly total of daily adjusted rainfall, mm. 1n1967-19990 3,0.6kn4.3 7ln5, 59, 9nn,, 135, (35,44n7,,000 10, 13.41.44m1140.33mll 2.t..3/501..522222222.....1111.. 18 137. Att.://8..SSS.........99..111111
9. In An
"Monthly total of httpa//wwwiiiiiiiiiimmimooooooaa
mm.
3,
1.4.4.ahhhhaaaaa..1222222222211. 13,22.44..44.1.88833.....
23,0.
Results presented in Figure 8
Figure H.30 and Figure H.31
50

---

examples
example
examples
12 examples
Location: ALERT. NU
Location: ALERT. NU
Location: ALERT, NU
Location: ALERT. NU
MAE 3.84375 NLL -0.95732
MAE 3.84375 NLL -1.88916
MAE 3.73766 NLL -2.04855
MAE 3.38531 NLL 11.4422
Preqgaation Jany
Preppraem on Jany
'6
15
70-
10
0-
Nenths
Nemts
Manths
Manthe
Location: CHARLOTTETOWN, PE
Location: CHARLOTTETOWN,
Location: CHARLOTTETOWN, PE
Location: CHARLOTIETONN,
202
meciptutoo mon)
152
100
20
20
32
20
10
35
Location: COMOX, BC
Location: COMOX, BC
Location: COMOX, BC
Location: COMOX, BC
MAE 05156 NILL 448994
MAE 70 75489 NLL = 3,70088
45000 NILL 3 3.67419
MAE
54 882281 NLL 14004
202
320
Pucyluster (rml
giester
220
100
Sto
102
200
120
20
10
35
20
30
1.0
35
Locationn: GOOSE,
Location:: GOOSE, NI
Location: GOOSE,
Location: GOOSE, NI
166F 49. 91 250 Ni
3.30955 0053
MAF
33 20781 NLL 199228
M6A 26.35 78735771 22
2,25742
MAF = 19.77 19.72864 NLL 2353886
155
250
Propplaiion (rre)
1:0
202
20
10
32
20
1.0
Northi
Location: GREENWOOD
Location GREENWOOD
Location GREENWOOD
Location: GREENWOOD
MRE 47.4 47.46944 NLL 3.75875
MIE 32 94944 NLL 6.08360
MAE 27 85312 NLL = 22.97142
MAE 28.67500 NLL 3.22651
Protitation unn) mm))
230
R02
Prectdaction
Proistation
10
Nonthi
Nonthu
Northa
Nontha
43380
MAE 29 75312 0.73431
0.33992
MAE 13 23.0000 0.63969
Pacphabe (rm)
Pecpratte (mm)
Pecpliaton Crmt
50-
50-
Northc
Norths
Norths
Morthi
Location: KONTREAL, DC
Location: MONTREAL,
Location: MONTREAL, Or
Location: MONTREAL
Free plaato mret
220
Proeptuaiinn ten)
50
20
20
Nonthe
Northh
Northe
Northi
Figure H.30: Visualizations of the predictions given by the Mixtral-8 X LLMP for seven locations locations accross Canada. Blue and black circles are training and test points, respectively. Red circles median predictions and shaded areas indicate renth-percentiles over 30 samples.
51

---

examples
example
4 examples
12 examples
Location:
Location: OTTAWA,
Location: OTTAWA
Location: OTTAWA
MAE 35.23488 NLLL 3.54943
MAE 26. $3555750 NIL 2.91439
MAE 22.10781 NLLL 2.82099
MAE 28.22812 NLL 2.86180
1.50
Pre todtn (ee)
Pocca (met
1200
Luben imel
2o
Northc
Northc
Norths
Northi
Location: RANFURLY,
Location:
RANFURLLY
.0.44551
2.49844
MAE
3.98546
12. 22-25406
MAE 1,93917
tenn)
Pretstatt tenerr
Predstacann
Predotaat
4oc
Nonthe
Northe
Northe
Norths Location: SAINTJCHN,
Location: SAINTJOHIN, NB
Location: SAINTOHN,
Location: SAINTJOHN, NB
3.33256
3-19853
3,07337
hredptation 1012
202
Presptutan mma
E50
100
420
co
10
20
Location: THOMPSON, MB
Location: THOMPSON,
Location: THOMPSON, MB
Location: THOMPSON, MB
1.53926
(ren
230
Poop Labrn [mm]
Pacc tatton
20
33
10
20
100
Location: WHITEHORSS, YK
Location: WHITEHORSE, YK
Location: WHITEHORSE, VK
Location: WHITEHORSE, YK MAE 9.61250 NLL 2.20566
10c
MAE 17.93750 NILL 1.86502
100
MAE 17,55469 NIL 1.05461
120
MAE 10.79531 NILL 0.79132
120
Fredb on 1987
teen)
Prede mene
20
co.
20
10
Location: YELLOW NT
Location: YELLOWONIFE
Location: YELLOWKNIFE NT
Location: YELLOWKKIIFE MAE - 17,04219 NLL 1,44717
MAE 17 17.04219 NLLL -1447727
MIE = 17 04219 NLL 1.44777
MAE 17,04219 NLL 1,4717
Predttaation bami
Procttiation jomt
ProUDRaTOO Jami
Prodstutton bpmt
20c-
230
50-
50-
50
50-
Marths
Murthh
Narths
Marthi
Figure H.31: Visualizations of the predictions given by the Mixtral-8x 7B LLMP for six locations locations accross Canada. Blue and black circles training and test points, respectively. Red circles are median predictions and shaded areas indicate tenth- metth-percentilss over 30 samples.
52

---

I Conditioning on Text Details and Additional Experiments
1.1 Scenario-conditional Predictions Details and Additional Experiments
we examine the influence of text providing information about various synthetic problem settings on the predictive distribution of an Llama-3-70B LLMP. In all of the following examples, we provide the same two synthetic training points, (1, 2.53) and (2, 2.21) to the LLM Process but change the prompting text that comes before the training data. We then use A-LLMP to forecast trajectories integer 50 steps ahead. Prompts were prepended to the standard data formatting scheme used for LLMPs (see Appendix
For the scenario-conditional predictions experiment in Section 5
The prompts provided to the LLMP visualized in Figure are:
(i.e. no text);
2. 'The following are daily temperature measurements from Montreal in January in degrees Celsius"
3. "The following daily temperature measurements from Montreal in May in degrees Celsius"
4. "In the following series, the first number is the number of Months from January and the second is the Monthly precipitation measurements inches from San Diego, CA"
5. "In the following series, the first number is the number of Months from February and the second is the Monthly precipitation measurements inches from Singapore"
The prompts visualized in Figure are:
1. "The following are daily stock prices from financial time series"
"The following are daily stock prices from a financial time series for a company that eventually goes out business"
3. "The following are daily average stock prices from a financial time series for company whose stock price goes to zero on day 30"
Lynx Hare Population Forecasting: Similar to the previous experiment, this experiment examines to what extent the predictive posterior of an LLM Process is influenced by textual information about the problem provided in the prompt. We preface the prompt with three different strings:
."' (i.e. no text);
"The following are samples from lynx-hare populations"
'The following are samples from the famous Canadian Hudson Bay Lynx- Hare population dataset. When hare increases, lynx increases. The first number of two is the year. The second number is the lynx population. follows the pattern when lynx population increases, hare decreases"
shows the predictive distribution of the LLM with 10 and 50 observed points. As the specificity of the text increases from to R, the posterior entropy decreases, and structure of the samples changes dramatically.
Figure 1.32
porutstons
Long Prompe
Figure 1.32: Results of condition both text and numerical data simultancously, on the Mixtral
model. Observed points are in purple. Colored lines show sampled trajectories. The blue shading is visualization of percentiles based 50 samples. Top: Conditioning 10 observed points. Bottom: Conditioning on 50 observed points. The predictive distribution changes as more information about the problem is added to the prompt.
53

---

1.2 Labelling Features Using Text Details and Additional Plots
In the experiments in section Section we examine the performance
Mixtral-8x7B Instruct
I-LLMP predicting American housing prices. The dataset [16] contains 39980 housing prices and various variables around housing and demographies for the top 50 American cities by population.. This dataset was generated on 12/09/2023, however it contains data from the 2020 US Census and the 2022 American Community Survey (ACS). possible that data within this dataset was used to train Mixtral-8x7B but it is very unlikely that it was trained on the exact strings presented in this experiment.
For each prediction task, we show the I-LLMP 10 randomly selected training examples from the dataset and predict 20 randomly selected test examples. In the prompt, before the numerical value (price) we provide a string which encodes the datapoint index/features that the model can use. For our first experiment we examine the behaviour of the LLMP when more features are added the prompt. We experiment with five ways of indexing the training and test points illustrated by the following training examples;
1. "32.74831, 997.21828, Price: 224900.00"
"Location: Fort Worth, Texas, Latitude: 32. 32,74831, Longitude: -97.21828, Price: 224900.00"
"Location:
Fort Worth, Texas, Latitude: 32.74831, Longitude: -97.21828, Zip Code: 761 12,
Median Household Income: 71452.0, Price: 224900.00"
"Location: Fort Worth, Texas, Latitude: 32.74831, Longitude: -97.21828, Zip Code: 761 12,
Median Household Income: 71452.0, Zip Code Population: 42404 people, Zip Code Density:
1445.0 people per square mile, Price: 224900.00"
"Location: Fort Worth, Texas, Latitude: 32.74831, Longitude: 997.21828, Zip Code: 76112, Median Household Income: 71452.0, Zip Code Population: 42404 people, Zip Code Density: 1445.0 people per square mile, Living Space: 1620 square feet, Number of Bedrooms: 3, Number of Bathrooms: 2, Price: 224900.00"
This procedure is repeated times to compute statistics. Results from this experiment are presented in Figure T0 (left, centre) and in I.34 We also ran this experiment using Mixtral-8x77 and found that the performance, shown in Figure L.33, was not as good as with the instruction tuned version of Mixtral-8x7B.
4.5
5.25
55.00
MAE $100K
4.0
-4.75
3.5
-4.50
3.0
-4.25
-4.00
3375
2.0
3.50
1) lat-long num
2) location, lat-long
3)+l income
popullat,,,,, 5) spasea density
bed, bath
Figure I.33: Average MAE and NLL performance of the Mixtral-&x7BLLMMP over 10 experiments with error bars representing the standard error.
An additional experiment is presented Section 5 to see examine effect of adding text labels to the features. This experiment run on 10 new random datasets providing the LLMP with either labeled or unlabelled numerical features. Due to the results of the previous experiment, a Mixtral-8x7B Instruct LLMP was used for this experiment. The following are example training strings for the four cases examined:
"30.45738, -97.75516 16, Price: 385000.00"
54

---

13 ter-lono
Fue Price
Fuc Mice
5) all trabures acteed
Pfice 1USD)
Piice
291332 15 16.31 House Exompie
2.7.6 1631112 23 25
House plc
House Examppic
Run
Run
Run
The Pius at long
PVice (USD)
51 features Mox
Pice 0US01
Si inatures in text
19:1
2032 2233 2433320471777101 2920
18:11 2914 241220
Run
Run 5
Run 6
Thue Price
luclorg
Feotures
PiiCe (USD)
Piice (USD)
features
62-
House Example
Hous0 Exom
HouSe Examplc
Run
Run 8
Run 9
lackang 51 all hulturos
Price BXSD)
House Example
Run 10
Figure 1.34: Results of runs using Mixtral-8x7B Instruct I-LLMP predicting US housing prices for 20 random houses from [16]. Predictions are visualized using index style 1) and 5). Xs are mean predictions using 30 samples from the LLMP and error bars indicate 2 standard deviations.
"Location: Austin, Texas, Latitude: 30.45738, Longitude: -97.75516, Price: 385000.00"
"30.45738, -97.75516, 78729, 107830.0, 30907, 1216.1, 1349, 3, 2, Price: 385000.00"
d. "Location:
Austin, Texas, Latitude: 30.45738, Longitude: -97.75516, Zip Code: 78729, Median Household Income:
107830.0, Zip Code Population:
30907 people, Zip Code Density: 1216.1
people per square mile, Living Space: 1349 square feet, Number of Bedrooms: 3, Number of Bathrooms: Price: 385000.00".
Results of this experiment are presented in Figure 10 (right).
55

---

J Additional Comments on Limitations and Societal Impact
Limitations As mentioned in the main text along with the flexibility of LLMs, LLMPs inherit their drawbacks. An additional drawback of using LLMs for probabilistic regression is that results from LLMPPs are inherently less interpretable than from methods like Gaussian processes where we explicitly encode priors. As with other black-box methods, we must, at the moment, rely on demonstrating empirically that makes well-calibrated predictions.
Societal Impact Our work has demonstrated new and useful zero-shot approach for generating probabilistic predictions using plain language augment numerical data. It has the potential to allow practitioners from fields such as medical research and climate modelling to more easily access probabilistic modelling and machine learning. We hope that such an impact would help researchers improve the lives of all humans by tackling the problems that humanity faces today.
Like all machine learning technology, there is potential for abuse, and possible consequences from incorrect predictions made with LLMPs. Due to the black-box nature of the method, we do not know the biases in the underlying LLMs used and what effect they may have LLMPss output. However, LLM researchers are striving to make LLMs more fair and equitable. An open area of research is whether LLM biases propagate to LLMP predictions and whether de-biasing LLMs helps fix such an issue.
56

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:[
Justification: We claim four contributions in paper in Section and devote entire section to each one to back up our claims.
The definition of LLMPss (Section 2)
Best practices for LLMP configuration (Section
LLMPs are competitive regressors (Section 4);
Conditioning LLMPs problem relevant text (Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Refer to Section 7 where we discuss several limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
57

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification: Our paper does not include any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Throughout the paper, we provide complete details for reproducing our experi- mental results. We do this by:
Detailing the algorithms used, see Algorithms to
Providing sampling diagram and sample prompts, see Figure 2 and Appendix
C
Complete source code at httttts:// //github. com/ rwq.eeaaaaeeaeeeeee bosqueimaallmmppeee 1mmppscessess
Extensive experiment sections in addition to lengthy appendices.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
58

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We published source code reproduce the experiments at htttp:///github com http/reqqeimm//Innn processes) Along with the code, we provide a README file that details installation, configuration, and options in order to execute the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//iips.cc/pple wttpcccppppppppeeeeeeeeeeeeeeeee wwweeeeggiisss//eee
sosoooooocss for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We supply all experimental setting/details required to reproduce the exper- iments. We do this by supplying full source code well thorough information in the experiment sections and the extensive appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
59

---

Justification:
Where the experiments have multiple runs, we show error bars in tables and charts. When we plot regression results, we report 95% confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix F details compute resources used and processing times needed reproduce the experiments. The full research project required more compute than the experiments reported in the paper due to early and failed experiments that didn't 't make into the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The authors have reviewed the NeurlPS Code of Ethics and will comply with them.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
60

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer Section f discussion of positive and negative societal impact of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We are not releasing any models and the datasets that we will release are simple numerical functions and pose pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this. but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
61

---

Answer: [Yes]
Justification: In our paper we use the following assets:
LLMs: All of the LLMs we use are open source and we properly reference them in the paper and list versions and URLs where the weights can be obtained in the README file included with our source code.
We use a dataset included with the LLMTime source code and this is referenced in the paper and acknowledged out source code.
We use datasets obtained from the internet (e.g. Weather, Housing) and properly acknowledge the source and abide by usage licences.
repurposed code for the black-box optimization functions which is properly refer- enced in the paper and acknowledged in the README file included with our source code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. thcode com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We create several new datasets for the paper and these are well documented either in the main body of the paper the appendix (e.g. Appendices [D] and D.2] We also
include these assets in the source code included in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects
Guidelines:
62

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
63


Reviews:
[{"review_id": "c1ZHpkF5ln", "reviewer": "Reviewer_dL9w", "summary": "This paper investigates the regression problem in large language models via in-context learning. They evaluate a variety of regression tasks such as for-casting and time series prediction, multi-dimensional regression, and more. They look into prompt engineering exploiting both numerical examples and their textual explanation for eliciting coherent predictive distributions.", "questions": "If I correctly understand the experimetnal part reports resutls on both training and only prompting. In this case make these two paradigms more clear in the organization of the experimental results. \n\nThe paper that is cited [21] : their experiments should be comparable to this work, why the results are not compared? while you apply more technical variations here what is the main message and your new contribution compared to them? can you put their results and yours side by side and compare them? \n\n\n[21] From words to numbers ....", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "-The problem of looking into the regression capabilities of language models in their in-context learning is very interesting and important. \n-the paper provides a large amount of experimental resutls and research work.", "weaknesses": "-some parts of the paper were unclear to me, especially the experimetnal part. \n-Though the related works are covered well in terms of citation the actual resutls are not compared to the previous ones. see below my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to review our paper and posing insightful questions. Answers to your questions are below:\n\n**If I correctly understand the experimetnal part reports resutls on both training and only prompting. In this case make these two paradigms more clear in the organization of the experimental results.**\n\nApologies if the organization of the experimental results section was not clear. We decided to organize our experiments into two main sections: the first section examines how LLM Processes (LLMPs) perform on purely numerical tasks, while the second section examines the influence of conditioning LLMPs on text and similar examples in-context.  Note that for our proposed methods (I-LLMP and A-LLMP), no training is ever involved. These methods use prompting only. Only the Gaussian Process hyperparameters are trained to compare with LLMPs. We will clarify this point at the start of the experimental sections in the next revision of the paper.\n\n**The paper that is cited [21] : their experiments should be comparable to this work, why the results are not compared? while you apply more technical variations here what is the main message and your new contribution compared to them? can you put their results and yours side by side and compare them?**\n\nThe primary reason we did not compare to the results in the “From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples” paper is that it was released on arXiv on April 11, 2024 only a little more than a month before the NeurIPS submission deadline of May 22, 2024. By the time we became aware of the paper, we had finished running our experiments and had started writing up the paper.\n\nHowever, below are the results of a new experiment where we compare our results to theirs on the Original #1 dataset (see section 4.2 in the “From Words to Numbers” paper). The experimental set-up is as follows: There are 100 trials with each trial consisting of 50 training points and a single target point. The training and target points for each trial are randomly generated using the function described in Equation 2 in the “From Words to Numbers” paper.  We use the code from their paper to generate the data and evaluate their approach and compare it to ours using identical numerical data. We use the Llama-2-7B LLM for both methods to ensure a fair comparison.\n\nOur method (I-LLMP) achieved lower Mean Absolute Error (MAE) on 78 of the 100 trials when compared to their method. When the errors are averaged over the 100 trials, our average error was 0.836 and theirs was 3.137. These results indicate that our LLMP approach is clearly superior to the approach employed in the “From Words to Numbers” paper. This is due to the facts that a) we sort the training points according to distance to the current target point when creating the prompt whereas they do not, and b) we form a distributional estimate for the predicted point and then take the median sample value as the best estimate, whereas they generate a single point estimate. We will include this comparison in the updated version of our paper.\n\nIn our paper, we also compare to the LLMTime method in Figure 5 and Gaussian Processes in Table 1 and Figure 7. These results show that we significantly outperform LLMTime, especially in terms of negative log likelihood (NLL) and are better overall when compared  to Gaussian Processes on a wide variety of functions.\n\nWhile we also show that LLMs are capable of non-linear regression, our contributions go significantly further than both the “From Words to Numbers” paper and LLMTime:\n- Our primary contribution is to condition on problem relevant text and demonstrate how this can improve prediction performance. Their work does not consider this.\n- Our work presents how to elicit full numerical predictive distributions from LLMs using two different methods (sampling-based and logit-based) while their method employs only point estimates for predictions.\n- We perform a comprehensive analysis of various prompt formats including ordering and scaling. They do no such experimentation and use only a single fixed prompt format.\n- We present a novel auto-regressive LLM sampling approach that yields superior results to the independent sampling approach used in their paper.\n- We compare our results to Gaussian Processes, viewed as the gold standard for probabilistic regression, while they compare to only simpler approaches.\n\nPlease let us know if you have any more comments or questions. If we have adequately addressed your concerns, we kindly ask that you consider revising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Zw5BwaORV", "reviewer": "Reviewer_Qkoz", "summary": "This paper proposes using LLMs to model joint distributions over numerical outputs while conditioning on potentially multiple covariates per data point. This is achieved through tokenizing a series of input and output pairs and then decoding corresponding outputs for another input. Outputs for numerical values are achieved by treating the number as a sequence of digits, each of which are represented with a token. The method is empirically validated with additional experiments showcasing capabilities of conditioning on additional contextual information, such as text describing the process.", "questions": "See weaknesses.", "limitations": "The authors extensively discussed the limitations and societal impact of the model.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The problem statement is innovative and clearly investigates many different ablations and alternatives. Extensive empirical results are shown, investigating different failure modes and comparing performance to traditional methods. The method demonstrates great versatility and interesting applications. Additionally, the paper itself is very well-written and clear in its presentation.", "weaknesses": "The main weakness in the approach that I potentially see is the runtime. From what I could tell, there is no runtime results in either the main paper or the appendix. For processing a sequence of input-output pairs, what is the general runtime exhibited, and how does it change as the sequence grows?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to review our paper and for the positive words about our contributions. We address your questions below.\n\n**The main weakness in the approach that I potentially see is the runtime. From what I could tell, there is no runtime results in either the main paper or the appendix. For processing a sequence of input-output pairs, what is the general runtime exhibited, and how does it change as the sequence grows?**\n\nWe agree that this is the main weakness our method. We view the main use case of LLMPs as being when the benefits of incorporating textual information into your regression problem outweigh the significant computational expenses involved. Appendix G “Additional Implementation Details” does contain some example runtimes. It states that processing times vary as a function of the:\n- GPU used.\n- length of the prompt.\n- number of target points queried.\n- number of tokens required to be generated for a particular target point.\n- the number of samples taken at each target point\n- and whether independent or autoregressive sampling is used.\n\nHowever, Appendix G does not specifically address your questions, so below is some new data that hopefully will give you a more tangible understanding of runtime performance. Note that this new data is quantitative as oppose to general \"Big O notation\".\n\nThe table below shows the time to load the LLM into GPU memory, the time for the LLM to generate all samples at all target points, and the time to compute the probability distribution over the true target points. All runs used the Llama-2-7B LLM and were executed on an NVIDIA 3090 GPU with 24GB of memory with a batch size of 10. All times are in seconds.\n\n| Function                                         | Model  | Load (s) | Sample (s) | Likelihood (s) |\n|:--------------------------------------------------|:--------:|:----------:|:------------:|:------------------------:|\n| Quadratic - 10 Training Points, 40 Target Points         | I-LLMP | 5        | 81         | 1                      |\n| Quadratic - 10 Training Points, 40 Target Points         | A-LLMP | 5        | 170        | 3                      |\n| Quadratic - 50 Training Points, 40 Target Points         | I-LLMP | 5        | 259        | 4                      |\n| Quadratic - 50 Training Points, 40 Target Points         | A-LLMP | 5        | 354        | 7                      |\n\nFrom the table, we can see that the longer the prompt, the longer the computation time for each target point. For independent sampling (I-LLMP), the prompt length is constant and is only a function of the number of training points as each target point is processed independently.  For autoregressive sampling (A-LLMP), the prompt length is a function of both the number of training points and the number of target points since each target point is appended to the prompt as it is sampled.\n\nWe will add these details in the next revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3IOEeYURir", "reviewer": "Reviewer_mWHr", "summary": "This paper focuses on the setting where the goal is to use LLMs to make numerical predictions, as these models can be grounded by the provision of side-information (given in text) as well as other information that may be learned during their pretraining. \n\nThey propose LLM Processes, an approach to apply LLMs to numerical tasks, such as density estimation and multivariate time series tasks. In computing their densities over continuous valued numerical inputs, they employ binning as is done in LLMTime, although they do not rescale to remove decimal points. They also add an additional terminal special token (<t>).\nIn selecting their input prompt formats, they use separators between each x, y and (x,y) pair, order points by distance to the current point, scale the y values to be closer to [0, 1] and not incorporate -1 values. \n\nOn many 1D synthetic tasks, LLMP demonstrates matching or better performance when compared to GPs with a RBF kernel. Their approach outperforms the most related baseline of LLMTime in terms of both NLL and MAE in predictions on a Weather time series benchmark. Their experiments also outperform a GP (with RBF kernel) on a multidimensional input/output task of simultaneous temperature, rainfall and wind speed regression task.\n\nThey also provide experiments looking at the ability of LLMP to handle textual information (alongside numerical inputs); they provide descriptions of the numerical features, which cannot be used in baseline forecasting approaches that do not understand text.", "questions": "1. It’s not immediately clear to me what the takeaways from Figure 9 are. What are the axes for each of the subplots? \n2. My current understanding is that incorporating some additional textual information should meaningfully change the predictive distribution, which seems to occur. Is there any way to measure how correct this change in distribution is?\n3. For instance, in Figure 9d, it seems that the median from 5 random samples seems to be roughly constant (around 2.5-3), which does not seem to match the ground truth values in 9f. Is this case just a failure of the LLM Process?", "limitations": "Yes, limitations are adequately addressed.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Good empirical results showing that the proposed approach improves over the baselines of LLMTime and a BP with an RBF kernel across a variety of univariate and multivariate forecasting tasks.\n2. Ablations supporting particular choices for input formatting and scaling", "weaknesses": "Slightly unclear in the evaluation; please see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to review our paper and for the positive words about our contributions. We address your questions below.\n\nFigure 9 is intended to demonstrate how predictions from an LLMP can be influenced by conditioning on various scenarios communicated via text prompts to the model. This experiment is a qualitative investigation with no specific ground truth information. However, we hope to show that a prompt containing two synthetic initial data points plus some descriptive text will yield samples that are meaningful.\n\nWe provide units for the axes in the prompts (detailed in Appendix J.1) as follows:\n- Figure 9b) “The following are daily temperature measurements from Montreal in January in degrees Celsius”\n- Figure 9c) “The following are daily temperature measurements from Montreal in May in degrees Celsius”\n- Figure 9d) “In the following series, the first number is the number of Months from January and the second is the Monthly precipitation measurements in inches from San Diego, CA”\n- Figure 9e) “In the following series, the first number is the number of Months from February and the second is the Monthly precipitation measurements in inches from Singapore”\n\nWe will add these units to the plots in the next revision.\n\nYou are correct that the median response in Figure 9d) does not match the ground truth in Figure 9f) particularly well. They match in the sense that both figures indicate that the precipitation is quite low all year round in Singapore, but it does not match the minor rising and falling trend. We think that this can be attributed to the fact that since the synthetic datapoints also do not match actual values very well, the LLMP had difficulty reconciling this mismatch between values and text information. We included this experiment primarily to demonstrate that the model is aware of this seasonal trend in San Diego vs. Singapore (Singapore has one of the least seasonal trends in precipitation on earth) and that can reasonably modify the LLMPs predictive distribution on the same collection of data values by varying the prompts.\n\nDifferent from Figure 9, Figure 10 is a quantitative investigation on how incorporating textual information can improve predictive performance. In this experiment, we measure how correct this change in predictive distribution is by examining improvements in the Negative Log-likelihood (NLL) and the Mean Absolute Error (MAE) given by the predictive distribution when text is included in the prompt versus when it is excluded. Figure10 shows that both these metrics improve (i.e. are lower) when conditioning on text.\n\nPlease let us know if you have any more comments or questions. If we have adequately addressed your concerns, we kindly ask that you consider revising your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the time and effort they put into reading and commenting on our paper. Our work has presented a new zero-shot approach for generating probabilistic predictions with LLMs using plain language to augment numerical data. Reviewers believe that our work is both important and innovative, with Reviewer Qkoz stating that the “problem statement is innovative” and reviewer dL9w saying that “The problem of looking into the regression capabilities of language models in their in-context learning is very interesting and important.” All reviewers agreed that the experiments, ablation studies, and competitive performance comparisons were extensive and comprehensive. Reviewer Qkoz added that “The method demonstrates great versatility and interesting applications.”

We believe that we have addressed all of the reviewer’s questions and concerns in the rebuttal. In addition, we added two new experiments to support some of the questions raised. The first new experiment shows that our LLM regression approach is superior to the one described in the recent “From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples” paper, and the second new experiment provides processing times for a representative use case.

Please let us know if there are any further questions.

Meta-review: The paper studies the use of LLMs and in-context learning for regression problems. The authors look into prompt engineering exploiting both numerical examples and their textual explanation for eliciting coherent predictive distributions. They evaluate a variety of regression tasks such as for-casting and time series prediction, multi-dimensional regression, and more. On many 1D synthetic tasks, LLMP demonstrates matching or better performance when compared to GPs with a RBF kernel. Their approach outperforms the most related baseline of LLMTime in terms of both NLL and MAE in predictions on a Weather time series benchmark. Their experiments also outperform a GP (with RBF kernel) on a multidimensional input/output task of simultaneous temperature, rainfall and wind speed regression task. Reviewers found the problem statement is novel and interesting, and the results are ample and promising. The paper can be improved by making experimental part clearer, and add analysis of runtime
Decision: Accept (poster)