IM-Context: In-Context Learning for Imbalanced Regression Tasks
Authors: Ismail Nejjar
Faez Ahmed
Olga Fink
Venue: Accepted by TMLR ()

Abstract: Regression models often fail to generalize effectively in regions characterized by highly imbalanced label distributions. Previous methods for deep imbalanced regression rely on gradient-based weight updates, which tend to overfit in underrepresented regions. This paper proposes a paradigm shift towards in-context learning as an effective alternative to conventional in-weight learning methods, particularly for addressing imbalanced regression. In-context learning refers to the ability of a model to condition itself, given a prompt sequence composed of in-context samples (input-label pairs) alongside a new query input to generate predictions, without requiring any parameter updates. In this paper, we study the impact of the prompt sequence on the model performance from both theoretical and empirical perspectives.  We emphasize the importance of localized context in reducing bias within regions of high imbalance. Empirical evaluations across a variety of real-world datasets demonstrate that in-context learning substantially outperforms existing in-weight learning methods in scenarios with high levels of imbalance.

Published in Transactions on Machine Learning Research (10/2024)
IM-Context: In-Context Learning for Imbalanced Regression Tasks
Ismail Nejjar Ecole Polytechnique Fédérale de Lausanne (EPFL) and Massachusetts Institute of Technology (MIT)
ismailnejarr@epf..cd
Faez Ahmed
facz edu
Massachusetts Institute of Technology (MIT)
Olga Fink Ecole Polytechnique Fédérale de Lausanne (EPFL)
fink
Reviewed on OpenReview: https: // opwpreveeeee...ee..em net/ forum? id- pdY28444vNGG
Abstract
Regression models oflen fail to generalize effectively in regions characterized by highly imbalanced label distributions. Previous methods for deep imbalanced regression rely on gradient-basee weight updates, which tend overfit in underrepresented regions. This paper proposes paradigm shift towards in-context learning conventional in-weight learning methods, particularly for addressing imbalanced regression. In-context learning refers to the ability of a model to condition itself, given a prompt sequence composed of in-context samples (input-label pairs) alongside a new query input to generate predictions, without requiring any parameter updates. In this paper, we study the impact of the prompt sequence on the model performance from both thcoretical and empirical perspectives. We emphasize the importance of localized context in reducing bias within regions of high imbalance. Empirical evaluations across variety of real-world datasets demonstrate that in-context learning substantially outperforms existing in- in-weight learning methods in scenarios with high levels of imbalance.
effective alternative to
1 Introduction
Imbalanced data distributions, common in the real world, pose significant. challenges to the generalization of conventional deep learning models due to variance across minority labels and bias toward majority labels (Zhang et al. 2023 While numerous studies have addressed lcarning from imbalanced data, most of them focus classifcaatio tasks (Yang & Xu 2020 Wang et al. 2024 Zhang al. 2022 Yang al. 2024 Recent work emphasizes that the continuity of the label space, and the relationship between features and labels (Shin et al. (022),, make imbalanced regression a fundamentally different problem from imbalanced classification (Yang et al.) 2021 Ren et al. 2022). Imbalanced regression problems are critical in many fields. For instance, computer vision, age estimation datasets are often imbalanced, with fewer samples for younger age groups due legal and ethical limitations, while older age groups are underrepresented as the population naturally declines with age. Similarly, in engineering design, the distribution of designs is often skewed, with the most desirable characteristics at the tail end of the distribution. This means that the region of greatest interest lios in the minority samples of the distribution. Enhancing model performance in these low-data regions is essential for achieving more accurate and reliable outcomes across these applications.
To address the specificities deep imbalanced regression, two main approaches have been proposed: sample re- no-weighting and embodding space regularization. Sample re-weighting techniques, as those proposed by Yang et al. (2021) and (Steininger et al. (2021), apply kernel density estimation smooth the label distribution. This method leverages local dependencies by enforcing similarity between nearby labels. Alternatively, embedding

---

Published in Transactions on Machine Learning Research (10/2024)
space regularization techniques, such as ranking (Gong et al. 2022) and contrastive learning (Keramati et al. 2024), preserve both local and global dependencies by leveraging label similarity rankings within the feature space. These methods, commonly referred to as in-weight learning, rely on gradient updates to adjust model weights, assuming that models can effectively generalize from limited data points in tail regions. However, with inherently less information in minority regions of the label distribution, these models are prone to overfitting samples from these regions (Cao et al., 2019
A different learning paradigm, the recently proposed in-context learning. has the potential to overcome the limitations of in-weight learning in minority regions. In-context learning refers to the ability of a model to rapidly generalize to the new concepts on which it has not been previously trained (Reddy] 2023 Bictti et al.)
2024,, using only a few examples often referred to as few-shot (Brown et al. 2020 Webb et al.,
2023 This
approach is particularly notable because it does not require any parameter updates on the model's weights.
Recent work has explored in-context learning for regression tasks. For instance, the recently proposod 'Prior-data Fitted Networks' (Müller et al.) 2022) were trained to mimic Bayesian inference, similar to Gaussian processes and Bayesian neural networks, and applied to Bayesian optimization (Müller et al.) 2023). Additionally, it has been empirically shown that transformers can learn from scratch using randomly sampled data from function classes, achieving performance comparable to that of the optimal least squares estimator in linear function scenarios (Garg et al. 2022j. Moreover, transformers can even learn from more complex function classes (Garg et al. 2022p. Interestingly, Chan et al. (2022) highlight that in-context learning particularly emerges in training distributions with properties such as long-tailedness. Unlike in-weight learning, which needs to be trained from scratch on specific datasets, an in-contert learning model is pre-trained on diverse, often synthetic, data and can adapt to different tasks using contert eromples. This means single model can perform multiple tasks without needing a separate model for each one.
In this paper, we introduce In-Context Learning for IMbalanced Regression Tasks, referred to as IM-Context, to explore this new learning paradigm from an imbalanced regression perspective. We present the counter- intuitive finding that increasing the context size can negatively impact imbalanced regression performance in minority regions. We establish a theoretical error bound for the expected error when in-context learning is applied to imbalanced regression, demonstrating that a large number of context examples can theoretically approximate the true predictive distribution. However, in the case of imbalanced regression, using the entire training set as context biases the model toward the majority region, leading to inferior performance on rare labels. As an alternative, we propase localized approach where only the 'closest' in in-context samples to a new query are used. This method mitigates bias and reduces the sequence length of context samples, thereby reducing the memory requirement for each inference.
In this localized setting, our findings reveal that in regions with dense label distributions (where the majority of samples are located), the expected error remains tightly bounded and relatively flat, indicating insensitivity to the number of context samples. Conversely, in regions with sparse label distributions (minority of samples), we observe shift in behavior where more context samples hurt the performances in the minority region: the bound widens as more context samples are added. This highlights the need for locality, as indiscriminately adding more context examples can skew the context. label distribution and worsen predictions. However, retrieving semantically relevant examples as context in practice is challenging, particularly for imbalanced datasots, because the probability of retricving samples from the majority region is higher. To address this challenge, we propose creating a second training set, where the number of samples in each region is inversely proportional to its representation in the original set: majority regions have fewer points, while minority regions are overrepresented, we refer to it as inverse density dataset. Given a new query sample, we retrieve neighboring samples from both training sets. The proposed strategy, referred to as Augmented, serves to mitigate potential biases toward the majority region; N.B. alternative sampling strategies could be applied.
To empirically validate these findings, we use two pre-trained models (Garg et al. 2022 Müller et al. 2023) and evaluate our methodology across eight imbalanced regression tasks. This comprises three benchmark datasets covering two facial age estimation tasks, one text similarity task, and six tabular datasets with different degrees of imbalance. Experiments in several real-world datascts show that our in-context learning approach consistently outperforms slate-of-the-arl in- wniwetght learning methods, particularly in regions with high imbalance. The code is available httpp::/ //gt:///bt www connismailneejii.. araaaar/IIMMM-- CM-Ctntett
2

---

Published in Transactions on Machine Learning Research (10/2024)
2
Related works
Imbalanced regression: The continuity in label space makes imbalanced regression different from imbalanced classification. One potential option is to discretize the continuous label space and apply methods typically used in imbalanced classification. However, this requires specifying a minimum acceptable crror threshold for the discretization process (Ren et al. 2022). Moreover, the continuity of the label space can provide additional information about the relationships between data instances, contrasting with the categorical distinctions in classification tasks (Yang al. 2021).
Various strategies have been proposed to refine how relationships in both label and feature spaces are handled in regression. Label Distribution Smoothing (LDS) (Yang et al. (021) and DenseLoss (Steininger et al. 2021 employed kernel density estimation to derive the 'true" label density from empirical data, thereby smoothing and reweighting the data labels. Feature Distribution Smoothing (FDS) (Yang et al.) 2021) applied similar principles to the feature space and performs distribution smoothing by transferring the first two feature statistics between nearby target labels. A different direction to overcome doep imbalanced regression is learn additional tasks to regularise the model feature representation. Different. lypes of tasks have been proposed to capture the relationships between features and labels at local and global levels (Gong et al. 2022
Keramati et al.) 2024: Wang & Wang, 2024). For instance, Gong et al. (2022) proposed ranking similarity
regularization, while Keramati et al. (2024) adapted contrastive learning to regression tasks by improving the continuity of the feature space. In a recent study, Waan & Wang (2024) leveraged deep evidential
regression (Amini et al., 2020) (a framework that learns to predict uncertainty in a single forward pass) and proposed leveraging data with similar target labels to compute the variational distribution of the latent representation, imposing probabilistic reweighting on imbalanced data.
However, these deep learning methods predominantly relied on in-weight learning, wherein achieving high-quality representations in a long-tailed setting is difficutt because the features in the minority can be easily overfitted. This issue was highlighted by Ren et al. (2022), as the Mean Square Error (MSE) loss used for training in imbalanced regression introduces biasos to the majority region. Alternative learning strategics may potentially help overcome the limilation of in-weight learning in minorily regions.
In-context Learning (ICL) in regression: ICL, based on Transformer architecture, enables a pre-trained model to learn new task with minimal examples (Vaswani et al., 2017 Brown et al. 2020). The remarkable success of transformers and their ability to do in-conlext learning shown in natura language processing has inspired line of research exploring the algorithmic power of transformers. Notably, Garg et al. 2022 investigated transformers in in-context regression tasks, ranging from learning linear regression to more complex function classes. Subsequent works have suggested that the attention mechanism in the transformer
may mimic various optimization algorithms such as gradient (Akyurek et al. 2023 Von Oswald et al. 2023
Ahn et al. 2024 Mahankali al. 2024 Bai et al. 2023).
From Bayesian perspective, Xie et al 2022 suggested that in-context learning is a form of implicit Bayesian inference. Building on this idea, Müller et al. (2022) trained transformer models using Priir Fitted Networks', where data sampled from prior distribution enables these models to approximate the posterior predictive distribution ion during inference for labular data (van Breugel & van der Schaar 2024). A theoretical foundation for these models was further developed by [agler] (2023) with a focus on classification tasks. However, the practical implementation of ICL in regression, particularly under conditions of extreme data imbalance or when dealing with real-world data, remains underexplored.
3 METHODOLOGY
3.1 Problem Setting
We address a regression problem where an input feature x E X S Rd is mapped to a label y E y = R. More specifically, in the imbalanced regression scenario, the training sct D, = [xx,,, R)) and the test set D. = [(x, ) are sampled from distinct joint distributions, (x( y) and p. y), referred to as the source and target distributions, respectively. In our setup, the source distribution typically exhibits skewed label
3

---

Published in Transactions on Machine Learning Research (10/2024)
distribution (v),,, while the target distribution is uniformly spread across the range of target values (t)). The goal is to improve the prediction on the tail regions.
In-Context Learning Consider a transformer model fo which capable of in-context learning. This capability is demonstrated when the model can accurately approximate the output Jquery for new query input Xquery based on a sequence of in-context examples. This sequence, called a prompt, is given by (x1; V/...................... Un Xquery) The training set D, can serve as basis for the prompt sequence. Given new query sample Xquery from the test set. Dr, the model predicts Oquery = So (uueuyryy D.). In the following sections, we evaluate the circumstances under which Iquery can be accurately similar to the true label Vquery-
3.2 Convergence
In the following sections, we delve into the theoretical analysis of the model convergence in the general case, We show that in imbalance scenarios, using the entire training set can introduce bias, primarily due to the overrepresentation of the majority samples. We introduce a localized version aimed at alleviating this bias and provide a theoretical bound for the expected prediction error, showing that the behavior differs depending on whether points are from majority or minority regions.
3.2.1 General convergence
Given a pre-trained transformer capable of in-context learning fe, let us consider the case where we sample points Dn from realization of the random training distribution Dz. We can decompose the model's Expected Prediction Error (EPE) for sample (x, y) as
5 (2) = En, [[o(x] Dr. - Ely x])]
= [[fo((ff(x Da) Lf, D.)l)) + [ED,,.F(( Dn)l Ely xx))²
(1)
= VVrDD.loxx D..]] (. D,))) +
Proposition 3.1 c-Lipschttz Continuity: Consider an infinitely large training set Da, from which subsets Dn and D are independently sampled. Assume that the label noise is constant. Given the observation from Garg et al.) (2022) that error decreases more samples given context, we consider that the model fo, is c-Tipschitz, with c = (cI....,Ca) RT where ai = Si-a for each E {1, (I,....,], with 8 as positive constant and > 0.5:
[f(( Dx) - 1 S Σ c cl(xxx())
(2)
Theorem 3.2 Application of McDiamdd''s s Inequality McDiarmid al. (1989) : Given that fo is c-Lipschitz under the defined conditions, for any 0, the tail probability is bounded by:
2ta
Pr(1fo(x D.) - E[ D 2t) 2 2xp
(3)
||B|
Since ||c|12 = E (6i-a)), which converges given a > 0.5, by the Borel-Cantelli lemma, so(w Dr) convergences the expected prediction almost surely:
fo(x D..) a.s. E[fo(x D.)] as n + co.
(4)
Variance : Under the assumption of Proposition the variance of the model's prediction should decrease as more samples are provided.
Proposition 3.3 Convergence to training label distribution: If we assume that the attention blocks in model fo are "mimicking' gradient descent steps (Von Oswald et al. 2023 Ahn et af.) (022)) on the context ezample D,, the model's behavior should resemble that of a neural network trained with Mean Square Error loss. Consequently, we can expect:
ED, [fo(x DnYl Bp.
(5)
4

---

Published in Transactions on Machine Learning Research (10/2024)
Bias While Proposition 3.3 suggests that the model converges to the conditional probability of ps the behavior of Jo crucially depends on Ps. If the samples from Dn are independently and identically distributed, fe can be considered an unbiased estimator of Ey x Conversely, if Dn contains biased or non-representative samples, as often occurs in imbalanced regression scenarios, the model's predictions will inherently reflect. these biases, and the model will tend underestimate rare labels.
3.2.2 Local convergence
Imbalanced regression: : A simple approach to reduce model bias in the tail regions of the label distribution is to select a finite set of neighboring samples of x from D,.- Selecting only the neighboring sample for context is equivalent to adjusting the distribution of labels ps(y x) so that it corresponds locally to pr x). This px(2(x) p=(M)
reflects the proportional relationship pa(vix)
p+(1)"
While previous propositions ensure convergencess for infinitely large datasets, it is also important to consider the expected prediction error in the case of small and finite numbers of context examples. Under this condition, we can assume that, in the worst case, the model will simply average the labels of the context samples. To the best our knowledge, there is no existing work that specifically addressos this issue in the context of imbalanced regression with k-NN. This assumption in line with previous works, for example Garg et al. (2022) showed that transformers always outperform sample averaging, while Bhattamishra et al. [2023 demonstrated that frozen GPT-2 mimie the nearest neighbor algorithm. Formally, with k samples from Dn closest to x, denoted as DR, the expected error bound is given by:
ED, [[B[y] x] fa(x Da)) - EDD [LE| x] F(x Dw))
(6)
where f(x = i- Mi represents the average of in-context labels, with k 1. The associated error can be decomposed following Hastic et al. (2009) into:
k
ED, [BB[ x] - ff((x Dx))
S
y
U1
+
+ o
(7)
Bias
Varisnce
In the case of imbalanced regression, the variance term depends on the number of selected neighbors k and should decrease more context examples selected, similar to before. The bias for the minority regions will increase with & as the averaging estimator will predict the mean of the context examples, which is also in agreement with the previous propositions. Figure I iilustrates this behavior by plotting the expected error for data points across different regions, for the Boston and AgeDB datasets. For points in the majority region, the expected error decreases as more context examples are given. However, for points in the tail region, the error initially decreases but then exhibits a U-shaped curve as k increases. Thus, for minority samples, the error bound is tighter for small number of selected neighbors, further emphasizing the need for localized context.
3.2.3 Empirical validation
To validate the error bound in practice, we compared the results obtained on two datasets: Boston and AgeDB, across three regions: many, medium, and few-shot regions. These regions are defined based on the number of training samples per label in each discretized bin, with 'Few' having fewer than 20 samples, 'Median' between 20 and 100 samples, and 'Many" exceeding 100 samples.
For cach test sample Xquerg, we retrieve its k nearest neighbors (xI, Xx) from the training set. Throughout the document, we use cosine similarity as a measure for retrieving neighboring points to the query point. Figure 2 empirically validates on the two datasets that in-context learning can perform better than simply averaging the context labels. Interestingly, for the few shot regions both datasets, we observe that the empirical error of averaging increases rapidly with the number of context examples. This underscores the challenge of retrieving neighboring samples that most useful for prediction in this region, even when
5

---

Published in Transactions on Machine Learning Research (10/2024)
100
Many anstrig Red tha reciin region
25
Hary reginn
320
Mary ahot rugion Meckum matio
Hasy Shat regien
reguon
Neston reghtr
60
2.0
320
shot
1.
3.5
2005 Sourd
Crer boars
230
120
190
30
3.0
52
-50
230
2000
Target-valaa
Martber
Number coronne samples
(a) Boston dataset distribution and Error bound
(b) Age dataset distribution and Error bound
Figure 1: Training distribution of two datasets: Boston (a) and AgeDB (b). The empirical expectations of errors are computed assuming ideal retrieval of neighbors for a new query sample (i.e., retrieving samples that are semantically relevant in terms of the target variable, rather than just close in the input space). The behavior of examples from different. shot regions varies distinctly with the number of context examples. In the many-shot regions, the error bound stabilizes as more examples are provided, whereas in the few-shot regions, additional context examples lead to an increase in the error bound.
Boston Dataset
Age Dataset
2.5
1.4
Many shot region In-context
2.0
1.2
Averaging
Empiriaa Error
1.5
1.0
Modium shot region In-context
Empirical
1.0
Averaging
0.6
Few shot region In-context Averaging
0.4
0.0
0.2
20
40
60
80
0
20
40
60
80
Number context samples
Number of context samples
Figure 2: Empirical Error: Averaging vs In-context learning (using GPT-2 model from Gaag al. 2022)
using a small number of context examples. As highlighted by Steck et al. (2024), samples that are close in feature space according to a given distance metric (e.g., cosine similarity) do not necessarily have similar target values. In imbalanced datasets, this discrepancy between proximity in feature space and similarity in target values makes difficult to select representative samples for the region, as the difference density makes more likely to retrieve samples from the majority region.
3.3 Retrieving neighbours
Retrieving representative samples as context in practice is challenging, particularly for imbalanced datasets, because the probability of retrieving sam- ples from the majority region is higher. To address this challenge, we propose creating second training set, where the number of samples in each region is inversely proportional to its representation in the original set majority regions have fewer points, while minority regions are overrepresented, we refer to this dataset as inverse density seen in Figure in the appendix. For each new query sample, retrieve k = K², + k, neighboring examples from both (1) the training set (K, neighbors) and (2) and inverse density dataset (h, neighbors), to ensure balanced representation and reduce the risk of bias toward the majority region. We refer to this version as aug- mented. In Figure 3 we compare the performance
AgeDB Dataset
0.9
0.8
Empirical Ereer
0.7
0.6
0.5
0.2
20
30
40
7.0
80
Number contoct samples
Many shot regton
Medium shot region
Few region teor
Neigghbors
Neighboes
Augmented Neighbors
Augmenttied Neighborrs
Augmented Neighbers
Figure 3: Impact of neighbors retrieval on performances (using GPT-2 model proposed in (Garg et al.) 2022)
6

---

Published in Transactions on Machine Learning Research (10/2024)
of in-context learning by retrieving neighboring context examples from the original training dataset and the augmented version. We can clearly see that this strategy decreases the error on both medium and few-shot. regions, and accentuates the U-shape curve, as expected theoretically. Other resampling strategies, such as Smoter and undersampling, are discussed in the ablation.
4 EXPERIMENTS
4.1 Experimental Setup
Datasets: We use three benchmark datasets curated by Yang al. (2021) specifically for imbalanced regression tasks: AgeDB-DIR, derived from the AgeDB dataset for age estimation Moschoglou et al. (0117);
another age estimation dataset sourced from IMDB- WIKI Rothe et al. (2018) and STS-B-DIR, which measures text similarity between two sentences based on the Semantic Textual Similarity Benchmark Wang ct al. 2018). Additionally, we use six tabular datascts, namely Boston Harrison & Rubinfeld
(1978), Concrete Yehl (2007); Abalone Nash et al. (1955), Communities Redmond (2009), Kin8nm, and an engineering design dataset: Airfoil Chen et al. (2019), with inherent imbalances. For these tabular datasets, created balancod test set to evaluate model performance, to assess model performance in scenarios with small number of training samples, different feature sizes, and varying levels of imbalance.
Evaluation Metrics: Consistent with established protocols by Yang et al. (2021), we assess model performance using Mean Absolute Error (MAE) and Geometric Mean (GM) for the AgeDB-DIR and WIDB-WIKIDDIR datasets, For the other datasets, we use Mean Squared Error (MSE) as the primary metric, consistent with Yang et al. (2021). We present our results across four predefined shot regions- All, Many, Median, and Few which categorize the subsets of the datasets based on the number of training samples available per label within each discretized bin. Specifically, the 'Few" category includes bins with fewer than 20 samples, 'Median' encompasses bins with 20 to 100 samples, and 'Many" refers to bins with over 100 samples per label.
Preprocessing of inputs for In-Context Learning: In this paper, we propose an in-contexi learning approach as opposed to in- in-weight learning, which means we do not train any models directly. For the benchmark datasots, we preprocess images and text into embeddings using the Hugging Face implementation of CLIP (Radford et al. 2021) and BERT ("all-mpnet-base-v) Specifically, for each image from the training and testing datasets, we extract embeddings from the CLIP image encoder before pooling, resulting in 768-dimensional features. A similar process is applied to the STS-B-DIR dataset, where we use BERT's text embeddings for each sentence, also resulting in 768-dimensional features. For the tabular datasets, we use the feature representations ions directly as inputs to the transformer. Moreover, we consider two in-context learning models: the first from Garg et al. (2022), referred to as GPT2, which uses the GPT-2 architecture and trained on non-linear data points; the second model from Mülle et al. (202)), referred to Prior Fitted Networks (PFN). They accept 20-dimensional and 18-dimensional inputs, respectively. When the number of dimensions exceeds the model's input size, we split the input feature representation into non-overlapping chunks and ensemble the predictions from each chunk. To ensure the robustness and reproducibility of our results, we conducted three separate experiments using different random seeds, between 0 and 3. We report the average results over all three seeds to account for the randomness introduced by inverse density sampling for the second training sct. In all experiments, for the GPT2 model from Garg et al. (2022), we retrieve Kg = h; = 10 nearest neighbors, for the PFN model we retrieve K, K, 15, except for the IMDB dataset where Kg was equal to 5. This choice is motivated by the preliminary experiments conducted in scction 3.3 as observed in Figure 3 where the minimum error in all regions is achieved for 10 neighbors. A sensitivity analysis on the number of neighbors is presented in Table An NVIDIA RTX 2080 GPU was used for all the experiments.
model) (Reimers & Curevych] 2019).
4.2
Experimental Results
AgeDB: In Table I we report the results obtained on the AgeDB-DIR benchmarks. We compared the best combination of previously reported results to our approach. Given the small size of the training dataset (12.2K images), observe that in-context learning outperforms all in-weight learning baselines in all shot. regions,
7

---

Published in Transactions on Machine Learning Research (10/2024)
surprisingly even in the many-shot region. In the few-shot setting, our localized version using the Garg et al. (2022) model achieves the lowest mean absolute error of 7.83, which is an improvement of 1.4 error points over the combination ComLZ-DD++DDS+Ranle
proposed in Keramati et al. (2024). Furthermore, on the Ag dataset, in-weight learning achieves the highest performance on all eight shot-metrics combinations with the best MAE results of 6.05 on the overall lest. set.
Table 1: Main results for AgeDB-DIR, Best results are in bold and second- best results are underlined.
Learning IWL ICL
Metrics
MAE +
GM
Shot
all
many
medium
few
all
many
medium
few
MSE loss
7.77
6.62
9.55
13.67
5.05
4.23
7.01
10.75
LDS + FDS Yang et al. 2021
7.55
7.01
8.24
10.79
4.72
4.36
5.45
6.79
RankSim (Gong et a 2022)
/
7.02
6.49
7.84
9.68
4.53
4.13
5.37
6.89
VIR Wang & Wang 2024
/
6.99
6.39
7.47
9.51
4.41
4.07
5.05
6.23
ConR TDS FDS
RankSim [eramati et al. 2024
/
6.81
6.32
7.45
9.21
4.39
3.81
5.01
6.02
PFN localized (Müller et al.
2023
(Ours)
X
6.58
5.61
8.49
10.49
4.29
3.58
6.30
8.19
GPT2 localized (Garg et 2022
(Ous)
X
6.05
5.67
6.71
7.83
3.79
3.59
4.17
4.90
Table 2: Main results for WIDB-WIKI-DIR. Best results are in bold and second-best results are underlined.
Learning IWL ICL
Metrics
MAE
GM
Shot
all
many
medium
few
all
many
medium
few
MSE
/
8.06
7.23
15.12
26.33
4.57
4.17
10.59
20.46
LDS + FDS (Yang et 2021
/
7.78
7.20
12.61
22.19
4.37
4.12
7.39
12.61
RankSim + FDS Gong et al. 2022
7.35
6.81
11.50
22.75
4.05
3.85
6.05
14.68
ConR FDS (Kermatii et al. 2024
/
7.29
6.90
12.01
21.72
4.02
3.83
6.71
12.59
VIR Wang & Wang 2024
V
7.19
6.56
11.81
20.96
3.85
3.63
6.51
12.23
PFN localized Miilerr et al.
2023
(Our))
X
8.96
8.71
10.79
16.33
5.26
5.17
6.00
9.42
GPT2 localized (Gar
2022
(urr)
x
7.76
7.35
11.15
17.71
4.29
4.13
5.96
11.00
IMDB: In Table 2 we report the results obtained on the IMDB-WIKI-DIH benchmarks. This is the largest dataset used in this study with 191.5K training samples. Both models (Garg et al. 2022 Müller et al. 2023) using our localized approach achieve the lowest error the few-shot and medium-shi regions. The best-performing in-context model is (Miller al. 2023), which achieves an MAE of 16.33 and 10.79, improving over state-of-the-art methods age estimation by 34.63 and 1.02 error points, respectively. In the many-shot region, in-weight learning outperforms in-context learning due to the abundance of training samples (almost 150k samples), which facilitates more effective learning and better generalization.
STS: Table 3 presents the results on the STDRB-DIR dataset. In this text modality, both models using our localized method consistently and substantially improve the results over state-of-the-art methods in all regions, achieving the best MSE of 0.528 on the overall test set and 0.566 in the few-shot region.
Table Main results for STS-B-DIR. Best results are bold.
Metrics
MSE J
Shot
all
many
medium
few
MSE loss LDS + FDS
0.974
0.851
1.520
0.984
0.903
0.806
1.323
0.936
VIR
0.892
0.795
0.899
0.780
Tabular: Table 4 shows the average results on six tab- ular datasets. While the previous results on the previous datascts rely on features extracted from pre-trained mod- cls, a direct comparison in- weight learning in-context learning be made using tabular datasets. The results clearly show that in-context. learning outperforms
RankSim
0.865
0.876
0.867
0.670
PFN localized
0.544
0.536
0.547
0.618
GPT2 localized
0.528
0.524
0.527
0.566
8

---

Published in Transactions on Machine Learning Research (10/2024)
different machine learning methods in the medium and few shot regions. Across the six datasets the proposed method using [Garg et al. 2022) model is almost consistently ranked first across all those datasets on the few shot region, and is highly competitive in comparison to other methods in the medium shot region.
Table 4: Average results on the Tabular datasets. Best results are in bold. (med.: medium)
Learning IWL ICL
Metrics
RMSE
Rank +
Shot
all
many
medium
few
all
many
med.
few
KNN
3.96
0.23
1.72
0.30
3.07
+ 0.48
6.41
0.47
4.5
6.0
6.5
Linear Regression
4.00
0.25
2.10
+:0000 0.24
3.28
+ 0.47
6.18
+ 0.54
5.3
5.3
6.2
5.5
Decision Tree
V
X
3.03
0.32
1.90
1 0.42
2.72
+ 0.63
4.29
+ 0.48
7.0
7.3
7.2
5.5
Gradient Boosting
2.50 1:00:0000 0.07
1.26 1 0.21
2.04
1 0.17
4.05
0.13
3.8
3.2
3.8
5.0
Random Forest
X
2.57 + 0.16
1.22 = 0.35
2.07
- 0.30
4.16
= 0.35
3.5
2.5
3.5
4.3
Neural Networks
X
2.49
+ 0.21
1.44
+ 0.25
2.13
+ 0.21
3.79
+ 0.51
2.7
3.2
3.2
2.6
PFN localized (Ours)
2.67
0.19
1.37
+ 0.23
2.03
+ 0.27
4.38
+ 0.26
5.2
4.5
3.3
5.3
-
GPT2 localized (Ours)
2.34
0.19
1.72
+ 0.35
1.95
+ 0.22
3.31
+ 0.39
2.3
5.5
2.8
1.2
5 ABLATIONS
Representation learning. In our experiment with the age datasets, we used CLIP embeddings across all age datasets to assess the effectiveness of in-context learning compared to in weight learning. The methods in Table 1 were trained to learn representations directly from the images, whereas we used pre-extracted CLIP embeddings. We trained a thyee-layer MLP with 256 neurons in the hidden layers using these embeddings. Additionally, we conducted an experiment where we fine-tuned the GPT2 model on the AgeDB dataset. Results presented in Table 5 indicate that also in this scenario in-context learning consistently outperforms in-weight lcarning. Interestingly, the finc-tuned GPT2 model achieves almost similar performance to the in-context learning approach. These results suggest that in-context learning is particularly effective solution for relatively small datasets with the presence of scarce data, such AgeDB.
Table 5: Ablation results for AgeDB-Dii benchmark.
Learning IWL ICL
Metries
MAE J
GM I
Shot
all
many medium
few
all
many medium
few
MLP
X
6.67
5.97
8.22
9.01
1.29
3.86
5.48
5.84
GPT2 - Finetuned
Garg et al.
2022
x
6.14
5.73
6.69
8.09
1.01
3.70
4.59
5.39
GPT2 localized Garg
2022) (Ours)
/
6.05
5.67
6.71
7.83
3.79
3.59
4.17
4.90
All Context vs. Localized Approach. A key rationale for using the PFN model from Müller et al. (2023) is its lack positional embodding. This allows for a dircct comparison the model's performance using the entire training sel versus a localized version as context. Figure presents results across the first five tabular datasets. The engineering dataset, which has more than 30k training samples, could not fit memory and was therefore excluded from this ablation. The results show a significant performance improvement with the localized approach in in-context learning compared to the non-localized version across all regions, with particularly significant improvement in the few-shot region.
Sampling strategy: In Section 3.3 we presented a stralegy to mitigate the underreepeeeentatio of minority points when retrieving neighbors. In this section, we compare different sampling strategies on the tabular datasct. The 'Vanilla' approach retrieves neighbors from the original training set. 'Downsampling' reduces the majority region, creating balanced training set and retrieving neighbors only from this balanced set. SMOTER (Branco et al. 2017 generates new synthetic data points for the minority region, and neighbors

---

Published in Transactions on Machine Learning Research (10/2024)
are retrieved exclusively from this augmented dataset. 'Inverse' uses dataset where the number of samples in each region is inversely proportional to its representation in the original set. Lastly, our strategy involves creating an Augmented dataset to ensure balanced representation while preserving diversity in the majority region. As seen in Table across the datasets, our method demonstrates the best trade-off for mitigating bias all regions.
Methos
PFN Localized
Metrics
RMSE +
PPN all training conteat
Shot
all
many
medium
few
RASE
Vanilla
2.43
1.41
1.99
3.70
Downsampling
2.65
2.42
2.38
3.17
Inverse
2.79
2.93
2.24
3.13
SMOTER
2.43
1.68
2.09
3.37
Augmented (Ours)
2.34
1.72
1.95
3.31
Shot
Table 6: Ablation results on the Tabular datascts using alternative sampling strategy.
Figure 4: Localization vs all training set set as context
6 Conclusion
In this work, we proposed the of in-context learning to address the challenge of imbalanced regression, where model can learn from context examples during inference without any additional training the model weights. We show that single in-context learning model can adapt to multiple tasks and improve performance in regions with high imbalance, unlike in-weight regression models, which often fail to generalize in minority regions and require specific training for each task. Through theoretical and empirical analyses, we highlighted the importance of localized context using a small number of selected neighbors in mitigating the bias inherent in imbalanced label distributions. Our evaluation across several benchmark datasets with diverse modalities revealed that in-context learning achieves superior results in regions of high imbalance.
Our comparative analysis demonstrated that two in-context learning models (Müller et al. 2023 Garg et al. 2022) based on Transformer architectures can achieve strong performance in imbalanced regression tasks for one-dimensional labels. While models are currently trained to predict single value for a given query, future work will focus on extending our approach to multi-dimensional labels, such as depth cstimation. Future work could also explore other model variants, and further investigate the factors contributing to performance differences between different in-context learning models for minority and majority regions.
References
Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. The Eleventh International Conference on Learning Representations, 2023.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. Advances in Neural Information Processing Systems, 33, 2020.
Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers statisticians: Provable in- context learning with in-context algorithm selection. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023. URL https://openrevier: hotpo//ooooooooommmmmddd111/oooo
Satwik Bhattamishra, Arkil Patel, Varun Kanade, and Phil Blunsom. Simplicity bias in transformers and their ability to learn sparse Boolean functions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki
10

---

Published in Transactions on Machine Learning Research (10/2024)
(eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5767 5791. Association for Computational Linguisties, 2023. doi: 10. v0.11/33/v1/ acl 1o1...1....
Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of transformer: A memory viewpoint. Advances in Neural Information Processing Systems, 36, 2024.
Paula Branco, Luís Torgo, and Rita P Ribciro. Smogn: a pre-processing approach for imbalancod regrossion. In First internations workshop on learning with imbalanced domains: Theory and applications, pp. 36-50. PMLR, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aobelllistibbuton-awee aware margin loss. Advances in neural information processing systems, 32, 2019.
Stephanie Chan, Adam Santoro, Andrew Lampinen, Jane Wang, Aadit Singh, Pierre Richemond, James McClelland, and Felix Hill. Data distributional properties drive emergent in-context learning in transformers. Advances in Neural Information Processing Systems, 35:18878 18891, 2022.
Wei Chen, Kevin Chiu, and Mark Fuge. Aerodynamic design optimization and shape exploration using generalive adversarial networks. In ATAA Seitech 2019 forum, pp. 2351, 2019.
Andrew Frank. Uci machine learning repository. http://archive. ics. uci. edu/ml, 2010.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in- context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35: 30583-3059h 2022.
Yu Gong, Greg Mori, and Frederick Tung. RankSim: Ranking similarity regularization for deep imbalanced regression. In International Conference on Machine Learning (ICML), 2022.
David Harrison and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1):81-102, 1978. ISSN 0095-0696. doi: https://doi.org:
1016/0005-0690((8000000--
Trevor Hastie, Robert Tibshirani, Jerome Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009.
Amin Heyrani Nobari, Wei Chen, and Faez Ahmed. Pedgan: A continuous conditional diverse generative adversarial network for inverse design. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery: Data Mining, KDD '21. ACM, August 2021. doi: 10.1145/3447544.34777111 URL http:
//dx. delxx.doi.org// 10. 1115/11447448.3345555 3467414
Mahsa Keramati, Lili Meng, and R. David Evans. Conr: Contrastive regularizer for deep imbalanced regression. In The Twelfth International Conference on Learning Representations, 2024.
Arvind V. Mahankali, Tatsunori Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. In The Twelfth International Conference on Learning Representations, 2024.
Colin McDiarmid et al. On the method of bounded differences. Surveys in combinatorics, 141 141(1):148-188) 1989.
Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. Agedb: the first manually collected, in-the-wild age database. In proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 51-59, 2017.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Representations, 2022.
Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. Pfns4bo: In-context learning for bayesian optimization. In International Conference on Machine Learning, pp. 25444-25470. PMLR, 2023.
Thomas Nagler. Statistical foundations of prior-data fitted networks. In International Conference Machine Learning, pp. 15660-256777 PMLR, 2023.
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1995. DOI: https://doi.org: 10.24432/C55666 C55C7W.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine lcarning in Python. Journal of Machine Learning Research, 2822225-280,, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-875 8763. PMLR, 2021.
Gautam Reddy. The mechanistic basis of data dependence and abrupt learning in an in-context classification task. In The Twelfth International Conference on Learning Representations, 2023.
Michael Redmond. Communities and Crime. UCT Machine Learning Repository, 2009. DOI: https://do..org/10.224422 10.rg//000.22//553333
Nils Reimers and Iryna Gurevych. Sentence-bert. Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL attts:///arxiss org/
10084
Jiawei Ren, Mingyuan Zhang, Cunjun Yu, and Ziwei Liu. Balanced msee for imbalanced visual regression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7926-7933 2022.
Rasmus Rothe, Radu Timofte, and Luc Van Gool. Deep expectation of real and apparent age from single image without facial landmarks. International Journal of Computer Vision, 126(2):144-157, 2018.
Nyeong-Ho Shin, Seon-Ho Lee, and Chang-Su Kim. Moving window regression: A novel approach to ordinal regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18760 18769, June 2022.
Harald Steck, Chaitanya Ekanadham, and Nathan Kallus. Is cosine-similarity of embeddings really about similarity? In Companion Proceedings of the ACM on Web Conference 2024, pp. 887-890, 2024.
Michael Steininger, Konstantin Kobs, Padraig Davidson, Anna Krause, and Andreas Hotho. Density-based weighting for imbalanced regression. Machine Learning, (00:2187-2211, 2021.
Boris van Breugel and Mihaela van der Schaar. Why tabular foundation models should be a research priority. arXiv preprint arXiv:2405.01144 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsov, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference Machine Learning, pp. 35151-35174 PMLR, 2023.
12

---

Published in Transactions on Machine Learning Research (10/2024)
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop BlackboXNLP: Analyzing
and Interpreting Neural Networks for NLP, pp. 353-355 -55, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10. N0.18653/vll vv WW16-546. URL https:://cclanthhll.gg/oo wwt.gyy..rg//18---10
Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. A unified generalization analysis of re-weighting and logit-adjustment for imbalanced learning. Advances in Neural Information Processing Systems, 36, 2024.
Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. Advances in Neural Information Processing Systems, 36, 2024.
Taylor Webb, Keith J Holyoak, and Hongjig Lu. Emergent analogical reasoning in large language models.
Nature Human Behaviour, 7(9):1526 1541, 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. Advances in neural information processing systems, 1119999-9901.1 2020.
Yuzhe Yang, Kaiwen Zha, Yingcong Chen, Hao Wang, and Dina Katabi. Delving into deep imbalanced regression. In International conference machine learning, pp. 11842-11851. PMLR, 2021.
Zhiyong Yang, Qianqian Xu, Zitai Wang, Sicong Li, Boyu Han, Shilong Bao, Xiaochun Cao, and Qingming Huang. Harnessing hierarchical label distribution variations in test agnostic long-tail recognition. arXiv preprint arXiv:2405.07780, 2024.
I-Cheng Yeh. Concrete Compressive Strength. https::/doi.org/// wtt.oorg//....... 11.21432//C5PPPE /CPPK77.
UCI Machine Learning Repository, 2007. DOI:
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition. Advances in Neural Information Processing Systems, 35:34077-34000 2022.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
13

---

Published in Transactions on Machine Learning Research (10/2024)
A
Appendix
B Retrieving neighbours
In this section, we further motivate the design choice of augmented strategy. As shown in Figure in the case of imbalanced regression, retrieving neighbors from the original training set can result in selecting points from the same region, which may not provide informative insights into local feature behavior. From Figure 5 Using the vanilla distribution will skew the choice of neighbours towards the majority region. Conversely, with 'inversc sampling', the distribution becomes more reprosentative of the minority region. The aim of our augmented stralegy is to find a good Irade-off' between these two sampling methods. Conversely, our strategy makes easier to retrieve diverse and informative neighbors around the query point Equerg-
(1) Training set
(2) Inverse Density
21
V1
Transformer
V2
Dipeyy 1)
Sowey
Ugary 1
10
62
2 query EORORD
Dguery
(a) Given training set,
(b) We extract neighbouring samples from (1) the original training set and (2) second training set (sampled points from the training set in inverse proportion the label distribution), as context.
(c) We evaluate the model's ability to predict accurately on a new query sample given the sclected context
and a new query sample
Figure 5: An overview of the proposed approach for imbalanced regression. Rather than relying on in-weight learning, which trains models dircctly on the training data, we propose leveraging in-context learning. For each query sample, we retrieve A Kg k2 neighboring examples from both (1) the training set (Kg neighbors) and (2) and inverse density dataset (h: neighbors), where the number of samples in each region is inversely proportional to its representation in the original set, and feed these as context to the model. This serves dual purpose: avoiding bias toward the mean of the training set, which is crucial for tail regions, and reducing the memory requirement of the transformer.
C Experiment Details
C.1 Datasets
Age Estimation We evaluated our method using two imbalanced regression benchmarks for age estimation provided by Yang et al. (2021): IMDB-WIKI-DIR and AgeDB-DIR. The IMDB-WIKE dataset (Rothe et al. 2018) includes 191,509 1,09 images for training and 11 11,022 images for validation and testing. The ages are discretized into 1-year bins, from age 0 to 186. The AgeDB dataset (Moschoglou et al. 2017 contains 16,488 samples. AgeDB-DIR was structured similarly to IMDB-WIKI-DIR, with age bins ranging from 0 to 101.
Text Similarity We evaluated our method using imbalanced regression benchmarks for textual similarity estimation provided by Yang et al. (2021). STS-B-DIR (Wang al.] 20188 consists of sentence pairs sourced from news headlines, video and image captions, and natural language inference data. Each pair is annotated by different annotators, resulting in an averaged continuous similarity to predict these similarity scores based on the sentence pairs. The training dataset includes 5,249 pairs of sentences, with validation and test sets containing 1,000 pairs each.
ranging from 0 to 5. The task is
We used five datascts from the UCI machine learning repository (Frank 2010 Boston, Concrete, Abalone, Kin8nm, and Communities, a sixth dataset from engineering: the airfoil dataset. The label distributions for all the tabular datasets is shown in Figure 7
Tabular Datascts
14

---

Published in Transactions on Machine Learning Research (10/2024)
14000
12000
250
Number of samples
10000
300
2.00
8000
200
6000
4000
100
50
2000
60
100
150
Target
Targut vallass.
Target
(a) AgeDB-DIR
(b) IMDB- WIIII-IR
(c) STS-DIR
Figure 6: Training distribution of labels for the image and text modality.
For the UCI datasets, we created training and testing sets, ensuring the test sets were balanced, using the Algorithm ??. Due to its small size, the Boston dataset was split 90% for training and 10% for testing, and the bin size was to 15. The other datasets were split 80% for training and 20% for testing and the bin size set to 50. The specific details:
Boston: 13 features, 462 training samples, 44 lesting samples.
Concrete: 8 features, 836 training samples, 194 testing samples.
Abalone: 7 features, 8,634 training samples, 543 testing samples.
Kin8nm: 8 features, 6,766 training samples, 1,426 testing samples.
Communities: 101 features, 1,684 training samples, 310 testing samples.
For the engineering dataset: airfoil dataset (Chen et al.) 2019 we used 96 Bézier features (Heyrani Nobari 2021 with 38,802 training samples and 9,701 testing samples.
et al.
C.2
Metrics
In the main paper, we reported results using the Mean Absolute Error (MAE), Root Mcan Squared Error (RMSE), and Geometric Mean (GM). The formulas for these metrics are as follows:
N 3 01l
N (3
N
1 N
1 9.)
MAE
RMSE
GM =
II
Osl
N
For the i-th sample, yi is the actual value, d is the predicted value, and N is the number of samples. Lower values of MAE, RMSE, and GM indicate better predictive accuracy.
C.3 In-context learning model Details
The model from Garg et. al. (2022) architectures from the GPT-2 decoder (ransformer with learnable positional embedding The PFN model from Müller et al. (2023) employs standard encoder transformer, without positional embedding. The detailed architecture is as follows:
C.4
Implementation Details
For the tabular datasets, we used baselines from scikit-learn (Pedregosa et al.) 2011 Specifically, for K-Nearest Neighbors (KNN), reported results using 10 neighbors, akin to localized method. For
15

---

Published in Transactions on Machine Learning Research (10/2024)
100
100
600
80
Number samples
Number of samp
of sammp
60
400
60
40
200
20
12.5
Targel
Tanget valluss
Target
(a)
BostLo
(b) Coucrete
(c) Abalone
175
250
150
1000
Number samples
2000
Number of samples
samples
800
100
600
75
400
200
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
Target valless
Target volues
Target values
(d) Kin8nm
(e) Communities
(l) Airfoil
Figure 7: Training distribution of labels for the tabular datasets.
Table Architecture details GPT2 and PFN models.
Model
Input Dimension
Embedding Size
Number Layers
Number Heads
GPT2
20
256
12
8
PFN
18
512
8
Decision Trees, Gradient Boosting, Linear Regression, Random Forest and Neural Networks, we used the default parameters.
and AgeDB-DIR, the baseline ResNet-50 backbone trained from scratch. For STSB-DIR, the bascline uses BiLSTM with GloVe word embeddings. The details on this datascts can be found in Yang et al. 2021).
For
Standard scaling was applied to the input features for these models. For in-context learning, we applied both standard scaling and a power transform to the features and then concatenated the two representations.
C.5 Computational Analysis
We conducted an analysis comparing in-weight learning and in-context lcarning approaches in terms of soitiing/processing
training/ processing time, inference time, and memory requirements. Table summarizes our findings for the IMDB-WE Wiki dataset. For in-context learning, the training time effectively zero, but we've included the time to process and extract features from the training set (189,794 images) using CLIP. The inference time includes both feature extraction and prediction. These results show that while the in-context learning method has a longer inference time, it requires less memory and significantly reduces the upfront training/ training/processing time.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Table 8: In-weight vs. In-context learning time analysis on IMDB- Wiki
Method
Train Process Time
Inference Time
GPU Memory
In-weight learning (ResNet-50)
8:12:00
0.01s
578M
In-context learning (GPT2 model)
2:12:33
0.06s
506M
This trade-off may be particularly beneficial in scenarios where quick adaptation new tasks is valuable, or when computational resources for extensive training limited.
Ablations
D.1 Sensitivity to context length size
In this section, conduct sensitivity analysis of the model to different context lengths. As shown in Table the averaged results across the six tabular datasets indicate consistent performance for 'small' number of neighbors.
Table 9: Sensitivity to the number of neighbors: Average results on tabular datasets
Metrics
RMSE +
Shot
all
many
medium
few
PFN localized k, K.
2.66
1.46
2.08
4.21
PFN localized k, K, 10
2.72
1.43
2.05
4.45
PFN localized k. Kg 15
2.68
1.38
2.03
4.38
PFN localized K. 20
2.66
1.38
2.02
4.39
GPT2 localized K, Kg
2.46
1.82
2.06
3.47
GPT2 localized b. K.- 10
2.35
1.72
1.95
3.31
GPT2 localized k. K, 15
2.32
1.94
1.89
3.20
GPT2 localized b. K. 20
2.40
1.92
2.03
3.32
E
Results
In this section, we report the average results and standard deviations for all nine datasets used in the paper. These results are obtained by rumning experiments with three different random seeds: 1, and 2. For clarity, neighbor selection is performed deterministically using the sklearn KNN implementation.
E.1 AgeDB-DIR
The complete results with the standard deviations for the AgeDB dataset are shown in Table 10
Table 10: Main results for AgeDB-DIR benchmark.
Metrics
MAE
GM
Shot
many
medium
few
many
medium
few
PFN localized 6.58 1 0.00 5.61 + 0.01 8.49 0.00 10.49 + 0.06
4.29
0.02 3.58 4 0.02 6.30 + 0.07 8.19 0.06
GPT2- localized 6.05 0.03 5.67 = 6.71 0.01 7.83
0.02
3.79
0.07
3.59
0.11 4.17 0.08 4.90 0.08
E.2 IMDB-WIKI-DIR
The complete results with the standard deviations for the IMDB-WTH WIKT dataset shown in Table 11
17

---

Published in Transactions on Machine Learning Research (10/2024)
Table 11: Main results for BMDB-WIKI-DIR
Metrics
MAE
GM
Shot
many
medium
few
all
many
medium
few
PFN localized 18.9 + 0.02 8.71 + 0.02 10.79 + 0.10 16.33 + 0.1215.26 + 0.02 5.17 + 0.01 6.00 + 0.12 9.42 + 0.22
GPT2 localized 7.76 + 0.05 7.35 + 0.04 11.15 E 0.06 17.71 + 0.25 4.29 + 0.06 4.13 + 0.06 5.96 E 0.07 11.00 0.35
E.3 STS-B-DIR
The complete results with the standard deviations for the STS-B-DIR dataset are shown in Table 12
Table 12: Results for STS-B-DIR.
Metrics
MSE
Shot
all
many
medium
few
localized
00.54 0.006 0.536
0.016 0.528 0.002 0.524 + 0.005 0.527 0.019 0.566 0.008
0.008 0.547 0.027 0.618
GPT2 localized
E.4
Tabular datasets
In this section report individually the results for each of the tabular datasets used. The results for Boston
(Harrison & Rubinfeld, 1978) are in Table 13 The results for Concrete (Yeh) 2007) are in Table 14 The same
applied for Abalone (Nash et al. [995) in Table 15
Communities (Redmond 2009) in Table 17 Kin8nm in
Table 16 and an enginecring design dataset: Airfoil
(Chen et al. 2019) in Table 18
Table 13: Average results the Boston dataset.
Learning IWL. ICL.
Metrics
RMSE 4
Shot
all
many
medium
few
Kmn
X
6.32 + 0.90
1.60 + 0.80
3.77
+ 1.02
9.39
+ 1.56
Decision Tree
5.40 + 1.53
2.71 - 0.71
5.39
2.62
5.83
- 0.70
Gradient Boosting
3.39 + 0.23
1.19
0.16
2.77
4.50
+ 0.01
Neural Networks
3.41
2.04 = 0.74
0.77
4.23
1.51
PFN localized (Ours)
X
3.90
+ 0.71
1.43 - 0.55
2.84 + 0.80
5.41
+ 1.11
GPT2 localized (Ours)
3.52
1 0.91
2.35 1 1.59
3.21 1 0.60
4.06
1 1.61
Table 14: Average results on the Concrete dataset.
Learning IWL. TCL.
Metrics
RMSE |
Shot
all
many
medium
few
Kun
X
12.39
+ 0.44
6.65
+ 0.92
10.36 + 1.67
19.91 + 1.14
Decision Tree
X
7.47
+ 0.23
5.78
+ 1.75
6.29
+ 0.72
10.93
+ 1.88
Gradient Boosting
X
6.58
+ 0.11
4.39
1.07
5.13
+ 0.38
10.66
4 0.71
Neural Network
x
6.99
+ 0.28
4.84
0.68
6.18 0.27
10.43 1 1.53
PFN localized (Ours)
7.02
+ 0.39
4.57
0.73
4.99
0.69
11.79
0.25
GPT2 localized (Ours)
5.85
0.09
4.54
0.35
5.16 0.48
8.31 + 0.41
18

---

Published in Transactions on Machine Learning Research (10/2024)
Table 15: Average results on the Abalone dataset.
Learning
IWL ICL
Metrics
RMSE 4
Shot
all
many
medium
few
Knn
X
4.39
+ 0.03
1.57
0.07
3.42 - 0.19
7.57 - 0.08
Decision Tree
X
4.48
+ 0.14
2.28
+ 0.07
3.66
+ 0.38
7.33
+ 0.28
Gradient Boosting
x
4.33
+ 0.04
1.51
+ 0.03
3.42 + 0.17
7.47
- 0.05
Neural Network
/
4.11 x
1 0.05
1.50
1 0.07
3.17 0.21
7.10
1 0.03
PFN localized (Ours)
/
X
4.54 + 0.01
1.83 - 0.09
3.60 + 0.13
7.72
+ 0.12
/
GPT22 localized (Ours)
x
4.16 + 0.12
2.99
0.18
2.80 = 0.24
6.59
+ 0.32
Table 16: Average results on the Kin8um dataset.
Learning IWL. TCT.
Metrics
RMSE +
Shot
all
many
medium
few
Kun
X
0.17 + 0.01
0.11 + 0.01
0.19 + 0.01
0.28 + 0.01
Decision Tree
x
0.24
- 0.01
0.20 - 0.01
0.25
+ 0.02
0.32 - 0.01
Gradient Boosting
X
0.24
+ 0.01
0.16
+ 0.01
0.27
+ 0.01
0.37 1 0.01
Neural Network
x
0.09
+ 0.01
0.07 - 0.01
0.09
- 0.01
0.14 + 0.01
PFN localized (Ours)
x
0.18
+ 0.01
0.13
0.01
0.18 + 0.01
0.28 0.01
GPT2 localized (Ours)
V
X
0.13 + 0.01
0.10
0.01
0.13 + 0.01
0,20 0.01
Table 17: Average results on the Communities catasset.
Learning IWL ICL
Metrics
RMSE 1
Shot
all
many
medium
few
Knn
/
0.21
0.00
0.07 1: 0.01
0.12 1 0.00
0.27 1: 0.01
Decision Tree
0.27
+ 0.01
0.10
= 0.02
0.16
0.02
0.34 - 0.01
Gradient Boosting
<
X
0.21
+ 0.01
0.05
+ 0.00
0.10
0:10+0000
0.27 + 0.01
Neural Network
X
0.22
+ 0.00
0.09
= 0.02
0.13
0.28 1 0.00
PFN localized (Ours) GPT2 localized (Ours)
/
0.21
0.00
0.09
= 0.01
0.09 + 0.00
0.28
0.01
0.19
0.00
0.10
= 0.00
0.10 + 0.01
0.23 - 0.01
Table 18: Average results on the Airfoil dataset.
Learning IWL ICL
Metrics
RMSE +
Shot
all
many
medium
few
Knn
0.27
+ 0.00
0.26 = 0.00
0.56 + 0.00
1.06 + 0.00
Gradient Boosting
X
0.25
+ 0.00
0.25
+ 0.00
0.54
+ 0.00
1.03
+ 0.00
Decision Tree
<
X
0.34
+ 0.00
0.33
- 0.00
0.56
+ 0.00
0.97
+: 0.00
Neural Network
/
x
0.12
1 0.00
0.11
1 0.00
0.27
1 0.00
0.56 1 0.00
PFN localized (Ours)
/
0.21
0.00
0.20
- 0.00
0.48 + 0.01
0.84 + 0.02
GPT2 localized (Ours)
/
0.23
0.00
0.23 = 0.00
0.28 + 0.01
0.47 + 0.00
19


Reviews:
[{"review_id": "Xj13R7Tr9R", "reviewer": "Reviewer_cCTG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The skewed distributions of quality in engineering design is a great motivating example. This is an important issue in automated design of anything of especially high quality.\n- The general approach of using only the nearest training samples to build the context prompt is simple, intuitive, and practical. I expect it will be used by many practitioners. I’d even guess it is being used by some using LLMs for regression, but without the motivation and theoretical context this paper develops.\n- Figure 1 is clear and compelling; a great way to immediately see the problem.\n- The empirical results are convincing.\n\nWeaknesses:\n- Value of theory is not clear\n- Motivation for augmented sampling is not quite clear\n- Some experimental details are missing\n- Some thought-provoking experimental behavior is not explained", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper argues that in-context learning is naturally well-suited to imbalanced datasets, particularly for regression. It points out that existing work on addressing imbalanced data is on classification tasks, not regression. Starting with existing pre-trained transformer-based regression models, the paper develops an approach that uses only the nearest training samples in the context prompt (both from the original dataset and an inverse density version of it), to avoiding biasing towards majority regions. A theoretical motivation for the approach is provided, and experimental comparisons demonstrate an advantage of the proposed method over existing approaches to regression on imbalanced datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested Changes/Clarifications:\n\nMethod and Theory:\n1. In what sense is Figure 1 showing “theoretical error bounds”? Aren’t these empirical expectations of errors computed from the datasets? Please clarify.\n2. Clarify what is meant by “accurate retrieval of neighbors”. Why is this a hard problem? Isn’t deterministically choosing the nearest neighbors from the training set “accurate”? I think the word “accurate” make this awkward; consider rephrasing.\n3. In 3.2.3 and 3.3, clarify which in-context learning method is used and plotted in Figures 2 and 3.\n4. Theoretical motivation for the augmented dataset is missing, i.e., compared to just using the inverse density dataset. Is just using the inverse density dataset equivalent to Downsampling in the ablations? If not, it should be one of the ablations, especially since downsampling does quite well in the `few’ category in the ablations. Is the idea of augmented that it trades off in a balanced way between downsampling and vanilla? Figure 6 in the appendix also does not explain why the method does not simply use the Inverse Density dataset alone.\n5. It should be clarified what the theory says beyond standard k-nearest neighbors regression theory. E.g., it suggests there is an optimal intermediate k, but does it suggest anything specific beyond that? Is the paper claiming that this is a novel theoretical perspective that also applies to standard k-nearest neighbors?\n\nExperiments:\n1. It would be valuable to also compare to RF and linear for a sense of where the method fails compared to standard off-the-shelf approaches.\n2. The paper says three “different random seeds, between 0 and 3” were used. Which three?\n3. Under “Preprocessing of inputs…” Should \"$\\tilde{k}_s = \\tilde{k}_s =$\" be \"$k’_s = \\tilde{k}_s =$\"?\n4. What is “VANILLA” in Tables 1 and 2 and Figure 4? This should be clarified before the Ablations section. It looks like an acronym in all caps.\n5. Clarify and/or hypothesize why PFN not as strong as Gradient Boosting in the Tabular datasets.\n6. Are the neighbors sampled according to a distribution or selected deterministically given the augmented dataset?\n\nMinor changes:\n\n- Fix quote directions, e.g., ‘Prior-data Fitted Networks’ -> \\`Prior-data Fitted Networks’, ‘closest’ -> `closest’, etc. “Prior Fitted Networks,” -> ``Prior Fitted Networks’,’ …\n- “…covering two facial ages, and one text similarity estimation, as well as…” -> “covering two facial age estimation tasks, one text similarity task, and…”\n- Fix citations: e.g., missing parentheses around many citations (probably \\citet v. \\cite or \\citep), and “the authors in Wang & Wang (2024) leveraged” -> “Wang & Wang (2024) leveraged”, “authors in Garg et al.”, … i.e., can generally remove “authors in”\n- “…each chunk To ensure…” -> “…each chunk. To ensure...”\n- “…expect for the…” -> “…except for the…” \n- “et al. (2024)” overlapping other text in Table 1.\n- Clarify what bold and underline indicate in tables.\n- Figure 4 should be labeled as a Table.", "broader_impact_concerns": "Using age estimation as a motivating domain raises potential ethical concerns. Any machine learning system can be used for unethical purposes, but for age estimation in particular it is particularly easy to imagine invasive applications. It could be helpful for the authors to mention why they might want to do age estimation. For example, they even mention that the reason age estimation of young people is hard is do to “ethical limitations”, so why would we want to be able to estimate the age of young people?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dQbuxj0HQU", "reviewer": "Reviewer_wjE2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n>- This paper explores a new learning paradigm to address the imbalance regression problems, which brings new insight into the community and highly promotes the development of long-tailed learning.\n>- The authors establish a theoretical error bound for the expected error when employing in-context learning for imbalanced regression, >showing that a substantial number of context examples could theoretically approximate the actual predictive distribution. However, they also highlight a critical issue where utilizing the entire training set as context can result in model bias towards the majority regions, impairing the performance on infrequent labels.\n>- Extensive experiments on various datasets demonstrate the efficiency of the proposed method.\n\n**Weakness**\n \n> - Although IM-Context has improved compared to competitors, it relies on pre-trained in-context learning models. This trade-off warrants further discussions, such as inference time and required memory space.\n> - Please elaborate on the backbone and implemental details of competitors. Is the backbone used by competitors consistent with the IM-context method?\n> - It is important to acknowledge the significant contributions of several influential studies in the realm of in-context learning [1][2], as well as those focused on long-tail learning [3][4][5], which have shaped our understanding and exploration in these fields. The authors are suggested to discuss these works in this paper.\n\n**References**\n\n>- [1] Birth of a Transformer: A Memory Viewpoint. NeurIPS 2023. \n>- [2] The Mechanistic basis of data dependence and abrupt learning in an in-context classification task. ICLR 2024.\n>- [3] Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition. NeurIPS 2022.\n>- [4] A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning. NeurIPS 2023.\n>- [5] Harnessing Hierarchical Label Distribution Variations in Test Agnostic Long-tail Recognition. ICML 2024", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\nThis paper introduces a novel approach to address the challenge of imbalanced label distributions in regression problems. Traditional deep learning models often struggle with generalization in regions of high imbalance due to overfitting on underrepresented data. The proposed IM-Context method leverages in-context learning, where a model conditions itself on a sequence of input-label pairs to make predictions without parameter updates. The study demonstrates that localized context is crucial for reducing bias in imbalanced areas and shows that in-context learning outperforms conventional methods, especially in minority regions, across various real-world datasets. Theoretical analysis and empirical results highlight the effectiveness of this paradigm in improving predictive performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please finish the discussion required in the **weakness**", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BU2Xn3UXui", "reviewer": "Reviewer_nEPj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an important problem of regression with data imbalance. This is of interest to a lot of problems in real world scenarios. \n- The empirical investigation on using in-context learning to solve the task is interesting. \n- The authors of the paper seems to demonstrate the effectiveness of the proposed method.\n- The paper is well written and easy to follow for the most part. \n- The proposed method is technically sound. \n\nWeaknesses:\n- The proposed method is pretty simple and straight-forward. \n- Based on my understanding, the theoretical derivations doesn't compare IWL vs. ICL? Any theoretical justification on why ICL is better than IWL? \n- The proposed method is not consistently good. For example, the proposed method is not the best performing one for Table 2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel approach for imbalanced regression using in-context learning. Key contributions include:\n\n- The paper investigates in-context learning where models use prompt sequences for predictions to tackle the problem of regression with imbalanced dataset, eliminating parameter updates.\n   \n- The paper highlights the importance of localized context in reducing bias in imbalanced regions through theoretical and empirical studies.\n\n- The paper demonstrates superior performance of in-context learning on real-world datasets with high imbalance.\n\n- The authors of the paper show that in-context learning mitigates overfitting in underrepresented regions.\n\nThese findings provide a robust alternative to conventional imbalanced regression methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I think the \"retrieving neighbors\" need more details. The subsection seems a little abrupt and disjoint from the rest of the section, in my personal opinion.\n- Do we do finetuning for IWL? What if we finetune the model instead of ICL? How do results compare? If not, this might be an interesting comparison.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that the paper should be accepted and will be a valuable contribution.

The updates to the paper so far are much appreciated and significantly improve the clarity of the work. There are still some minor typos that should be addressed in a minor revision:

- Citations must be updated to include parentheses when appropriate, as noted by one of the Reviewers
- “retrieving semantically relevant as context” -> “retrieving semantically relevant examples as context”
- “To the best of our knowledge, we are not aware of existing references that specifically address this issue in the context of imbalanced regression with k-NN” -> “To the best of our knowledge, there is no existing work that specifically addresses this issue in the context of imbalanced regression with k-NN”
- ‘Inverse’ -> `Inverse’
- ‘inverse sampling’ -> `inverse sampling’
- “Gradient Boosting,Linear Regression” -> “Gradient Boosting, Linear Regression”

Please upload this revision when you have the chance.

Thanks!
---AE
Decision: Accept (poster)