GIT-BO: High-Dimensional Bayesian Optimization with Tabular Foundation Models
Authors: Rosen Ting-Ying Yu
Cyril Picard
Faez Ahmed
Venue: ICLR 2026 Poster (2026)
Keywords: Tabular Foundation Models
Bayesian optimization
TabPFN
Primary area: foundation or frontier models, including LLMs
TL;DR: Tabular foundation models performs well as a surrogate for high-dimensional Bayesian optimization.

Abstract: Bayesian optimization (BO) struggles in high dimensions, where Gaussian-process surrogates demand heavy retraining and brittle assumptions, slowing progress on real engineering and design problems. We introduce GIT-BO, a Gradient-Informed BO framework that couples TabPFN v2, a tabular foundation model that performs zero-shot Bayesian inference in context, with an active-subspace mechanism computed from the model’s own predictive-mean gradients. This aligns exploration to an intrinsic low-dimensional subspace via a Fisher-information estimate and selects queries with a UCB acquisition, requiring no online retraining. Across 60 problem variants spanning 20 benchmarks—nine scalable synthetic families and ten real-world tasks (e.g., power systems, Rover, MOPTA08, Mazda)—up to 500 dimensions, GIT-BO delivers a stronger performance–time trade-off than state-of-the-art GP-based methods (SAASBO, TuRBO, Vanilla BO, BAxUS), ranking highest in performance and with runtime advantages that grow with dimensionality. Limitations include memory footprint and dependence on the capacity of the underlying TFM.

Published as a conference paper at ICLR 2026
GIT-BO: HIGH-DIMENSIONA BAYESIAN OPTIMIZA- TION USING TABULAR FOUNDATION MODELS
Rosen Ting- Ying Yu, Cyril Picard, Faez Ahmed Massachusetts Institute of Technology
Massachusetts, MA 02139, USA (rosenyu, cyrilp, faez @mi eduu
ABSTRACT
Bayesian optimization (BO) struggles in high dimensions, where Gaussian- process surrogates demand heavy retraining and brittle assumptions, slowing progress on real engineering and design problems. We introduce GIT-BO, a Gradient-Informed BO framework that couples TabPFN v2, tabular founda- tion model (TFM) that performs zero-shot Bayesian inference in context, with active-subspace mechanism computed from the model's own predictive-mean gradients. This aligns exploration to an intrinsic low-dimensional subspace via a Fisher-information estimate and selects queries with UCB acquisition, requiring no online retraining. Across 60 problem variants spanning 20 benchmarks- scalable synthetic families and eleven real-world tasks (e.g., power systems, Rover, MOPTA08, Mazda) up to 500 dimensions, GIT-BO delivers better performance- time trade-off than state-of-the-arr GP-based methods (SAASBO. TuRBO, Vanilla BO, BAxUS), ranking highest in performance and with runtime advantages that grow with dimensionality. Limitations include memory footprint and dependence on the capacity of the underlying TFM.
INTRODUCTION
Optimizing expensive black-box functions is central to progress in areas such as machine learn- ing (Dewaneker al., 2016; Snoek et al., 2012). engineering design (Kumar et al., 2024 Wang & Dowling, 2022] Zhang et al. 2020; Yu et al. 2025). and hyperparameter tuning (Klein et al., 2017 Wu et al.) 2019). Bayesian optimization (BO) has become the method of choice in these settings due to its sample efficiency. Yet, despite its successes, standard BO with Gaussian pro- cesses (GPs) is widely viewed as limited to diw-dimensional regimes, typically fewer than a few dozen variables (Liu et al.) 2020 Wang et al. 2023; Santoni et al. 2024). Scaling BO to hundreds of dimensions remains critical barrier, where the curse dimensionality. GP training costs, and hyperparameter sensitivity severely hinder performance. Research has sought to overcome these challenges through three main strategies: (1) Exploiting intrinsic low-dimensional structure, c.g., random embeddings and active subspaces (Wang et al., 2016 Nayebi al. 2019 Letham al., 2020 Papenmeier et al. (22 Additive functional decompositions (Kandasamy et al.) 2015
Gardner et al.) 2017 Rolland al.) 2018 Han al. 2021; Ziomek & Ammari [202]]; and (3) Al-
ternative GP priors and trust-region heuristics (Eriksson et al. 2019] Eriksson & Jankowiak 2021
Hvarfner et al. 2024 Xu et al., (025). These innovations push the Trontier but still facc two practical
obstacles: (1) prohibitive computation from iterative GP retraining, and (2) brittle reliance on hyper- parameter tuning. including determining appropriate intrinsic dimensionality and selecting optimal
kernels and priors (Rana et al. 2017 Letham et al. 2020 Eriksson & Jankowiak/ 2021).
Recent advances tabular foundation models (TFMs) provide radically different surrogate model- ing paradigm. Prior-Data Fitted Networks (PFNs) (Müller et al. [2022) Hollmann et al. 2002 Müller
ctt al. 2023) perform Bayesian inference in-context with frozen weights, eliminating kernel re- fitting and delivering 10-100x speedups on BO tasks (Rakotoarison et al. 2024 Yu et al. 2025 These approaches address computational bottlenecks by leveraging pre-tr models' in-context learning capability, which requires only a single forward pass at inference during optimization. These pow- erful TFMs trained on millions of synthetic prior data can also perform accurate inference without additional hyperparameter tuning for new domain.

---

Published as a conference paper at ICLR 2026
The newly released TabPEN v2 (Hollmann et al. 2025 extends this capacity inputs up to 500 dimensions, opening the door to foundation-mont time. However, prior analyses have shown that TabPFN v2, despite its strong performance on small- to medium-scale tasks, exhibits performance degradation in high-dimensional regimes, with recent work proposing divide-and-conguer or feature-extraction strategies mitigate these limitations (Ye) iet al. [2025) Reuter et al.) (025). This raises a fundamental question: Are frozen TFMs sufficient for high-dimensional optimization, or must they be coupled with classical algorithmic strategies to succeed?
surrogates for high-dimensional BO for the first
We answer this question by introducing Iraodien-Informed Bayesian Optimization using TabPFN (GIT-BO), a framework that integrates TabPFN v2 with gradient-infomme active subspaces. Our key idea to exploit predictive-mean gradients available from the frozen model itself construct low-dimensional gradient-informed subspaces. This provides al algorithmic guidance for adaptive ex- ploration while preserving the inference-time efficiency of TFMs. In doing so, GIT-BO aligns foun- dation models with classical subspace discovery, combining the speed of in-context surrogates with the structural power needed in extreme dimensions. We perform comprehensive algorithm bench- marking against the state-of-the af-theeart (SOTA) GPU-accelerated high- tigh-dimensional BO algorithms and test them on commonly used synthetic benchmarks as well as several real-world engineering BO benchmarks.
Our contributions are:
We propose GIT-BO, a gradient-infomen high-dimensional BO method that adaptively discovers active subspaces from frozen foundation model's predictive gradients, requiring no online retraining.
Across 60 diverse problems comprising synthetic and real-world benchmarks (including power systems, car crash, and dynamics control), GIT-BO consistently achieves state-of- the-art optimization quality with ceders-of-magnitue runtime savings compared to GP- based methods.
Our results demonstrate that foundation model surrogates, when paired with structural algorithmic guidance, emerge as viable and competitive alternatives to paaassinnproces--baeeeee BO for high- dimensional problems. Beyond the core algorithm, we conduct extensive ablation and diagnostic studies to understand when GIT-BO works and why. We (i) compare gradient-informed subspaces to alternative projections such as trust regions and BAxUS- acquisition rules (UCB vs. EI), subspace dimension and sampling schemes, and initialization size, and (iii) evaluate GIT-BO with GP surrogates and with fine-tuned TabPFN variants. These analyses, summarized in Section ( and detailed in Appendices B and C show that our gains are not due to single design choice and that the GI-subspace mechanism improves both TFM- and GP-based BO.
embeddings, (ii) study the impact of
BACKGROUND
2.1
BAYESIAN OPTIMIZATION
Bayesian optimization is a sample-efficiens approach for optimizing expensive black-box functions where the objective is to find x= E arg max f (x) with X' = [0, 1] achieved by sequentially querying promising points under the guidance of surrogate model. Gaussian processes (GPs)
main the dominant surrogate due to their effective uncertainty- -bsed exploration and exploitation, but their cubical computational scaling and deteriorating performance with increasing dimensional-
ity pose serious challenges (Liu et al. 20200 Wang et al. 2023: Santoni et al. 2024 Ramchandran
al. 2025). Three main families address these issues:
Exploiting intrinsic low-dimensional structure. A common strategy in high-dimensional BO is to assume the objective depends on only few effective directions and to project the search into that subspace, where GPs perform more reliably. REMBO introduced random linear projections (Wang et al. 2016). while HESBO (Nayebi et al., 2019) and ALEBO (Letham et al. 2020) refined this idea using sparse embeddings and Mahalanobis kernels. More recently BAxUS (Papenmeier al.)
2022]. adaptively expands nested subspaces with guarantees. These succeed when a meaningiul active subspace exists, but degrade when structure is weak mis-specified.
2

---

Published as a conference paper at ICLR 2026
Additive decompositions. Another approach assumes the objective decomposes into sum of low-dimensional components, enabling separate GP models. Additive GPs use disjoint decompo- sitions (Kandasamy al.) 2013). while later work allows overlaps (Rolland et al. 2018) or tree- structured dependencies (Han et al., 2021) to improve tractability. Randomized decompasitions offer a lightweight alternative (Ziomek & Ammar 2023). These methods improve sample efficiency since each component is easier to model, but remain limited by the difficulty of discovering the right decomposition from sparse data and the overhead of structure learning, restricting adoption in practice (Rolland et al. 2018 Han et al. 2022 Ziomek & Ammar 2023).
Alternative modeling and trust-region strategies. Beyond embeddings and additive decompo- sitions, another line of work rethinks the surrogate itself. SAASBO introduces (Eriksson & shrinkage priors GP length-scales to identify relevant dimensions automatically
Jankowiak 2021) ). TuRBO (Eriksson et al.] 2019) replaces global modeling in favor of multiple local GP surrogates confined dynamically adjusted trust regions. More recently, studies show that vanilla BO with carefully chosen priors (Hvarfner et al. (024) and standard GPs with robust Matérn kernels (Xu al. 2025) can remain competitive high dimensions.
Despite these advances, such methods still depend on seah--oo-oo-dimensione learning, sensitive kernel choices, or strong structural assumptions mootivating foundation model surrogates as fun- damentally different path forward.
2.2 TABULAR FOUNDATION MODELS AS BO SURROGATES
Tabular foundation models (TFMs) provide amortized Bayesian inference through in-context learn- ing (ICL). Prior-Data Fitted Networks (Müller et al. [022] Hollmann al.) 2022 2025) are transformer-based TFMs trained on massive synthetic priors. At inference time, the observed dataset of BO evaluations is fed as the context input, which acts the optimization history. Each new sam- ple is appended to this context, and a single forward pass produces updated predictive means and variances. Thus, although PFNs have frozen parameters, their predictions adapt dynamically to the growing context, mimicking Bayesian updating without retraining (Müller et al. 2023 Rakotoarison al. 2024).
This approach eliminates the iterative kernel re-fitting required by GPs, yielding 10-100x speedups in various BO applications (Müller et al., 2023, Rakotoarison al. 2024 Yu al. 2025). However, PFNs cannot explicitly tune kernels or priors, which limits their ability to exploit low-rank structures when dimensionality grows. Moreover, recent analyses reveal that while transformer-basee exhibit vanishing variance with larger contexts, their bias persists unless explicit locality is enforced, resulting in degraded accuracy in high- dige-dimensional regimes (Nagler 2023). Although TabPEN v2 extends the model's capability to regression tasks with up to 500-dimensional inputs, its predictive performance still deteriorates without additional structural guidance (Ye et al.] [2025] Reuter et al.) 2025). These limitations highlight the necessity of incorporating additional guidance sustain the effectiveness of TabPFN v2 in nigh-dimensional BO.
PFNs
2.3 DISCOVERING EMBEDDED SUBSPACES: CLASSICAL AND DEEP LEARNING PERSPECTIVES
Since PFNs are frozen models, discovering intrinsic low-dimensional subspaces is the most viable high-dimensional BO strategy requiring no fine-tuning of the foundation model. Classical applied mathematics offers a principled method that we can leverage here. Active subspaces (Constantine al. 2014) use gradient covariance to identify influential directions. while ieeeihoood-nffrmmnn sub- spaces (Cui et al. 2014) detect posterior-sensitvv directions. Spectral approaches such as Laplacian eigenmaps (Belkin& & Niyogi 200T) learn nonlinear embeddings. Recent advances provide certified gradient-based dimension-reductie
methods, showing that the leading eigenvectors of Fisher-type gradient covariance matrices recover low-dimensional, high-information subspaces (Pennington & Karakida et al., 2019 Zahm et al. 2022 Li et al. 2024 2025). In contrast, deep learn- ing methods typically learn mappings into latent manifolds, e.g., variational autoencoders designed for BO (Tripp et al. 2020)
Worah 2018 8:
analyses of neural representations (Li et al.) 2018 Ansuini et al. (2019). These approaches require training additional models, which conflicts with the
TFM paradigm of fast inference without retraining.
3

---

Published as a conference paper at ICLR 2026
The literature suggests potential synthesis: pair the inference-tine efficiency of TFMs with structure discovery to address high-dimensional optimization. This leads to our central con- tribution: Gradient- predictive-mean gradients from TabPFN v2 to estimate gradient-informed active subspace, then performs acquisition-driven search within that subspace. This design (i) avoids GP retraining and heavy hyperparameter tuning, and (ii) supplies the locality and structure that TFMs lack in there
Tradient-Informed Informed Bayesian Optimization using TabPEN (GIT-BO), which extracts
high-ddmensionall thereby targeting the exact failure modes surfaced above.
THE GIT-BO ALGORITHM
GIT-BO consists of four main components: the surrogate model (TabPFN v2), the gradient-based subspace identification, an upper confidence bound acquisition function, and a method that combines these for high-dimensional optimization.
Tabular Foundation
Get Principal subspace (1-) From Fisher Information Matrix (H)
3. Frigh-Dimensional Bayesian Optimization
Model Prediction
Samples
Observed
Data (Prior)
TabPENv2
Estimated Gradients
Project Bocktt
V,2 R
Search the Subspace
Vese
Dimansion
Original
Acquisition Function
Predictiva Mean Unsartainties
o(()
Append next sample Xneut (*) and iteratively search
Figure 1: GIT-BO algorithm overview. The method operates in five stages: (I) Initial observed sam- ples are collected in the tigh-dimensional space IR D: (2) TabPFN v2, fixed-weight tabular foun- dation model, generates predictions of the objective space at inference time using in-context learn- ing: (3) The gradient from TabPFN's forward pass (V (v((z)) is used to identify low-dimensional gradient-informed (GI) subspace. The predicted mean and variance are used for acquisition value calculations o( (x);; (4) The next sample point (Inexs) is selected from GI subspace's projection back to the high-dimensional space (V,2) with the highest acquisition value; (5) Appended T'next to the "context" observed dataset for iterative search until stopping criteria is met.
3.1 SURROGATE MODELING WITH TABPEN
We use TabPFN v2, a 500-dimensional TFM from Hollmann et al. 2025), as the surrogate model for Bayesian optimization framework. TabPFN leverages in-context learning provide dynamic predictive posterior distribution conditioned on the observed dataset Dobs = { y/) ) which expands iteratively during optimization. At each iteration, we random sample (from Sobol sequence) huge discrete set of candidate points X {zz 1 (where (10k-n) 2 m >>> n. TabPFN can
take at most 10k samples) from the search domain X' C TRD to approximate the continuous search space. TabPEN (qo) processes both the context set (Dot.) and candidate set (Xeand) simultaneously. generating predictive means Mm (x) and variances of (x) for the search space formed by all can- didates in a single forward pass (2), o², (x) ~ qo (Yaand (aaadd, Dobs)). This efficient, adaptive inference step enables rapid identification of promising regions in high-dimensional optimization problems without surrogate retraining.
3.2 GRADIENT- INFORMED ACTIVE SUBSPACE IDENTIFICATION AND SAMPLING
To identify an active subspace for efficient exploration, we leverage gradient information obtained from TabPIN's predictive mean, defining g(x) : 2 (z) via a single-step backpropagation. Following gradient- baaded subspace methods in inverse problems and Fisher-eigenstructure analy- that show small number of dominant, high high-sensitivity directions (Pennington & Worah] 2018 Karakida et al. 2019 Zahm et al. 2022; Li et al., 22255 2024; Ly et al. (017), we form the empir- This matrix captures the local sensitivity structure of the
ical Fisher matrix H E2
4

---

Published as a conference paper at ICLR 2026
predictive model. The algorithm then selects the top eigenvectors of H as the gradient- active subspace (GI-subspace) V, IR..
For all the results presented in Section and Appendix D selected fixed 10 for experiments. Ablation studies on the effect of GI-subspace on BO performance and the selection of presented in Appendix [.I
Next, we uniformly sample m candidate points for exploration from the low-dimensional (r- dimensional) hypercube 2 ~ U([-1, 1)") and map these back to the original high-dimensional space via:
XGI = Zref + V
the m candidates are centered around the centroid of observed data, Tref = Tobs. which guides the search towards promising regions discovered far, while the acquisition function later promotes exploitation. These generated candidates. XaI. are then evaluated using the acquisition function to select the next point sample. The theoretical detail and experimental results for the GI subspace are in Appendix A and
3.3 ACQUISITION FUNCTION
We adopt the Upper Confidence Bound (UCB) as acquisition function, heuristics BO and PFN- based BO both use in previous studies (Srinivas et al. 2010 Xu et al. 2025 Müller et al.
2023 UCB selects points by maximizing aucs [uz) (x) where ((x) denotes the sur- rogate's 's predictive mean, o (zx surrogate's 's predictive standard deviation, and 3 represents the exploration level. In GIT-BO algorithm, plxx) and (x:) are the TabPFN predictive mean and
standard deviation given data Dobs, and the / is set to 2.33. Further details of / ablation are in Appendix(B.. with theoretical analysis Appendix [A]
Putting everything together, Figure | and Algorithm I outline the GIT-BO procedure combining TabPFN with gradient-informed subspace search. The technical implementation details of GIT-BO are stated in the Appendix
Algorithm 1 Gradient-Infonne Bayesian Optimization using TabPFN (GIT-BO)
Require: objective f. domain X' c RP, initial sample size Rg. iteration budget I, subspace dimen-
sion r, a acquisition function
1: Draw LHS points 2; and set y = f(;);; D, +- ());
2: for = to / do
3: Pm. - <- Run TabPFN in-context on D,, and predict Xeand randomly sampled from Sobol
4: Calculate backprop gradient V aa (x) from TabPFN's in-context learning of Dn,
5: Form approximated Fisher matrix H V, 2 (z) Valm(()) [1
6: V, + top-r eigenvectors of H
7: XaI <- Fief + V,z, with uniform sampled from the low-dim hypercube U([-1,1)))
8: Trext <- arg max, (XG)))
9: Evaluate Faex = f (#nex) and append the query point data Dn + Dr, {F Nooxe)
10: end for
return x= arg (ax
EXPERIMENT
This section outlines our empirical approach to evaluating and comparing different high-dimensional Bayesian optimization algorithms, highlighting the assessment of different algorithms' performance across a large number of complex synthetic and unique engineering benchmarks. To conduct a fair, comprehensive comparison, we benchmark GIT-BO against four other algorithms from the state-of- the-art BO library, BoTorch (Balandat et al. 2020), on 60 problems, and conduct statistical ranking evaluation experiment trials.
5

---

Published as a conference paper at ICLR 2026
4.1 BENCHMARK ALGORITHMS
We benchmark GIT-BO against random search (Bergstra & Bengio 2012) and four high- dimensional BO methods, including SAASBO (Enksson Jankowiak 2021). TuRBO (Erksson 2019), Vanilla BO for high-dimensional (Hvarfner et al. (024). and BAxUS (Papenmeier et al.
et al. 2020
2022) from the state-of-the-art (SOTA) Py PyTorch-based BO library BoTorch (Balandat et al.)
To ensure a fair comparison with our GPU-accelerated GIT-BO framework, we deliberately selected only algorithms that can be executed efficiently on GPUs, as runtime scalability is a central evaluation criterion. All methods were run on identical compute resources (one node with the same CPU and GPU specifications), and additional implementation details are provided in Appendix
4.2
TEST PROBLEMS
This study incorporates diverse set of high-dimensional optimization problems, including 9 syn- thetic problems and 11 real-world benchmarks. Synthetic and scalable problems include: Ackley, Rosenbrock, Dixon-Price, Levy, Powell, Griewank, Rastrigin, Styblin-Tang, and Michalewicz. We note that this set of synthetic functions is taken from BoTooc (Balandat et al. 2020) with their de- fault setting, and therefore all the baseline algorithms from BoTorch have been tested on this set of synthetic functions.
The rest of the application problems collected from previous optimization studies and confer- ence benchmarks: the power system optimization problems from CEC2020 (Kumar et al.) 2020), Rover (Wang et al. 2018), MÓPTA08 car problem (Jones 2008), two Mazda problems (Ko- hira et al. 2018). 1), and Walker problem from MuloCo Todorov et al. 2012). As this study focuses on the high-dimensional characteristic of the problem, we make all our benchmark problems sin- gle and unconstrained for testing. Therefore, we have applied penalty transforms to all real-world problems with constraints and performed average weighting the two multi-objective Mazda prob- lems. Among the 20 benchmarks, 10 (Synthetic Rover) are scalable problems. To evaluate the algorithms' performance with respect to dimensionality, we solve the scalable problems for D (100, 200, 300, 400, 500). Therefore, we have experimented with total of 5 x 10 + 10 = 60 different variants the benchmark problems. Details benchmark selections and their implemen- tation details are listed in Appendix
4.3 METHODS FOR ALGORITHM EXPERIMENT
The algorithm evaluation aims to thoroughly compare GIT-BO to current SOTA Bayesian opti- mization techniques. This study focuses on maximizing the objective function for the given test problems. For each test problem, distinct random seed. To ensure fair comparison, we initialize each algorithm with an identical set of 200 samples, generated through Latin Hypercube Sampling with consistent random seeds across all trials. During each iteration, each algorithm selects one sample to evaluate next.
experiment consists of 20 independent trials, each utilizing
To execute this extensive benchmarking process, utilized distributed server infrastructure fea- turing Intel Xeon Platinum 8480+ CPUs and NVIDIA H100 GPUs. For all algorithms, each indi- vidual experiment (run) was conducted with the same amount of compute allocated: a single H100 GPU node with 24 CPU cores and 224GB RAM.
4.4 EVALUATION METRICS
Optimization Fixed-budget Convergence Analysis Fixed-budget evaluation is standard tech- nique for comparing the efficiency optimization algorithms by allocating a predetermined amount of computational resources for their execution (Hansen et al. 2022). In our study, we adopt fixed- iteration budget, running all algorithms (GIT-BO, SAASBO, TuRBO, Vanilla BO, and BAxUS) for 500 iterations. We report performance using plots of average regret versus the number of function evaluations (iterations), which illustrate the convergence behavior of each algorithm.
In addition, measure the wall-clock runtime of each algorithm capture efficiency in terms of computational cost, we report plots of average regret versus elapsed runtime (in seconds).
the same 500 iterations. To

---

Published as a conference paper at ICLR 2026
Statistical Ranking To comprehensively compare and evaluate the performance of the Bayesian optimization algorithms, statistical ranking techniques are employed instead of direct performance measurements of the optimization outcome. In this study, define the optimization performance result as the median of the optimal found across the 20 optimization trials of each algorithm. By sta- tistically ranking the results, were able standardize the comparisons across different problems, since various optimization challenges can produce objective values of vastly different magnitudes. Furthermore, using this ranking allowed data points that might influence our evaluation.
reduce the distorting effects of unusual or extreme
We conduct our statistical analysis using the Friedman and Wilcoxon signed-rank tests, comple- mented by Holm's 's alpha correction. These non non- parametric approaches excel at processing bench- marking result data without assuming specific distributions, which is critical for handling optimiza- tion results with outliers. These statistical methods effectively handle the dependencies in our setup, where we used the same initial samples and seeds to test all algorithms. The Wilcoxon signed-rank test addresses paired comparisons between algorithms, while the Friedman test manages problem- specific grouping effects. For multiple algorithm comparisons, we used Holm's alpha correction to control error rates (Wilcoxon) 1945 Holm 1979).
5 RESULTS
Overall Statistical Ranking and Algorithm Runtime Tradeoffs Across all benchmark variants, F (a) shows that GIT-BO achieves the best overall statistical performance rank (1.92) across 60 problems, consistently outperforming competing baselines in terms of final solution quality af- ter 500 iterations. In terms of computational cost, Figure 2 (b) demonstrates that GIT-BO remains runtime-competitive despite its stronger optimization performance. To provide further insight, Fig- c)) d) decompose performance by problem class: BAxUS achieves the hest ranking on synthetic benchmarks, whereas GIT-BO dominates real-world engineering tasks. This contrast underscores that methods excelling on or even fine-tuning toward synthetic tests may not generalize practical applications.
The joint performance-runting tradeoff is visualized in Figure 3 (a), where GIT-BO and TuRBO both lie on the Pareto frontier: GIT-BO attains superior optimization quality, while TuRBO provides a speed advantage. Finally, Figure 3 (b) tracks the evolution of average algorithm rank over itera- tions, showing that GIT-BO rapidly rises to the top within the first 50 iterations and maintains its lead thereafter. Together, these results highlight GIT-BO as the most balanced method, achieving state-of-the-arr performance while retaining favorable computational efficiency.
(a)
(b)
Algorithm Performance Rank
Runtime Rank
3
2
Random SAASBO
GIT-BO
SAASBO
Random TurBO Vanilla_BO
BAXUS
BAXUS
Vanilla_Bo BB 3
TurBO
GIT-B0 333
(c)
(d)
Synthetic Problems Rank
Real-world Problems Rank
4
3
2
1
Random
$55
GIT BO
Random 5.71
GIT-BO 5: TurBO
SAASBO
BAXUS
SAASBO 4M
Vanil a BO
TurBO
BAXUS
Vanilla 80
Figure 2: Statistical ranking across benchmark problems. (a) Overall algorithm optimization perfor- mance of all 60 problems (synthetic + real-world) ranking based on final solution quality at iteration 500. (b) Algorithm runtime ranking of all 60 problems (synthetic + real-world) based on the time takes for 500 iterations of optimization. (c) and (d) Optimization performance ranking on only synthetic and only world benchmark subsets, respectively.
7

---

Published as a conference paper at ICLR 2026
(a)
Pareto Frontier Plot (b)
Rank Iteration
Runtime Rank
Average Rank
BAXUS
4
4
TurB0
Vanilla B0
SAASBO
Random Search
2.5 5.0
Performance Rank
200 Iteration
Figure 3: (a) Pareto frontier plot of runtime rank vs. performance rank (lower is better) over 60 benchmark problems. (b) Evolution of average algorithm rank over iterations, showing that GIT-BO converges rapidly to the top within 50 iterations and sustains its lead.
GIT-BO
BAXUS
TurBO
Vanilla BO
SAASBO
Random Search
Ackley (D-100)
Ackiey (D=200)
Ack ey (D-300)
Ackley (D=400)
Ackley (D-500)
Arange Nogrst
Aereon Rearet
Deeeun Regret
20
20
200
400
0
200
400
200
400
200
400
200
400
Kumber teneloons
Number Rerations
Number Ferations
Number lterations
ors
(D=100)
(D=200)
StyolinskiTang (D=300)
DixonPrice (D=400)
Restrigin (D=500)
Aecogg Hogee
10%
10"
10
100
1000
Aeeroge Rogret
10
10x
20
10a
110
107
10a
400 Numbar torations
200
200
400
200
400
200
400
200
400
humber of Ronations
Number har zions
Number Itarations
Number lterations
Rover (D-100)
Active Power (AP) Loss (D-153)
Carlx2) Mazda (D-148)
Carbx31 Morda (D-222)
Walleer Policy (D-102)
Acceaee hapet
Averoon Recret
Avcrage Regret
1.0
101-
-0
10
1.0
10P
100
200
200
400
200
Figure 4: Average regrets vs. iteration convergence on a subset benchmarks (10 synthetic real-world) comparing our method against SOTA high-dimensional BO algorithms. Full statistical tests and per-problem plots for problems are provided in the Appendix D
Convergence Performance Figure summarizes the convergence plots across representative set of 15 synthetic and engineering benchmarks iterations, and plotts the average regret against the algorithm's elapsed runtime, with shaded regions indicating the 95% confidence inter- val. Due to page number limitations, the convergence plots for all sixty benchmark problems are reported Appendix D For the Ackley function (100-500D). we observe that GIT-BO starts in the second performance tier but steadily improves relative to competing methods as dimensional- ity increases. Unlike GP-based approaches such as TuRBO, whose performance deteriorates with higher D, GIT-BO maintains stable convergence rates, suggesting that TabPFN's universal modeling capacity generalizes robustly even extreme dimensions.
Across the broader set of synthetic problems, GIT-BO achieves top-ranked regret curves in most cases, including Rosenbrock (200D), Dixon-Price (400D), and Rastrigin (500D). However, its fail- ure on Styblinski-Tang highlights the distributional limits of the TabPFN pre-training regime, an example where GP-based surrogates still dominate. On the engincering side, GIT-BO again demon- strates strong performance, consistently outperforming baselines power system tasks and auto- motive design benchmarks, while struggling with the Rover problem.
Convergence performance when considering runtime Figure 5 the trade-off becomes even more pronounced. Methods such BAxUS match or
When runtime taken into account in
8

---

Published as a conference paper at ICLR 2026
GIT-BO
BAXUS
TurBO
Vanilla BO
SAASBO
Random Search
Ackley (D-100)
Ackley (D=200)
Ackley (0-300)
Ackley (D-400)
Ackley (D=500)
Nacrets
Averyye Reyrets
Average Average Royrets
20²
10*
10'50"10'10
10"10'1""00
10"10"1'11"
10"10"0010"
(scconds)
Time (scconds)
(D=200)
Styblinski Tang (D=300)
DixonPrice (D=400)
Restrigin (D=500)
Avsrage hegeet
10.
10'
10
fowenge Beprey
10
10
10rd
110
10
10t
10
10'
10a
0 10"10"10"10"
0.6" 16"16"10"10"
0 10'10"10"10"
0.0 10"10"10"10"
0. 16"10"10"10"
Time (secorsds)
Time (seconess)
(secondu)
Time (arconda)
Time (secunds))
Rover (D-100)
Art YVA Power (AP) Lass (D=153)
Carix2) Mazda (D-148)
Carixa] Mazda (D=222)
Wa ker Palicy (D=102)
Ausroago Haquess
Averry Regres
10
102.
10
20a
0.00 10'10'10"10" Time (scconds)
10.0% 10'10-10'10"
010 10'10'10'10" Time (seconds)
010 10'10"10"
00.0 10'10" 10'10*
Time (secands)
Time (Scconds)
Time (scconds)
Figure 5: Average regrets (log-scaled) vs. algorithm runtime (log-scaled seconds) ("time taken for running 500 iterations") convergence subset of 15 benchmarks (10 synthetic & 5 real-world) comparing our method against SOTA high-dimensional BO algorithms. Full statistical tests and per-problem plots for all 60 problems provided in the Appendix
occasionally surpass GIT-BO in final regret, but only after additional hour of wall-clock time. In contrast, GIT-BO reaches competitive superior regret levels within minutes, providing decisive advantage time-critical engineering settings. Taken together, these iteration- and runtime-based analyses establish GIT-BO as both the most efficient and broadly effective algorithm among current high-ddmennional BO methods.
Summary of additional ablation studies To better understand the drivers behind these empirical trends, we next highlight the key findings from our ablation studies detailed in Appendices B and
GI-subspaces vs. alternative subspaces: Compared GIT-BO (TabPEN GI-subspace) to vanilla TabPFN BO that samples in the 500D space, TabPFN Trust- Trust-Region (Eriksson et al. 2019 and TabPFN + BAxUS projection (Papenmeier et al.) 2022). Trust region and BAxUS S-styeetyyee projections also help, but consistently underperform Gl Gl-subspaces (Appendix B.4).
2. Acquisition function (UCB vs. El): Empirically, we show that both EI and UCB benefit sub- stantially from GI-subspaces, and UCB provides modest but stable advantage, matching prior observations about El's numerical instability in nigh-dimensional settings (Appendix B.5
Subspace dimension and sampling: Performance is robust across a range subspace dimensions Very large (e.g., 40) hurts performance, while small fixed r and variance-explained criteria (92.5-95% perform best (Appendix B.I.
4. Sampling in subspace and reference point Teel' Uniform, Random, and Sobol sampling in the Gl-subspace lead to similar trends, with mild problem-dependent differences (Appendix [B3). Empirically we verified that Fief Johs has better optimization performance than Fief
Targ max yuu. (Appendix B.6).
Varying the initial Latin-hypercube sample size from 20 1000 points still leaves GIT-BO as the top-ranked method across all sizes, while GP-based baselines degrade or fluctuate, especially in the large-data regime (Appendix B.7).
Initialization sample size:
6. Alternative surrogates and fine-tuning: Mild fine-tuning of TabPEN on each benchmark yields small but consistent gains (Appendix(B.3) When we replace TabPFN with a standard GP sur- rogate and reuse Gl-subspaces, the algorithm still identify the effective subspace in the

---

Published as a conference paper at ICLR 2026
embedded high-dimensional problem, confirming that Gl-subspace discovery is not specific to TFMs (Appendix
6 DISCUSSION
Our experiments highlight several strengths and limitations of GIT-BO in high-dimensional Bayesian optimization. GIT-BO consistently lies on the Pareto frontier of performance versus run- time: while BAxUS and Vanilla BO can occasionally match final regret, they require orders of magnitude more wall-clock time, whereas GIT-BO reaches near-optimal solutions within minutes. At the same time, TuRBO emerges a compelling alternative when runtime alone is the primary criterion, underscoring the practical trade-off between speed and accuracy. We also observe plateau- ing convergence in both GIT-BO and BAxUS, reflecting the known bias plateau of TabPFN pre- dictors as sample sizes grow (Nagler 2023) and pointing to broader challenges for probabilistic surrogates. Although GIT-BO excels on most synthetic and engineering tasks, its failures on Rover and Styblinski-Tang reinforce the "no free lunch" theorem (Wolpert & Macready 1997). Finally, practical limits persist: TabPEN requires large GPU memory, enforces a 500D cap, and demands sper-spe subspace thresholds. Even without retraining, its inference is slower than fitting simple GP in TuRBO or Vanilla BO. These findings suggest two directions for future work: scaling TFMs with memory-efficient architectures for faster inference, and designing benchmark suites that capture the heterogeneity of real-world tasks beyond synthetic testbeds.
CONCLUSION
We presented GIT-BO, a Gradient-Informed Bayesian Optimization framework that integrates TabPFN v2 with adaptive subspace discovery to tackle high-dimensional black-box problems. Across sixty benchmark variants, including scalable synthetic functions and challenging engineering tasks, GIT-BO consistently achieves state-of-the-ee performance while maintaining favorable run- time profile, often reaching near-optimal solutions minutes. By leveraging foundation model in- ference and gradient-ing informed exploration, GIT-BO eliminates costly surrogate retraining and scales effectively up to 500 dimensions. At the same time, limitations remain: performance plateaus on cer- tain tasks, GPU memory requirements of TabPFN and the need for user-defined subspace thresholds. Looking forward, future work should pursue more memory-efficient TFM architectures, automated strategies for subspace selection, and broader benchmark suites that bridge synthetic testbeds and real-world engineering problems. Extending GIT-BO to constrained, mixed-variable, and multi- objective optimization also represents promising avenue for further impact.
gradient-informed
ACKNOWLEDGMENTS
PI Ahmed acknowledges support from the National Science Foundation under CAREER Award No. 2443429, Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
REPRODUCIBILIT STATEMENT
We are committed to ensuring reproducibility of results. Our code is publicly available at httptps: //qithub. wtt.c//osseey3334/111122
ETHICS STATEMENT
This work does not raise any ethical concerns.
REFERENCES
Sebastian Ament. Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Un- expected improvements to expected improvement for bayesian optimization. Advances Neural Information Processing Systems, 36:20577-20612. 2023.
10

---

Published as a conference paper at ICLR 2026
Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Sys- tems, 32, 2019.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems, volume 33, pp- 21524 21538,, 2020.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in neural information processing systems, 14, 2001.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(10):28-303 2012.
Paul G Constantine, Eric Dow, and Qiqi Wang. Active subspace methods in theory and practice: applications kriging surfaces. SIAM Journal on Scientific Computing, 3604)A15500A1524 2014.
Tiangang Cui, James Martin, Youssef M Marzouk, Antti Solonen, and Alessio Spantini. Likelihood- informed dimension reduction for nonlinear inverse problems. Inverse Problems, 14015, 2014.
Ian Dewancker, Michael McCourt, and Scott Clark. Bayesian optimization for machine learning: A practical guidebook. 16Xiv:1612.0459 [cs.LGG] 2016.
David Eriksson and Martin Jankowiak. High-dimensional bayesian optimization with sparse axis- aligned subspaces. Proceedings of the Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume 161, pp. 493-503 PMLR, 2021.
David Eriksson, Michael Pearce, Jacob Gardner, Ryan D) Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. Advances in neural information processing systems, 32, 2019.
Runa Eschenhagen, Aaron Defazio, Tsung-Hsien Lee, Richard E. Turner, and Hao-Jun Michael Shi. Purifying shampoo: Investigating shampoo's 's heuristics by decomposing its preconditioner. In The Thirty-minth Annual Conference on Neural Information Processing Systems, 2025.
Jacob Gardner, Chuan Guo, Kilian Weinberger, Roman Garnett, and Roger Grosse. Discovering and exploiting additive structure for bayesian optimization. In Artificial Intelligence and Statistics, pp- 1311-1319. PMLR, 2017.
Joshua P Gardner, Juan Carlos Perdomo, and Ludwig Schmidt. Large scale transfer learning for tab- ular data via language modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, and Frank Hutter. Real-tabpfn: Improving tabular foundation models via continued pre-training with real-world data. arXiv preprint wwXiv:2507.03977 2025.
Léo Grinsztajn, Klemens Flöge, Oscar Key, Felix Birkel, Philipp Jund, Brendan Roof, Benjamin Jäger, Dominik Safaric, Simone Alessi, Adrian Hayler, al. Tabpfn-2.5: Advancing the state of the art in tabular foundation models. arXiv preprint Sxiv:2511.00666 2025.
Eric Han, Ishank Arora, and Jonathan Scarlett. High-dimensional bayesian optimization via tree- structured additive models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7630-7638, 2021.
Nikolaus Hansen, Anne Auger, Dimo Brockhoff, and Tea Tušar. Anytime performance assessment in blackbox optimization benchmarking. IEEE Transactions on Evolutionary Computation, 26 1693119333355 2022.
11

---

Published as a conference paper at ICLR 2026
Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter. Tabpfn: A transformer that solves small tabular classification problems in a second. arXiv preprint arXiv:2207.018843 2022.
Noah Hollmann, Samuel Müller, Lennart Purucker, Arjun Krishnakumar, Max Körfer, Shi Bin Hoo, Robin Tibor Schirrmeister, and Frank Hutter. Accurate predictions on small data with a tabular foundation model. Nature, 637(8045):319-336, 2025.
Sture Holm. simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979.
Carl Hvarfner, Erik Orm Hellsten, and Luigi Nardi. Vanilla Bayesian optimization performs great in high dimensions. In Proceedings of the 4/ International Conference on Machine Learning, volume 235, pp. 20793-20817. PMLR, 2024.
Donald R Jones. Large-scale multi-disciplinaryy mass optimization in the auto industry. In MOPTA 2008 Conference (20 August 2008), volume 64, 2008.
Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. High dimensional bayesian opti- misation and bandits via additive models. In International conference machine learning, pp- 295-304. PMLR, 2015.
Ryo Karakida, Shotaro Akaho, and Shun-ichi Amari. Universal statistics of fisher information in deep neural networks: Mean field approach. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 1032-1041. PMLR, 2019.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54, pp. 528-536.
PMLR, 2017.
Takehisa Kohira, Hiromasa Kemmotsu, Oyama Akira, and Tomoaki Tatsukawa. Proposal of bench- mark problem based on real-world car structure design optimization. In Proceedings of the Ge- netic and Evolutionary Computation Conference Companion, pp. 184, 2018.
Abhishek Kumar, Guohua Wu, Mostafa Ali, Rammohan Mallipeddi, Ponnuthurai Nagaratnam Suganthan, and Swagatam Das. A test-suite of non-convex constrained optimization problems from the real-world and some baseline results. Swarm and Evolutionary Computation, 56: 100693, 2020.
Nilay Kumar, Mayesha Sahir Mim, Alexander Dowling, and Jeremiah J Zartman. Reverse engineer- ing morphogenesis through bayesian optimization of physics-based models. npj Systems Biology and Applications, 10(1):49, 2024.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approx- imation for natural gradient descent. Advances in neural information processing systems, 32, 2019.
Ben Letham, Roberto Calandra, Akshara Rai, and Eytan Bakshy. Re-examining linear embeddings for nigh-diimesiooal bayesian optimization. In Advances in Neural Information Processing Sys- tems, volume 33, pp. 1546-1558, 2020.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Leurning Representations, 2018.
Matthew TC Li, Youssef Marzouk, and Olivier Zahm. Principal feature detection via ø-sobolev inequalities. Bernoulli, 30(4):297933033, 2024.
Matthew TC Li, Tiangang Cui, Fengyi Li, Youssef Marzouk, and Olivier Zahm. Inference: A Journal of the IMA, 14(3)iaaf021, 2025.
Information and
Haitao Liu, Yew-Soon Ong, Xiaobo Shen, and Jianfei Cai. When gaussian process meets big data: A review of scalable gps. IEEE transactions on neural networks and learning systems, 31(11); 4405 4423, 2020.
12

---

Published as a conference paper at ICLR 2026
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul PPP Grasman, and Jric-Jan Wagenmak- ers. A tutorial on fisher information. Journal of Mathematical Psychology, 00:40-55,44 2017.
Junwei Ma, Valentin Thomas, Rasa Hosseinzadeh, Alex Labach, Jesse C. Cresswell, Keyvan Golestan, Guangwei Yu, Anthony L. Caterini, and Maksims Volkovs. TabDPT: Scaling tabular foundation models on real data. In The Thirly-ninth Annual Conference on Neural Information Processing Systems, 2025.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Trans- formers can do bayesian inference. In International Conference on Learning Representations, 2022.
Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. PFNs4BO: In-context learning for Bayesian optimization. In Proceedings of the 40th International Conference on Machine Learning. volume 202, pp. 25444 25470. PMLR, 2023.
Thomas Nagler. Statistical foundations of prior-data fitted networks. Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 25660-25676. PMLR, 23-29 Jul 2023.
Amin Nayebi, Alexander Munteanu, and Matthias Poloczek. A framework for Bayesian optimiza- tion in embedded subspaces. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 4752-4761. PMIR, 2019.
Ismail Nejjar, Faez Ahmed, and Olga Fink. Im-context: In-context learning for imbalanced regres- sion tasks. Transactions on Machine Learning Research, 2024.
Leonard Papenmeier, Luigi Nardi, and Matthias Poloczek. Increasing the scope as you learn: Adap- tive bayesian optimization in nested subspaces. Advances in Neural Information Processing Sys- tems, 35: 011111566111 11886-166000 11601, 2022.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. arXiv preprint arXiv:1301.3584, 2013.
Jeffrey Pennington and Pratik Worah. The spectrum of the fisher information matrix of a single- hidden-layer neural network. Advances neural information processing systems, 31, 2018.
Herilalaina Rakotoarison, Steven Adriaensen, Neeratyoy Mallik, Samir Garibov, Eddie Bergman, and Frank Hutter. In-context freeze-thaw Bayesian optimization for hyperparameter optimiza- tion. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp.
PMLR, 2024.
Siddharth Ramchandran, Manuel Haussmann, and Harri Lähdesmäki. High- nigh-dimensional bayesian optimisation with gaussian process prior variational autoencoders. In The Thirleenth International Conference on Learning Representations, 2025.
Santu Rana, Cheng Li, Sunil Gupta, Vu Nguyen, and Svetha Venkatesh. High dimensional Bayesian optimization with elastic Gaussian process. In Proceedings of the 34th International Conference on Machine Learning. volume 70, pp. 2883-2891. PMLR, 2017.
Arik Reuter, Tim G. J. Rudner, Vincent Fortuin, and David Rügamer. Can transformers learn full bayesian inference in context? In Forty-second International Conference on Machine Learning, 2025.
Paul Rolland, Jonathan Scarlett, Ilija Bogunovic, and Volkan Cevher. High-dimensional bayesian optimization via additive models with overlapping groups In International conference artifi- cial intelligence and statistics, pp. 298-307. PMLR, 2018.
Maria Laura Santoni, Elena Raponi, Renato De Leone, and Carola Doem. Comparison of high- dimensional bayesian optimization algorithms on bbob. ACM Transactions Evolutionary Learning, 4(3): 2024.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2012.
13

---

Published as a conference paper at ICLR 2026
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimiza- tion in the bandit setting: no regret and experimental design. In Proceedings of the 27th Interna- tional Conference International Conference Machine Learning, ICML' 10, pp. 1015-1022. Omnipress, 2010.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-502 5033. IEEE, 2012.
Austin Tripp, Erik Daxberger, and José Miguel Hernández-Lobato. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Infor- mation Processing Systems, 33:1 1259-11272 2020.
Ke Wang and Alexander W Dowling. Bayesian optimization for chemical products and functional materials, Current Opinion in Chemical Engineering, 36: 100728, 2022.
Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black-box optimization using monte carlo tree search. Advances Neural Information Processing Systems, 33:19511-19522, 2020.
Xilu Wang, Yaochu Jin, Sebastian Schmitt, and Markus Olhofer. Recent advances in bayesian optimization. ACM Computing Surveys, 55(13s): 2023.
Zi Wang, Clement Gehring, Pushmeet Kohli, and Stefanie Jegelka. Batched large-scale bayesian optimization in high-dimensional spaces. In Proceedings of the Twenty-First International Con- ference on Artificial Intelligence and Statistics, volume 84, pp. 745-754. PMLR, 2018.
Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando De Feitas. Bayesian op- timization in billion dimensions via random embeddings. Journal of Artificial Intelligence Research, 55:361-387, 2016.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945.
David H Wolpert and William G Macready, No free lunch theorems for optimization. IEEE trans- actions on evolutionary computation, 1(1):67-82, 1997.
Jia Wu, Xiu- Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng. Hyperparameter optimization for machine learning models based on bayesian optimization. Journal of Electronic Science and Technology, 17(1):26-40, 2019.
Zhitong Xu, Haitao Wang, Jeff M. Phillips, and Shandian Zhe. Standard gaussian process is all you need for high-dimensional bayesian optimization. In The Thirteenth International Conference Learning Representations, 2025.
Han-Jia Ye, Si- SiYang Liu, and Wei Chao. A closer look at tabpfn v2: Understanding its strengths and extending its capabilities, 2025.
Rosen Ting-Ying Yu, Cyril Picard, and Faez Ahmed. Fast and accurate bayesian optimization with pre-trained transformers for constrained engineering problems. Structural and Multidisciplinary Optimization, 68(3):66, 2025.
Olivier Zahm, Tiangang Cui, Kody Law, Alessio Spantini, and Youssef Marzouk. Certified dimen- sion reduction in nonlinear bayesian inverse problems. Mathematics of Computation, 91(336): 1789-1835, 2022.
Yichi Zhang, Daniel W Apley, and Wei Chen. Bayesian optimization for materials design with mixed quantitative and qualitative variables. Scientific reports, 10(1):4924, 2020.
Juliusz Krzysztof Ziomek and Haitham Bou Ammar. Are random decompositions all we need in high dimensional bayesian optimisation? In International Conference Machine Learning, pp- 43347-43368. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
TABLE OF CONTENTS FOR APPENDICES
Theoretical Analysis
16
A.1
Preliminaries
16
A.2
Assumptions
16
A.3
Confidence Bounds for TabPFN- based Surrogates
17
A.4
Subspace Information Gain Analysis
18
A.5
Acquisition Function Analysis
18
A.6
Main Regret Bounds
19
A.7
Information Gain Bounds for High Dimensional Subspaces
19
A.8
Convergence Rate
19
B
Ablation Studies
20
B.I
Why r 107 Parameter Sweep ablation of GI subspace's 's principal dimension r
20
B.2
Why sampling-basee UCB? Ablation study on different 3 factor of UCB Acqui- sition Function
B.3
Why uniform sampling in the pradieet-informer subspace? Ablation Studies
over GI Subspace Sampling
B.4
Why GI subspace? Ablation study applying subspace identification TabPFN for high- high-dimensional BO
22
B.5
Why UCB? Ablation study different Acquisition Functions
22
B.6
Why Frel Tobs?
Ablation study on Frel = Fobs VS. Frel Targ Winss
23
B.7
Why Nint 2007
Ablation study different Nini
23
B.8
Will tine-tuning TabPEN further improve GIT-BO's performance? Ablation study TabPFN vs. fine-tuned TabPFN
25
B.9
How sensitive is GIT-BO to origin-centered optima? Ablation on shifted syn- thetic functions (GIT-BO BAxUS)
25
Experimental Analysis on Gradient-Informen Active Subspace Identification
26
Performance and runtime results analysis on 60 benchmarks
27
High-dimennional Benchmark Algorithms Implementation Details
28
Benchmark Problems Implementation Details
28
Additional Implemeetationnn Details
32
G.I
Hardware and Operating System
32
G.2
GIT-BO Algorithm Implementation Details
32
LLM Usage Statement
33
15

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS
In this section, we establish the theoretical foundations for GIT-BO by developing confidence bounds and regret guarantees. Our analysis builds upon the framework of [Srinivas et al. (2010) for GP-UCB while accounting for the unique properties of TabPFN as surrogate model and our gradient-informed subspace identification.
A.1 PRELIMINARIES
Problem Setup. We consider the optimization problem:
x" E arg max (()) ØER
where X' [0, 1jl is compact domain and X R is unknown objective function. At each
optimization iteration t, we observe Vr = fx;) ) + G. where & is r-sub-Gaussian noise.
TabPFN Surrogate Properties. Let go(y) ær Dt) denote the TabPFN's posterior predictive dis- tribution at point given observed data Dr = { B:)l-11 We denote the predictive mean and variance as:
(+(xr) = Egg (yll ar, Di), (z Var [v|ar D:)
Reference GP class & Information Gain. For kernel and noise o². define the (maximal) information gain
((AAfA)) = max log det(/ + ²KA)) A:AA=T
max
(1)
22
where KA is the kernel matrix evaluated on A. In GP-UCB, cumulative regret admits the canonical bound Ry Olv Br with Br confidence parameter depending on 5 and the RKHS norm llf (Srinivas al. 2010).
A.2 ASSUMPTIONS
Assumption 1 (TabPFN Approximation Quality). Based on the empirical results in Müller et al. (022) and the statistical analysis from Nagler (202)) showing that TabPFN can approximate GP posteriors with high fidelity, there exists constant Capprox 0 such that for any dataset D, and
query point æ e X:
|1((x) - 1 [x)) s Capprox
[o](x) - (0)(((()) < Eapprxx (t)
where μ (x) and d are the corresponding GP posterior mean and standard deviation, and Capprox (t) + as the TabPFN training data size increases.
Assumption 2 (Bounded Function Complexity). The true function has bounded RKHS norm: ||/ < B for some reproducing kerne k and constant B >>> 0.
Subspace). Following et (2025 2024). let y be a reference measure (e.g., standard Gaussian) and define the diagnostic/Fisher matrix
Assumption (Gradient-Inforee
H = E, V log ( ((x) [[
with dr(x) x ((x) du(x).
Let V, E Dxr contain the top-r eigenvectors of H. The best r-dimensional ridge approximation T. to x enjoys certified error
(a(T(( s Jo Ca(l)) Σ {m
D
k=r+1
for all a E (0, 1], where A (H) are the eigenvalues of H in descending order and C, (μ) depends only μ.. Thus choosing V, by Fisher-eigenvectors minimizes tight majorant of the divergence,
16

---

Published as a conference paper at ICLR 2026
with sharper (dimensional) certificates available for 1 (KL). We use this to quantify subspace truncation error. The certificate above follows from y--obbolvv logarithmie-Sobol (i) deliver the same V, for KL and Hellinger and (ii) upper- upper-bound the divergence by the tail trace Eksr (H) (Li et al. 2024). Dimensional LSI further sharpens the KI. majorant and yields match- ing minorants at the minimizer eLi al. 2025
bounds that
We approximate this Fisher matrix using surrogate gradients. BO the exact score V log (x) unavailable, so we adopt the widely used empirical-Fisse approximation based surrogate
gradients g(z) (Pascanu & Bengio 2013 Kunstner et al. 2019 Eschenhagen et al. 2025 setting
H = (=)
g(x) = V (x)
Prior Fisher-spectral analyses Pennington Worah (0088; Karakida et al. (2019) and likelihood- informed subspace theory Zahm et al. (2022) show that the leading eigenvectors of such approximate Fisher matrices still recover the dominant sensitivity directions, validating the use H as the GI- subspace estimator.
Empirical Verifications Assumption To assess whether the approximation error TabPFN's predictive posterior remains small in the regimes relevant for GIT-BO, measure the approxi- mation error between TabPFN's predictive mean and GP fitted on the same context set D,. We report the average discrepancy Capprox (t) across five high- oigh-dimessionl benchmarks: Ackley 100D, Rosenbrock 100D, Levy 100D, Dixon-Price 100D, and the 118D Reactive Power Phase problem. For each benchmark and each data-sample size i E [10, 5000]. evaluate both models fixed candidate grid and compute fapprox (t) MSE(2, TabPEN GP), Figure 6 shows that empirically the approximation error decreases sharply the dataset grows consistent with Assumption where Eapprox (t) + 0 as the context length increases.
MSE of TabPFFN & u
Eapprox
0.4
0.2
0.0
1000
2000 t (iteration)
3000
4000 5000
Figure 6: Average approximation error (appoox()) between TabPEN and GP predictive means across five problems (Ackley 100D, Rosenbrock 100D, Levy 100D, Dixon-Price 100D, Reactive Power Phase 1 118D). The plot shows the mean-squared difference evaluated on a fixed candidate grid as the context size grows from 10 to 5000. The rapid decay demonstrates that TabPFN's predictive posterior converges toward the GP surrogate as more data are incorporated, confirming Assumption empirically.
A.3 CONFIDENCE BOUNDS FOR TARPFN-BASHR SURROGATES
Define:
2t22
8 = 2B + 2lo
+
20
2C2
approx S (t)
35
Lemma 1 (TabPEN-UCB Confidence Bounds). With probability least 5, for all 1 and X:
(()))m(())) < / (x)
This follows the martingale concentration approach of Srinivas et al. (2010) but includes an ad- ditional approximation error term Capprox Eaproxx (t) to account for the difference between TabPFN
17

---

Published as a conference paper at ICLR 2026
and the ideal GP posterior. The bounded RKHS norm assumption ensures the function lies in well-defined function class, while the approximation quality assumption controls the deviation from GP-based confidence bounds.
A.4 SUBSPACE INFORMATION GAIN ANALYSIS
To analyze GIT-BO's regret, we must characterize how much information can be gained about the objective function when optimization restricted to the gradient-informed subspace.
Definition (Subspace Information Gain). For subspace S C X and set of points A = (II...., ,T. C S. the subspace information gain is:
TT,8
max ACS, A|-T JAYAJA)
where [VVA; fA) = 2 log I + o Kal is the mutual information between observations уa and function values fA-
Lemma 2 (Subspace Approximation Error). Under Assumption 3, the approximation error for re- stricting optimization to the gradient-informed subspace V,, satisfies:
DKL. (aTantaall|
S
(H)
k=++1
where Tiall represents the target distribution in the full space and 7, is its approximation in the subspace V,,
Lemma 3 (Subspace Information Gain Bound). Under Assumption 3, the information gain in the gradient-informed subspace V, satisfies:
Try, 2 a'iiuul - Csub T
where Tr. is the information gain the full space and Club is constant depending on the subspace construction quality. This follows from Assumption 3, which ensures that the Fisher eigenvectors V, minimize a Sobolev-type divergence between the full distribution and its subspace projection. The information gain in V,, is therefore lower-bounded by the variation captured in the retained eigenvalues, linking subspace structure directly to information-theoret, quantity.
A.5 ACQUISITION FUNCTION ANALYSIS
We adopt the Upper Confidence Bound (UCB) acquisition, standard principle in Bayesian opti- mization (Srinivas et al. 2010). At each iteration f, given a predictive posterior with mean (z(r) and standard deviation oe (v), UCB selects:
Ft. = arg max (n)),, (=<x)= = =eX
+ (x)((x)
where 8g > 0 balances exploration and exploitation.
We instantiate B, two equivalent ways:
Definition (Sampling-UCB)) Draw S i...d. samples (e(x) ~ (Nr((z), (z()) and set
ar (2) = max (z).
i
By extreme-value theory, the corresponding exploration parameter satisfies
(1-3).
3. ~ Φ
which asymptotically behaves V2log 5 with standard corrections
(Srinivas al. 2010
18

---

Published as a conference paper at ICLR 2026
Definition 3 Quantile-UCB from Müller al. 2023 For one-sided Gaussian quantile q (0, 1), set
3, = x-)
where 4-1 is the standard normal inverse CDF. Then
(x(x) Quantile, (
This corresponds to selecting the g-th posterior quantile, with higher producing more exploration. In code, this is parameterized by "rest probability" Prests where q Prest
Lemma 4 (Equivalence). Quantile-UCB with quantile level
1 - 1/S is asymptotically equiva- lent to Sampling-UCB with S posterior draws. Both implement the same exploration policy, differ- ing only in whether the quantile is computed analytically or via sampling.
Remark A.5. In practice, we adopt the sampling formulation of UCB, which introduces mild stochasticity by drawing finite posterior samples. This choice yields trajectories that may vary more across runs, akin to the exploratory effect of UCB. By contrast, the quantile formulation produces deterministic acquisition rule given the posterior, leading to more stable and less variable optimiza- tion behavior. We provide an ablation of both variants in Appendix[I.. [B2] Presenting the two side by side highlights their close equivalence while ensuring transparency in how exploration is controlled.
A.6 MAIN REGRET BOUNDS
We now establish our main theoretical result for GIT-BO's regret performance.
Theorem (GIT-BO Regret Bound). Under Assumptions 1-3, let (0, 1) and run GIT-BO with confidence parameter:
222
B = 2B² + 2 22ogg S + 2 2lo
+ 20
approx f appprox (t)
35
Then with probability least 1 8, the cumulative regret after T iterations satisfies:
Rr S VCTTBTY Σ (1- - En- (a))(a)) + Taappoo Spppoox (T)
where Ci = 8/llog1 + +o 22 and the second term accounts for subspace approximation error.
A.7 INFORMATION GAIN BOUNDS FOR HIGH-DIMENSIONAL SUBSPACES
Lemma 5 (Polynomial Information Gain). For common kernel functions (RBF, Matérn) restricted to an r-dimensional subspace where < D, the information gain satisfies:
YT,V (Or(lg T))+1)
This represents a significant improvement over the full-dimensional case where (T, OOoll D+1). This follows the spectral analysis of kernel functions in lower-dimensional spaces, adapting the techniques of Srinivas al. (2010) to the subspace setting.
A.8 CONVERGENCE RATE
Combining our results, we obtain the following convergence guarantee:
Corollary (Convergence Rate). Under the conditions of Theorem 1, if the TabPFN approximation emror satisfies fapprox (1) = =((=E) for some ş >> 1/2, then:
RT T
lim
T000
19

---

Published as a conference paper at ICLR 2026
with convergence rate Ry O 2) when < D.
This demonstrates that GIT-BO achieves sublinear regret with dimension-indee independent rates when the effective dimensionality is small, addressing the curse of dimensionality that plagues standard GP-based methods.
B ABLATION STUDIES
B.1 WHY 10? - PARAMETER SWEEP ABLATION OF GI SUBSPACE'S PRINCIPAL DIMENSION r
To evaluate the sensitivity of GIT-BO the dimensionality of the gradient-informed active subspace, we conducted a parameter sweep across both fixed subspace dimensions (r = 5. 10, 15, 20, 40) and variance-explained criteria (92.5%, 95%, 97.5%) for a subset of four problems. The results, summa- rized in Figure 7 and Table | highlight two consistent trends. First, very high-dimensional subspaces (e.g.. 40) exhibit clear performance degradation, indicating that overly broad subspaces dilute the effectiveness of the gradient-informer search direction. Second, low- to moderate-dimension subspaces and variance-based selections generally perform better, though the best choice of varies across problem families. For example, variance-based selection at the 92.5% and 95% thresholds achieves the top overall results.
5 yields the top average rank among different rs, while
To ensure faimess and avoid additional hyperparameter tuning, fixed r 10 for all benchmarks reported in the main text. This choice provides a stable middle ground, neither overly restrictive nor excessively large, while still yielding competitive performance across diverse problem classes. Notably, adaptive variance- based selection strategies further improve performance on average, un- derscoring the potential benefit of problem-dependent tuning, but we leave such extensions for future work. Overall, these ablation results confirm that GIT-BO remains robust to the specific choice of r, with consistent advantages over GP-based baselines even under fixed setting.
r=
== 10 (GIT-BO)
F= 15
20
40
92.5% Var
95% Van.
97.5% Ver.
Ackley 100)
Rosenbrack (D-100)
Reactive Power Phase (D-118)
Cor(s2) Marda (D=148)
Axcrgg hoyee
Asarage Seguet
207
20c
10a
100
200
100
200
100
200
100
200
Mumber Aaraliars
Beratiors
Beretiato
Figure 7: Performance of GIT-BO under different gradient-informed subspace dimensions (r) and variance-explained criteria. Average optimization regret across 20 trials is shown, with shaded regions denoting 95% confidence intervals. High-dimensional subspaces (e.g., r = 40) consis- tently degrade performance, while smaller fixed dimensions and adaptive variance-based selections achieve stronger results. We fix r 10 across all benchmarks in the main text for faimess, as it provides a balanced and competitive setting without tuning.
Table 1: Average performance rank of GIT-BO across different fixed subspace dimensions r and variance-based adaptive selections.
Selection of
Average Rank
92.5% Variance
1.75
95% Variance
2.25
97.5% Variance
3.5
r =
3.25
10
5.5
r = 15
5.5
20
6.35
40
8.0
20

---

Published as a conference paper at ICLR 2026
B.2 WHY SAMPLING-BASED UCB? ABLATION STUDY ON DIFFERENT 3 FACTOR OF UCB ACQUISITION FUNCTION
We compared two equivalent parameterizations of UCB: (1) quantile-UCB, which uses the analytic Gaussian quantile, and (2) sampling- CapllinggUCB,,,, which approximates it via the maximum over 5 posterior draws. Both induce similar exploration levels for a 1-11----0
L-1/S, SS but differ in that sampling introduces mild stochasticity. Our ablation results in Figure g and Table 2 shows that moderate exploration (3 x 1.86 1.96, i.e., quantile 95% 97.5% or sampling with 5 = 250) achieves the best ranks. Larger 3 values (S 512, 1024) lead to over-exploration and degraded performance.
In the main body of the paper we pre-committed to a single, conservative default (S=512) across all 60 tasks and 500 iterations per task. We did this deliberately for three reasons: 1. Fairness and reproducibility: Using one global setting avoids per- ber-benchmark tuning (or hindsight "cherry pick- ing") and makes results easy to reproduce and audit across a large suite. 2. Isolating the algorithmic contribution: We wanted to attribute gains to the proposed GI subspace + TabPFN framework rather than problem-speciff hyperparameter search. fixed keeps the evaluation focused on the method, not tuning effort. 3. Practicality and compute parity: Sweeping / across 60 problems x 500 iterations would multiply the already substantial compute; fixing a robust default is closer to how one would deploy the method under realistic constraints.
Despite this conservative choice (which the ablation shows not the best), GIT-BO still outper- formed all baselines in main results. The ablation simply reveals additional headroom: modestly smaller 3 values improve performance further. Designing an automatic / adaptation (e.g., schedule data-driven calibration) is promising future work, but orthogonal the contribution and therefore left out of the main comparison.
8-1.65 (q-25%)
B-1.96 (Q-97.5%) 5%)
B-2.19 (S-256)
8=232 (S-512)
B-245 (S-1024)
Ackley (D=200)
Rosenbrock (D=100)
Reactive Power Phase (D=118)
Car(k2) Mazda (D=148)
Average Rogret
Averagg Regret
Meceaa Regret
Average Regret
10a
101
10a
10a
10a
100
200
0
100
200
0
100
200
0
100
200
Number Iaaations
Number Itarations
Number Itarations
Number of Itarations
Figure 8: Ablation study of UCB with different exploration factors (3) using quantile- and sampling- based parameterizations. Moderate values (quantile 95-97.5% or sampling S 250) yield the best performance, while larger (e.g., S = 512, 1024) leads over-exploration and weaker results
Table Average performance rank of GIT-BO across different 3 from quantile-basee basedd and sampling- based UCB.
Selection of 3
Average Rank
8 1.65 (q -95%)
2.0
3 = 1.96 (q = 97.5%)
2.25
3 2.19 (S 256)
2.75
3 = 2.32 (S =512)
3.25
8 2.45 (S 1024)
4.75
B.3 WHY UNIFORM SAMPLING IN THE GRADIENT-INFORMED SUBSPACE? - ABLATION STUDIES OVER GI SUBSPACE SAMPLING
We conducted an ablation study to evaluate the impact of three different GI subspace sampling methods on GIT-BO's optimization performance: uniform (default), random, and Sobol sampling. Figure 9 shows the comparative convergence results.
21

---

Published as a conference paper at ICLR 2026
Our findings indicate mixed results without universally optimal sampling strategy. Uniform sampling generally provided stable and reliable convergence, while random sampling occasionally achieved better outcomes but with greater variance, similar Sobol sampling. These observations highlight the potential for adaptive strategies in selecting GI subspace sampling methods based on problem-specific characteristics, representing important for future exploration.
GITBO Random sample subspace
GITBO Sobol sample subspace
GITBO Uniform sample subspece
Acdlay (D=109)
Levy (D=100)
Rocenbrack (D-100)
Reactive (D=118) Mopra (D-124)
Rover (D=100)
10
207
10
10F
101
160
200
100
200
100
200
100
200
0
1000
200
180
200
Numberoofioon
Figure 9: Comparative convergence of uniform, random, and Sobol sampling strategies within the GI subspace on selected benchmarks. Shaded regions represent 95% confidence intervals over 20 trials. Random and Sobol sampling can achieve similar or superior performance than uniform sampling GI subspace in engineering problems, while struggling at the synthetic tasks.
B.4 WHY GI SUBSPACE? ABLATION STUDY APPLYING SUBSPACE IDENTIFICATION ON TABPEN FOR HIGH- BIGH-DIMENSIONAL BO
We compare using vanilla TabPFN v2 with three other subspace identification methods:
1. TabPEN: Using vanilla TabPFN v2 for BO without any subspace identification method.
TabPFN TR: TabPFN v2 BO with Trust Region (TR) EErikssn et 2019 Papenmeier al., 2022).
3. TabPFN BAXUS Projection: TabPFN v2 BO with the SOTA method that combined subspace projection method with TR from BAxUS (Papenmeier et al. 2022).
4. TabPFN GI-Subspace (GIT-BO): our GIT-BO algorithm with TabPFN v2 and Gl-subspace method
With other hyperparameters, acquisition function, and initial samples remains fixed across the tested algorithms, we run each algorithm 10 times and plot the average regret results in Figure TO] We ob- serve that though TR and BAxUS contribute improvements to the algorithm performance, where BAXUS projection improve the performance significantly on the synthetic problems. However, GI- subspace identification still outperform both other methods in the optimality of final result, maintain- ing its leading performance in both synthetic and real-world problem, confirming that our gradient subspace aligned exploration more effective than local restriction.
We believe that the reason why TR and BAXUS projection is not as effective GI-subspace is due to the inevitable implementation differences of TR GP- and TFM-based BO. For our TabPFN TR calculation, since TabPFN does not have kernel structure as GP, we can only equally weighted the TR hypercube search area but not weighted based on the GP kernel length scales as designed in the original algorithm for the TR construction (Eriksson al. 2019). This is the main reason why we explore alternatives than the existing SOTA subspace identification method.
B.5 WHY UCB? - ABLATION STUDY ON DIFFERENT ACQUISITION FUNCTIONS
We tested GIT-BO on two different acquisition functions: Expected Improvement (El) and Upper Confidence Bound (UCB, default acquisition function for GIT-BO). The average regret result from 10 trial runs on the five problems with all combinations different subspace embedding methods listed in B.4 are shown in Figure
The ablation result highlights that between acquisition functions, UCB outperforms across all settings by minor difference, consistent with prior findings that EI suffers numerical vanishing in
22

---

Published as a conference paper at ICLR 2026
TabPFN
TabPFN + TR
TabPFN + BAXUS Projection
TabPFN Gl-Subspace (GIT-BO)
Acklay (D-100)
Levy (D-100)
Rosenbrock 100)
Reactive
(Dw118) Carlx3) (D-124)
107
325
12
Ancraze Pogret
Aacenge Rogree
1.1
10
10
10
100
200
10
100
200
10a
100
200
300
200
250
100
200
Namber on
Numbor of Rerabors
Numbert fitentioo
Numbor teretors
Namber of lioeoooo
Figure 10: Ablation on different subspace identification strategies: Trust Region (TR), BAxUS Projection, and Gl-subspace (our method). Adding TR and BAxUS projection provides less perfor- mance gains compared to Gl-subspace used by GIT-BO with the best overall performance.
high dimensions Ament al. (023) and that UCB remains more stable under such conditions Xu ctt al. (025). Hence, GIT-BO adopts UCB and GI-subspace identification as default configuration for reliable high-dimensional optimization with TabPFN v2.
TabPAN EI
TabPAN TR
TabPEN EI BAXUS Projection
TabPFN + EI Gl-Subsaac
Projection
Subspace (GIT
Ackley (D-100)
Levy (D=100)
Resenbrock (D= 100)
Reactive Power Phase (D=118)
Carixll Mopts (D-124)
10*
325
Juwenge Reayre
Aucraye Sogret
22
10
10
275
10a
10a
100
100
100
200
100
200
100
200
Figure 11: Ablation two acquisition functions: El and UCB. UCB-based methods consistently outperform EI, validating GIT-BO's choice of UCB GI (red line) as the main setup.
B.6 WHY TREE Toas? ? ABLATION STUDY ON PREF FOBS VS. TREF
Targ goas
Existing high-ding dithdniinnsiinall BO methods such as BAxUS and TuRBO center their local search trust regions on the incumbent Zarg max tie where we get the best observation so far (Eriksson et al. 2019 Papenmeier et al. 2022). To assess whether GIT-BO should follow the same choice, compare two reference points Exe for generating Gl-subspace candidates: the final incumbent Targ max glas versus the centroid of observed data Bobs-
Across the average regret plot of six representative benchmarks over 10 trial runs shown in Figure 12 using the incumbent systematically worsens regret and slows convergence, and thus empirically select Jabs as our reference point. We hypothesize that the centroid offers a more stable anchor by avoiding over-concentration around incumbent stuck in local optimal location and by keeping candidate generation within the well-sampled region where TabPFN's in-context predictions are most reliable, which is consistent with the observations of TabPFN's locality and imbalance analysis in recent work (Ye et al. 2025 Nejjar et al. 2024).
B.7 WHY NISIT 200? - ABLATION STUDY ON DIFFERENT NINUr
Our choice of Nini = 200 in the main paper follows the dimensionality-aware scaling used in prior high-dimensional BO work. For example, TuRBO increases its Ninit initialization from 20 (10D problem) to (12D problem) to 200 (200D problem) (Eriksson et al. 2019). As the number of initial sample may affect the algorithm performance,
ablate the effect of initialization size by
testing Nun E {20, 50, 200, 1000) with GIT-BO and the baseline algorithms. The average regret results from 10 trial runs the five problems are shown in Figure 13
23

---

Published as a conference paper at ICLR 2026
GIT-00 Xep at best observation
GFTBC Xuy centrold lour methodi
Roserbrask (D-500)
320
300
250
240
Na.ber of barsione
Na of Serrices
Number Pf
sunberd of fumsioes
namber Stannice
Hurcer of Barston
Figure 12: Ablation study comparing two choices of the reference point Tnef for generating GI- subspace candidates: the incumbent best observation (Farg max yus blue) vs. the centroid of the observed data (oobs- red). Across all six benchmark problems, centering the GI-subspace at the centroid yields consistently lower regret and faster convergence, empirically supporting our choice Jiel Tabs in GIT-BO Ye et al. 2025).
Figure 13 demonstrated that across all initialization sizes, GIT-BO consistently attains the best statis- tical rank, even in the large-data setting (Nmm 1000) that favors GP-based competitors and should disadvantage TabPFN's amortized inference. Notably, while Vanilla BO and SAASBO are sensitive the choice of Nini, GIT-BO's performance remains remarkably stable, with only minor variation across all regimes. To complement these convergence results, Table 3 reports the corresponding runtime profiles. The runtimes show the same trend: GIT-BO's computational cost remains nearly constant across initialization regimes, while the GP-based methods incur substantial overhead as the initial dataset grows.
GIT-B0
BAXUS
TurBO
Vanila 80
SAASBO
Random Search
Average of Nax
No. 20
SANRRO
230
206
wiber
(D-509)
Aaresarcck
0--500
D-110)
Car921
Average Rank of Nye 50
Mo. +50
Med
5001
Marce
Average
No.-
SANDRO
42140
Sudke
(D0000)
12e100
Carbdl Manta (Dw34t)
Average Rank Now 1000
N 1000
Figure 13: Convergence behavior of five BO algorithms under varying initialization sizes. Across all Num (20, 50, 200. 1000) values, GIT-BO exhibits stable convergence with minimal sensitivity to initialization size, whereas several Ga-bd baselines (e.g., Vanilla BO, SASSBO) degrade or fluctuate noticeably as Ninit changes. The average statistical rank of each algorithm of each Nim also shows that GIT-BO remains the top-ranked method across all four regimes.
24

---

Published as a conference paper at ICLR 2026
Table 3: Average wall-clock time (in seconds) required to complete 500 BO iterations for all al- gorithms under Nit {20, 50, 200, 1000}. These results complement Figure 13 by showing that GIT-BO maintains stable runtime across initialization scales.
Nini 20
Nin 50 Average Runtime
Niit 200
Nini 1000
BO Algorithm
Average Runtime
Average Runtime
Average Runtime
GIT-BO
2700
2722
2734
3404
BAxUS
3666
3981
4008
4061
TuRBO
430
445
444
1137
Vanilla BO
2708
2557
2708
4650
SAASBO
13406
12617
13870
16663
B.8 WILL FINE-TUNING TABPEN FURTHER IMPROVE GIT-BO's PERFORMANCE? ABLATION STUDY ON TABPEN vs. FINE-TUNED TABPEN
Recent work on tabular foundation models consistently shows that continued pre-training or light task-specific
improve surrogate accuracy on Nomain-speciallln objectives (Gardner et al. 2024 Ma et al. 2025 Garg et al. 2025| Grinsztajn et al. 2025). Motivated by these findings, further enhance GIT-BO's per- formance beyond the frozen-TFI setting. For every problem, we generate 1,000) Latin Hypercube samples that are strictly excluded from the BO initialization set for avoiding performance gain from data leakage, and use them as the dataset for fine-tuning, Following the Real-TabPFN (Garg et al.) 2025) pipeline M we continue pre-training TabPFN for 100 epochs on this task-specific dataset, producing separately fine-tuned surrogate for each benchmark. GIT-BO is then run with these fine-tuned models drop-in replacements for the frozen TabPFN surrogate, evaluating each config- uration over 10 independent BO trials and reporting the mean regret curves.
investigate whether fine-tuning TabPEN on each benchmark
Across all benchmarks, fine-tuning consistently improves the surrogate's accuracy and yields uni- formly stronger optimization curves when inserted into GIT-BO. While the frozen TabPFN already provides competitive performance, fine-tuning enables noticeable gains on domain-specific struc- ture, demonstrating that GIT-BO can further benefit from TFM adaptation when additional data available.
GIT-BO (TabPFNv2)
GIT-BO (TabPFNv2 Finetuning)
Roperibrack (D-100)
Aridey
Reactive (Q=118])
(D-124)
320
12.
750-
Keregg Regme
500
10
280
250
8
0
100
Figure 14: Ablation comparing GIT-BO using the default TabPFN v2 surrogate versus fine-tuned TabPFN v2 model. Fine-tuning on 1,000 task-specific samples consistently improves optimization performance across synthetic and engineering benchmarks, demonstrating that domain adaptation can further boost GIT-BO beyond the frozen- TFM setting.
B.9 How SENSITIVE IS GIT-BO TO ORIGIN-CENTERED OPTIMA? - ABLATION ON SHIFTED SYNTHETIC FUNCTIONS (GIT-BO vs. BAXUS)
One concern from previous research is that synthetic benchmark with an optimum the origin could yield free wins for subspace projection methods. The BAXUS paper (Papenmeier et al. 2022) highlights that its sparse random embeddings contain the global optimum whenever lies at the origin, situation common in synthetic functions such as Ackley, Rastrigin, Powell, and Griewank
TabPFN fine-tuning code from Prior Lahs
25

---

Published as a conference paper at ICLR 2026
with their optimum at origin (0, O, ...) in benchmark problems. Thus, BAxUS and methods with such subspace embedding can exploit these problems and achieve efficient convergence.
In this section, elaborate mathematically and empirically why this is not concern for GIT-BO even though it is also projecting a www-dmmensionll subspace to the high-dimensional search space.
Mathematically, GIT-BO's gradient- infoommd subspace fundamentally differs from BAxUS's sparse random embedding. BAxUS constructs a sparse projection matrix S where each input dimension is randomly assigned exactly target dimension with random sign. This construction guarantees that the origin (0, (o....,,0) O) maps to the origin in the embedded space, since S 0 = 0 regardless of the random assignment. Consequently, any optimum located at the origin is automatically contained in BAx 's embedded subspace with probability one. In contrast, GIT-BO's projection is derived from the empirical Fisher matrix TabPFN's predictive mean. The innfientiinfrmeed subspace V,, consists of the top-r eigenvectors of H, which are determined by the local sensitivity structure of the predictive model conditioned on observed data Dats- Critically, V, is dense matrix whose structure depends on the gradient covariance across the observed samples, not on any pre-specified sparse assignment.
(())(())(())") where g(x) == Va (I) are gradients of
Furthermore, GIT-BO generates candidate points via Xar Jief + V,z, where Jief Johs is the centroid of observed data and = ~ u[[-1, 1]"). This centering at Jabs does not guarantee that the origin lies within the explored subspace unless the observed samples themselves are centered near the origin. Since the Fisher eigenvectors V,, are data-dependent and the reference point tracks the search trajectory, GIT-BO's subspace does not systematically favor origin-centered optima by construction.
Empirically, evaluate whether if the origin optima affect optimization performance by replicating Papenmeier et al.] (2022)'s experimental protocol from their supplementary material. We evaluate both BAXUS and GIT-BO with different subspace projection method on shifted versions of five 100D benchmarks:
JShiftedProblem (x) = SProblem (xr + 8), 8, ~ U(s UB
where Problem E (Ackley, Griewank, Powell, Rastrigin, Rosenbrock) D=100 U is a uniform distri- bution, and ,,LBB _ are the lower bound and upper bound of the search space. In addition to the problems with optima at origin (0, O, ...) (Ackley, Rastrigin, Powell, and Griewank), we included Rosenbrock with optima at (I, ...)D for harder shifted problem.
Figure 15| shows that even with coordinate shifts that displace the optimum away from the origin, GIT-BO consistently achieves lower regret than BAxUS across all five benchmarks. This demon- strates that GIT-BO does not rely on "free wins" from that its gradient-informer subspace mechanism is robust to optimum location.
problem structure, confirming
BAXUS
GIT-BO
2t Shifted feed Ackley (100D)
Shifted Gricwank (100D)
Shifted Powell (100D)
2250
Shifted Rastrigin (100D)
Shifted Resenbrack (1000) 10"
Aearge fayros
5000
Recrage Nayros
300000
Reorgge heurets
8000
2000
10
200000
1750
21-
3900
10
2500
1500
200
200
200
Number beraiios
Number teratiom
Number Eaniaurs
Number of lanatiom
Numcer llanatiors
Figure 15: Performance of GIT-BO vs. BAXUS on shifted 100D synthetic benchmarks; GIT-BO consistently achieves lower regret despite coordinate shifts that displace the optimum away from the origin.
C EXPERIMENTAL ANALYSIS ON GRADIENT-INFORMED ACTIVE SUBSPACE IDENTIFICATION
A central question for gradient-informed (GI) subspace identification is whether it can reliably re- cover the intrinsic dimensionality of problem when the objective is embedded in high dimen-
26

---

Published as a conference paper at ICLR 2026
sions. In principle, eigenvalue thresholds on gradient covariance spectra might fail- oicillaag around spurious directions or overestimating dimensionality-uni unless unnless the surrogate provides suffi- ciently smooth and informative gradients. We were therefore curious to test whether GIT-BO's GI subspace mechanism can autonomously identify the correct intrinsic dimension or not.
To probe this, we evaluate on Branin (d=2 embedded in 100D), Ackley (d=3 embedded in 100D), and Levy (d=3 embedded in 200D). GIT-BO with TabPFN surrogates consistently auto-selects a subspace dimension r (via a 95% variance threshold on the gradient covariance spectrum) that con- verges to the ground-truth d after ~0 iterations, while simultaneously reducing regret. These results in Figure I6 suggest that TabPFN provides a smooth and informative gradient field that allows the GI subspace to identify the correct intrinsic structure of a problem, enabling efficient search in that space.
Figure 16 also confirms the generalizability of Gl-subspace for GP surrogate. We utilize the Vanilla GP BO's setting of GP P Hvarfner al. (2024) for this experiment with BoTorch's 's setting for calculating the gradient from GP model. We then implemented Gl subspace identification using this output GP gradients (Va.u GP (x)). We see that GI subspace on GP is similarly effective since it converges to the correct d for Branin (d = 2) and Ackley (d 3), but having r - d| = for the Levy (d = 3) problem, and the average regret is indeed converg- ing towards the optimal value. We hypothesize that this behavior is due to the fact that the 95% explained-variee variance threshold not being universally suitable for GPs or this problem. Future work can be further investigating the best auto selection mechanism of for GP-based BO.
propagate_grade
(d-2, D-100)
Ackley (d-3, D-100)
Levy (d-3,
Tasrrh Subopcce (crr.0)
Taaer% GD Sutsppp IomT-00
TasPP, + Susopace (CIT-80)
G1 suboppcce dimersion (3
GP - Ei-aurcaare
D1 Bupdpppr dimersce 62
- Geouuepare
Gl sulsipace dimendoon (/
-57- Gi-uutepae
--. Emocelded Olm. 16)
-.e tmbooodds Dim. 16)
-.m Embocaded 2im, (6)
ML
Number Barations
15
Number Iterations
15
Number Iteraiion
Branin Id-2, D=100)
Ackeyy (d=3., D=100)
Levy (d=3. 2=000)
ThoPPI G Sabopace (G/180)
TheRTh Subspace iomsac
TePPT, 5 Subipaco (GM)
Uisuusssce
(sSulssacc
CASutspace
Iverage Reyrets
Averoge Bogiets
Average Regret
Number
Berations
Number Lerationns
Number Berakions
Figure 16: GI subspace behavior high-dimensional embeddings. Top: Average regret (20 trials, 95% CI). Bottom:
Auto-selected subspace dimension r under a 95% variance rule. TabPFN+GI converges the correct intrinsic dimension (r d) with strong regret reduction. GP(ED)+I also shows the convergence of intrinsic dimension in most cases, indicating the generalizability of GI- subspace method. The instability of GP'S r at the early stage of search could be stemmed from the fixed 95% for selection (maybe 92% or 97.5% 5% are better for Levy).
D PERFORMANCE AND RUNTIME RESULTS ANALYSIS ON 60 BENCHMARKS
We report comprehensive optimization outcomes across all benchmark problems considered in the main paper. Figure 17 presents regret trajectories on all 60 benchmarks, and Figure 18 compares regret versus runtime.
Changes to default BoTorch covariance and likelihood modules #2451 botorch. ceattings.propaggee grads
27

---

Published as a conference paper at ICLR 2026
Overall, GIT-BO exhibits consistently strong performance across diverse sish-dimensional prob- lems, with clear advantages on most engineering benchmarks (with the exception of the Rover fam- ily). This highlights its ability to balance convergence speed and final solution quality relative to other state-of-the-art (SOTA) methods.
For synthetic problems, GIT-BO maintains robustness dimensionality increases, whereas compet- ing methods degrade more noticeably. Nevertheless, there are cases where GIT-BO underperforms across all D (e.g., Styblinski-Tang and Michalewiez), consistent with the "No Free Lunch" theo- rem (Wolpert & Macready 1997). We also observe plateauing in the convergence of both BAXUS and GIT-BO. For GIT-BO, this behavior aligned with the known bias plateau of TabPFN predic- tors under increasing sample sizes (Nagler (023). The similar plateau in BAxUS suggests a broader phenomenon affecting probabilistic surrogates that merits further investigation.
On real-world engineering problems, GIT-BO ranks first overall, despite poor performance on the Rover tasks, again reinforeing "No Free Lunch." Interestingly. BAxUS, which dominates synthetic benchmarks, drops to fourth place on engineering problems. This discrepancy underscores the gap between synthetic and real-world benchmarks and motivates the need for more optimization bench- mark design and evaluation.
E HIGH-DIMENSIONAL BENCHMARK ALGORIHTMM IMPLEMENTATION DETAILS
We benchmark GIT-BO against four high- -igh-dimensional BO methods that GPU can also accelerate compute using PyTorch, including TuRBO (Eriksson al., 2019). Vanilla BO (Hvarfner et al.] 2024. BAxUS (Papenmeier et al. 2022). and SAASBO (Friksson & Jankowiak 2021).
TuRBO: The implementation is taken from BoTorch's GitHub repository (Balandat et al.] 2020)) (link: Itttps:/
sutorials Churbo__/turbooo ipynb license: MIT license, last accessed: Sep
s1st, 2025)
Vanilla BO: The implementation is taken from (Balandat et al. 2020) BoTorch ver-
sion 13's GitHub repository (link: hhttps:// com/ http://ooorc//booor...e
discussi iescussions//e /
license: MIT license, last accessed: Sep 1 2025)
BAXUS: The implementation is taken from BoTorch's GitHub repository (Balandat et al.] (020) (link: htttps://githui com wwt.:.ccopppo..oee..
tutorialss wataaallaaaaaabbaaaaaaaa wttss/bauss... ipynb 2025)
license: MIT license, last accessed: Sep 21st,
SAASBO: We use the SAASBO-MAP version of the algorithm for comparison. The code is taken from Xu et al. (2025) (link: https:
/ wtooX//000/////////
--yolly----- weyouu--ee---e need- for- commit 060elc6, license: license, last accessed: Sep 21st, 2025) where they implemented the MAP estimation of SAASBO based on the original paper (Eriksson Jankowiak (001), with all the hyperpa- rameter settings following precisely the same as the original paper. The reason for not using SAASBO-NUT is due to our computational resource limitations. We fixed maximum time of 10000 seconds for each trial to run, since running all benchmarking experiments takes roughly 72 million compute hours. Unfortunately, SAASBO-NUT can only run ~310 iterations given this time budget, making it unfeasible for comparison with other algorithms that can finish running 500 iterations under 10000 seconds.
Standard-GP-is--
We use BoTorch 12.1 for all algorithms mentioned above. The environment setups are detailed in the provided code zip file.
F BENCHMARK PROBLEMS IMPLEMENTATION DETAILS
The source and license details of our benchmark problems provided in the following paragraphs. We restrict our evaluation to problems with meil-maantai publicly available code to ensure re- producibility and stability across our benchmark framework. Benchmarks that require complex or
28

---

Published as a conference paper at ICLR 2026
GIT-BO
BAXUS
TurBO
Vanilla B0
SAASBO
Randorn Search
Acktey (D-1
Accley (D-200)
Arkwy (D-100)
Acktey (D=+400)
Accley (3-500)
Ropel
101
400
700
400
400
Namber Dertions
Humberr tarator
Namberel bersiins
107
Pesenbreck 100100)
Rosenbrack (D=2000)
Reserbrock (Dw300)
Resenbrece (D=400)
Rosentrack (D=500)
10
400
200
400
200
400
200
400
200
400
sonnhicc (D-100)
DiasePiice (3-2001
Disonihice (D-103)
DovnPhize (D-600)
DisznPPicc (3-5000)
200
400
200
400
200
6000
200
400
200
400
(001
(D-200)
(0-300)
(D-400)
(Da500)
200
600
200
400
260
600
200
400
200
400
(D-300)
107
107
1.0
10
105
10
10'
400
200
400
200
400
200
400
200
400
bestters
Mumber faretor
bersters
(D-100)
(D-200)
(0-300)
(D-403)
10f
10P
10"-
101-
10P
100
100
200
400
200
400
400
200
400
Narber mesters
Nuvber owatuss
Rastrigin (0-100)
Rasatign (D=208)
Rasorigin (2-300)
Rastrigin (0-400)
Rasztign (D=500)
10-
101
12a
100
10
200
400
Number of leue
Number hentlers
Mumber teratorn
Number of teoe
Number hereters
Styb Sybinneiann (D-100)
Styb Tang (D-200)
Styclistriang 10-100]
Stybi (D=4
Styb tmal Tang (D-500)
10P
200
Numorr of tondooe
Navber mentirn
Mumber terdio
Numorr of tandione
Nuvber mestirs
Michalowicz (0-100)
Micha cwicz (D-200)
102
Michslewliz (D=)00)
Micholowicz (Dm400)
Micha ewice (D-300)
200
Namder tooteess
Mamber tersine
touboro
Hamber foorees
Rover (D=100)
Bover (D=2200)
Rover (D03
Rover (D=400)
Bover (D=500)
200
400
Namber nertitts
Mumber
taratona
Namber purtistt
Reactive Power Phose (D=118)
Active Power (AP) Loss (D-153)
Resetive Power Lass (D+158)
Power Flow AP (D-126)
Power Pue Cost (D-125)
400
200
400
200
400
200
400
200
400
Fow AP Fuel (D-125)
Carbe11 Mopta (D-124)
Car/)2) Nuada (D-148)
10a
Carbe3) Marda (0-222)
Vi ker Policy (D=102)
200
400
200
400
200
400
200
400
200
400
Figure 17: Average regret vs. iterations (# function evaluations) with budget of 500 iterations for all benchmarks. Average regrets are illustrated by solid lines, with shaded bands denoting 95% confidence intervals. The y-axis is log-scaled. GIT-BO finds the optimal value for 29 out of the total 60 problems.
29

---

Published as a conference paper at ICLR 2026
GTT-BO
BAxUS
TurBO
Vanilla 80
SAASBO
Random Search
Ackley 10-100)
Ackley (Dv200)
Ackley (D- 100)
Ackley (Dv:00))
Ackley (D-100)
10
10
10t
10
10
Tim Secores)
Tim becores)
Tim bacorda)
Time Secordar
Tiue lscorca)
107
Roaeen brock 100100)
topeo
Rosembace (0=200)
Rover brock (0=300)
Rosembrock (D-400)
Boverbrock (0-500)
10
101
10
THe beuweb))
Texe
DixonPrice (D-100)
DoxonPitce (D-200)
DisonPrice (D-300)
DosonPrice (D-400)
DivonPicce (D-500)
1.0
10a
10
Time
Lasy (D-1080
Lasy (D-203)
(D-300)
(D-400)
(D-300)
Pasaag Sageds
MaNNy Ragens
20
10
010
10
10 30
15
010
10t
30
10
010
10
10
10
10
010
10
10
10
013 10 10
12
Poweell (0-100)
Powall
Powell (D- 300)
Powell (Dw400)
Powell (D=500)
1.0
10
100
10t
104 101
10
10r
10
10.
10
10
010F
10t
10'
30
15r
010
10
10
10 10"
010
10
10
30 107
010
10t
10
10t 10"
013F
10t
10"
10
10
(D=
110%
100
3 300
$ $.00
1.0
10"
10P
102
100
110
100
100
[10)
10
10
100
010 10t
10" 10 15
010
10t 10' 10
10
310"
10 10t 13 16*
012
10
10 19 10
013'
10 10' 10
10
becrnss
beconda)
beoies)
Thu
bsorca)
Rastrigin (0-100)
gin (0=200)
Rastrigin (D-300)
Rastr 10-400)
Rastrigin (0-500)
Acupp Fape0
100
103
101
10c
10
010F 16t 10 10 16
010
10 10 30 10
510 10 10
107 10
016"
151 10
15' 10
015F 16 107 10
10"
Time (eore))
Tim Secondu
Thme (enoes))
Stybilnsiminng (D-100)
Styblnscranng (0-2
Styblnskmang 3D-1
Styblinsrring 400)
StybinskMang (D-500)
10+
10
10P
310
10
010 10a
013 10
Time lecorco
Tim beton60)
The (econoo)
Time jecondo
Tru teconcn
Michalowizz (00)
Michakowiz (0m200)
Mic Nichalonizz 300)
Michalomicz (D=400)
Nichalimize (D=500)
Awraga Tapato
Aeragg Sagues
Aetrag Sagatt
10
10
104
10
10
Tim::
Rover 10-100)
Rover (002
Rover (D- 300)
Rower (D=4000)
Rover (D=500)
ANDEEH Tapros
10'
10
10t
10th 10 10"
10
10 10
Power Phase (D-118)
Agorr Power (API (D-153)
Reactiue (D-158)
Power How AP 1281
Fower Tuel Cost (2-128)
Ietrgg Rogners
20'
10
010
10
13a
10
10a
010
19t
10
10a
510
18
10
10
010 10
10
10
15a
013
10
10
10 10
Flow Fudi
Carix1) Moota
Carix21 Meada
Carik3) Mcado
Walker Policy
Mereag Ragera
Surrp Pacr Regees
100
010P
15t
16
70t
10"
010P 16t
16F
16t
18
010F
10F
10
16
10
0.0PP
16
15
10
611P
16
10
10 70
Figure 18: Average regret vs. algorithm runtime (seconds) records of running 500 iterations for all benchmarks. Average regrets are illustrated by solid lines, with shaded bands denoting 95% confidence intervals. Both axes are log-scaled.
30

---

Published as a conference paper at ICLR 2026
incompatible environment configurations not included in the present study. Looking ahead, advocate for a standardized collection of benchmarks with actively maintained codebases to facili- tate broader adoption and more rigorous comparisons in future research.
Synthetic Problems: The implementations for the nine synthetic functions are taken from BoTorch (Balandat al., 2020) (link: htttps://github. wtwp:hhub..oo........ http///yoooo//oooooeeeeeo
license: MIT license, last accessed: May 1st, 2025). The bounds of each problem are the default implementation in BoTorch. De- tailed equations for each problem can be found here: https: www sfu. ca/ -ssur jano/ opt imi zation inn html
Functions wetteiinnssntttiiiiee
Power System Problems: We examine subset of six problems, specifically those with design spaces exceeding 100 dimensions, from the CEC 2020 Real World Constrained Single Objective problems test suite (Kumar et al., (2000) (link: https: https://github... Cttp:uub.ccoooo---uuuuuaaee-- 2020-RN-Constan
ned Opt m sation] license: no license, last accessed: May 1st, 2025). The code is initially in MATLAB, and translate into Python, running pytest ensure the implementations are correct. While these problems incorporate equality constraints (z)) they transformed into inequality constraints (gj (z)) using the methodology outlined in the original paper (Kumar et al. (020), as constraint handling is not the primary focus of this research. These transformed constraints are subsequently incorporated into the objective function f(x) as penalty terms.
( gz(x) - 50, c= 1=
(x) ((x) max (0, 9 (x))
We set a different penalty factor for each problem, respectively, to make the objective and con- straint values have a similar effect on Spenalty(1))
Table 4: Penalty Transform Factors P of Benchmark Problems from CEC 2020
CEC's Problem Index
Our Naming
p
Reactive Power Phase
0.01
35
Active Power (AP) Loss
0.0002
Reactive Power Loss
0.001
37
Power Flow AP
0,04
38
Power Fuel Cost
0.02
39
Power AP+Fuel
0.04
Rover: The implementation is taken from Wang et al. 2018 (link: attpp:///githbe com/
2i-w/ Btte://nseebeeeeeeee -
license: MIT license, last accessed: May 1st,
2025).
Car(xl) Mopta: The MOPTA08 originally proposed by Jones 20088. The executable used in this study are taken from the paper Papenmeier el al. (2022) personal website (link: httpp::/ leonard. peomardd. 1 2023 02 02/00 00///m/mm wtt.aaa0eeeeeeelabebeeaaa aataeehhslmlllll.......mm.... license: no license, last accessed: May 1st. 2025). The MOPTA08 Car's penalty transformation follows the formation Friksson & Jankowiak (20T))ss supplementary material of one-car car crash design problem.
Car(x2) and Car(x3) Mazda Cars Benchmark Problems: The implementation taken from Kohira et al.) (2018) (link: ptsps://ladas eng. isas. jaxa. jp/benchmark/ li- cense: no license, last accessed: May 1st, 2025). The Mazda problem has two raw forms: a 4- objectives problem 148D optimizing two-car car design problem (Car(x2)) and 5-objectives
31

---

Published as a conference paper at ICLR 2026
222D problem chree-car car design problem (Car(x3)), and both of them have inequality constraints. For both problems, we equally weight each objective to form a single objective and perform penalty transform:
couultiobj.penattee (3)
1 N
Σ ( + Σ max (z))
where N is the number of objectives, C' is the number of inequality constraints, and we use p 10 for both variants of Mazda problem.
The problem is originally a locomotion task from MuJoCo (Multi-Joinl dynamics with Contact) physics engine (Todorov et al.) 2012) (Walker-2D), one of the most popular Rein- forcement Learning (RL) benchmarks. The implementation of this RL policy search problem directly taken from Wang et al. (2020) (link: /https://gtt tht.bbb.ttbt....../... thub. com/ http///aacooooeeeeeeeee
Walker Policy:
LA-MCTS/ SOCTS//teed reee aateee/main// main/ examp 1e juco
license: CC-BY-NC 4.0 license, last accessed:
May 1st, 2025).
Table summarizes the type of problems and their respective tested dimensions.
Table 5:
Benchmark Problems
Problems
Source
Type
Dimension (D) Tested
Ackley
BeTorch
Balandat al.
2020
Synthetic
100, 200, 300, 400, 500
Dixon-Price
Belorch
Balandal et
2020
Synthetic
100, 200, 300, 400, 500
Griewank
BeTorch
Synthetic
100,
300,
Levy
BoTorch
Halandat al.
4120
Syntheric
100. 200. 300, 400, 500
Michalewicz
BoTorch
Balandii ct.
2020
Synthetic
100, 200, 300, 400, 500
Powell
Belorch
Balandal et
2020
Synthetic
100, 200, 300, 400, 500
Rastrigin
BeTonch
[allandal el
20.0
Synthetic
100, 200, 300, 400, 500
Rosenbrock
Belorch
(Blatdal
Synthetic
300, 400,500
Styblinski Tang
BeTorch
[Balandal
2020
Synthetic
100, 200, 300, 400, 500
Reactive Power
CBC2020
Stite
(Kumar et
2020
World
118
Active Power (AP) Loss
CBC2020 Benchmark Suite
(Kimar et m
MAA
Real-World
153
Reactive Power Loss
CBC2020 Benchmark Suite
Kunaai 0t 4 20A
Real-World
158
Power Flow AP
CBC2020 Benchmark Suite
2027
Soul-World
126
Power Fuel Cost
CBC2020 Benchmark Suite
Kumar et m 4147
Real World
126
Power AP4Fuel
CEC2020 Benchmark Suite
Real-World 126
Rover
Previous BO studies (Wany el
2018
Real World
100, 200, 3000, 400, 500
MOPTAOS CAR
Previous
(Pipenmeser
2023
124
MAZDA
Mazda Car Benehmark
Köhira 6
2018
Real-World
222
MAZDA SCA
Mazda Car Benchmark
Kohnra V sB
2015
Real- Real-World
148
Walker Policy
MuJeCo (Toduroy el
2012
2020
Real World
102
G ADDITIONAL IMPLEMENTATION DETAILS
G. HARDWARE AND OPERATING SYSTEM
Due to the large number of benchmark problems and random seeds, the experiments conducted in parallel on a distributed server with nodes of the same compute spec: a node with 24 Intel Xeon Platinum 8480+ CPU cores and NVIDIA H100 GPU. All experiments were conducted GNU/Linux 6.5.0-15-generic x86 64 system running Ubuntu 22.04.3 LTS as the operating system, ensuring a consistent computational environment across all benchmark tests. As for the environment, we use Bo 12.12.0 and PyTorch 2.6.0+cul 126 for all underlying optimization frameworks for the benchmark algorithms except GIT-BO.
G.2 GIT-BO ALGORITHM IMPLEMENTATION DETAILS
The GIT-BO algorithm was implemented using Python 3.12 with the TabPEN v2.0.6 imple- mentation and model (link: https::/ggthbti http://twww....oo..........aaaaa
and https:// huggingface, httppP/irr-bbbb//abee----------- license: Prior Lab License (a derivative of the Apache 2.0 license
32

---

Published as a conference paper at ICLR 2026
Would making TabPEN differentiable hurt the performance? Since there is stable release of a TabPEN v2 code that allows full model differentiation as far as we know, we get rid of some marginal performance boosting numpy code in the official TabPFN v2 code (e.g.. ensembling of 8 TabPFN v2 for increasing the accuracy marginally D or rewrite the numpy-based operations (e.g., numerical transformations D Py'Torch code into single model TabPFN v2 in complete PyTorch code that allows us to use torch aut ograd. wawkwardcwwdddd () for gradient calculations.
This change results in our implementation as faster inference speed due the full GPU paralleliza- tion of using Py PyTorch and getting rid of the default n.est mestimmtrorrg TabPFN v2 eight ensemble calculation (we use b.estimators performance accuracy degradation as presented in Figure 19 without transformation permutations with ensembling. That said, the GIT-BO method would have even better performance if, in the future, TabPFN v2 releases differentiable option.
with a fixed standardize transformation), but suffers from
TabPFNv2
Differentiable TabPFNv2
T8PPNN Inference Time
2.5
Mean Square Error
0.12
0.10
1.5
0.08
1.0
0.0
0.02
Problem Dimension
Problem Dimension
Figure 19: Comparison of TabPFN v2 and our implementation of GIT-BO TabPFN v2 across in- creasing problem dimensions. Left: inference time (seconds) grows substantially for TabPFN v2 due to ensemble evaluations, while GIT-BO's PyTorch implementation achieves consistent GPU- accelerated speedups. Right: mean squared error (MSE) highlights the accuracy trade-off, where climinating TabPFN's default ensemble (n estimator=8 - 1) leads to modest degradation. Overall, GIT-BO achieves faster inference with competitive accuracy, demonstrating the benefits of differen- tiable integration of TabPFN into BO pipelines.
H LLM USAGE STATEMENT
We acknowledge the use of LLMs (ChatGPT, Claude, and Gemini) only for polishing the writing of this paper.
http:///ww.....o.ooooooooooooo/a quantile transformer.py
33


Reviews:
[{"review_id": "0hPfe7hCpB", "reviewer": "Reviewer_MZj9", "summary": "This paper introduces GIT-BO, a gradient-informed Bayesian optimization framework that leverages TabPFN v2, a tabular foundation model, to perform high-dimensional Bayesian optimization without surrogate retraining. The key idea is to extract predictive-mean gradients from the frozen TabPFN model, identify a low-dimensional active subspace via Fisher information, and perform UCB-based acquisition within this subspace. The method is evaluated on 60 high-dimensional benchmark problems (up to 500D), demonstrating strong performance and runtime advantages over GP-based baselines like SAASBO, TuRBO, and BAXUS.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "++ This method combines frozen tabular foundation models (TFMs) with gradient-informed subspace discovery, and provides a novel fusion of amortized inference and classical dimension reduction.\n\n++ Comprehensive benchmarking against SOTA methods, with rigorous statistical ranking and runtime analysis.", "weaknesses": "-- The performance depends on the pre-trained foundation model. The frozen TFM may not adapt well to functions outside its pre-training distribution, leading to poor performance on certain tasks. \nAdditionally, no fine-tuning or domain adaptation is performed, which limits generalization to highly specialized or out-of-distribution objectives.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mukC7ZU0xl", "reviewer": "Reviewer_RQPR", "summary": "The paper presents a novel method GIT-BO that utilises the TabPFN foundational model as a surrogate, and uses its gradients to identify a low-dimensional subspace over which to conduct the acquisition optimisation. Authors benchmark the time and sample complexity of their method against other baselines and also conduct a number of ablations showing the importance of each of the components of the final algorithm.", "questions": "- In Appendix B1, Figure 6, you show that replacing UCB with EI in your method makes the performance drop drastically on Levy and Rosenbrock. This is quite unprecedented. Can you provide some explanation as to why that happens?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is faster (in terms of wall-clock time, while maintaining performance comparable to the existing state of art\n- The authors conduct plenty of interesting ablations, showing the importance of each of the components used in the final algorithm \n- The additional theory in the appendix, while not particularly novel and easily following from preceding work, is still a nice addition for completeness", "weaknesses": "- Since authors emphasise the importance of time-complexity, as opposed to pure sample complexity as it is typically done in BO literature, it would be nice to demonstrate a problem setting, where we actually care about time-complexity (e.g. high-throughput BO), as in most classical BO problems, sample complexity is paramount, whereas wallclock time is of secondary importance\n- It seems to be authors focused on a relative low-data regime, where fitting a GP is still relatively fast, it would be more interesting to see what happens when the size of the dataset is much bigger, where technically fitting a GP should be slow, yet TabPFN should still be fast", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2I06yuNOio", "reviewer": "Reviewer_S5Jo", "summary": "This paper proposes GIT-BO, a Bayesian optimization (BO) algorithm for high-dimensional black-box problems. GIT-BO uses prior fitted networks (PFNs) as the surrogate model, which allows for fast inference, where the popular Gaussian process surrogate has cubic inference complexity. While TabPFNv2, a tabular foundation model, already allows for modeling problems with up to 500 dimensions, the authors argue that additional steps are necessary to perform BO efficiently on high-dimensional problems. To this end, they use the gradient information provided by the PFN to estimate an active subspace. Like other subspace-based approaches for high-dimensional Bayesian optimization, they then sample points in the active subspace and project them to the full-dimensional space to evaluate the acquisition function and choose the next point to evaluate.\n\nBy comparing to several state-of-the-art algorithms for high-dimensional BO on various benchmarks, the authors provide extensive empirical evidence for the effectiveness of the proposed approach.", "questions": "-\tGIT-BO currently uses a search space of fixed dimensionality. Do you think the method would benefit from expanding subspaces, similar to BAxUS?\n-\tThe reference point is centered on the centroid of the observed data. What is the impact of that choice? What happens if you center it on the incumbent observation, and why is this choice necessary?\n-\tWhy does EI perform so poorly in Figure 6?\n-\tWhat is the difference between the sampling schemes in section B.4? What’s the difference between random sampling in the subspace and uniform sampling? And do you have any insight into why one performs better than the other?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a relevant problem. High-dimensional BO has received considerable attention in the past and is an active field of research. The paper is the first method that uses PFNs, which is an interesting surrogate model due to its in-context capabilities, for high-dimensional BO.\nThe approach is well-motivated, and the paper is well-written. The storyline is clear, and the paper features an extensive empirical evaluation that shows the benefits of the approach. The evaluation is open about the limitations of GIT-BO and the performance of other state-of-the-art methods. The paper identifies the superior inference time of GIT-BO as a key strength, which is reasonable.\nThe paper provides an appendix with extensive ablation studies and additional experiments. The appendix complements the paper and preempts many questions I had when reading the paper.", "weaknesses": "The main concerns I have with this paper are the large performance degradations upon minor modifications of the algorithm. For instance, Figure 6 shows that the GIT-BO with expected improvement instead of the upper confidence bound performs considerably worse, worse than TabPFN without the gradient-informed subspace. Similarly, Figure 9 shows that the technique for sampling candidates in the low-dimensional subspace is crucial for performance. I wouldn’t expect such a big impact from these choices, and it makes me question the generalizability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f9w82Jg8LL", "reviewer": "Reviewer_hwuD", "summary": "This paper introduces GIT-BO, a gradient-informed BO framework that aims to address the challenges of high-dimensional BO. The core idea is to integrate TabPFN v2, a tabular foundation model, as a surrogate model with a gradient-informed active subspace mechanism. This approach leverages the predictive-mean gradients from TabPFN v2 to identify a low-dimensional active subspace, aligning exploration to this subspace and using a UCB acquisition function without requiring online retraining. The authors claim that GIT-BO delivers a strong performance-time trade-off across 60 problem variants, including synthetic and real-world tasks, outperforming state-of-the-art GP-based methods.", "questions": "- Can you provide empirical evidence that TabPFN's posterior approximation error $\\epsilon_\\text{approx}(t)$ actually vanishes (or remains small) on your benchmark tasks?\n- The $\\Phi$-Sobolev certificate in Li et al. (2024) applies to likelihood-informed subspaces in inverse problems, where the Fisher matrix is computed from a known likelihood model. In GIT-BO, you compute $H$ from TabPFN's predictive mean gradients, which are outputs of a learned, black-box transformer. Why is it valid to apply the same certificate?\n- Why fix $r = 10$ for all problems? The authors mentioned in Appendix B.2 that adaptive variance-based selection (92.5%, 95%) achieves better average ranks than $r=10$, yet the main results use $r=10$", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The core idea of integrating TabPFN v2 with gradient-informed active subspaces is an intuitive approach for handling the curse of dimensionality\n- The paper highlights the potential for significant runtime advantages (orders-of-magnitude speedups) compared to GP-based methods, which is a crucial for real-world applications\n- The paper presents extensive and thorough experimental results across a wide range of synthetic and real-world benchmarks", "weaknesses": "- All methods start with 200 LHS samples. For 500 total iterations, this means 40% of the budget is spent on initialization. Why such a large initialization? This heavily favors methods that converge quickly (like GIT-BO with fast TabPFN inference) over methods that need more iterations to refine their models (like SAASBO-NUT).\n- Appendix B.1 (Figure 6) shows that vanilla TabPFN v2 without GI subspace performs poorly (8.6× worse regret). But this comparison is misleading. Vanilla TabPFN v2 uses random candidate sampling in the full 500D space, which is obviously inefficient. A fairer baseline would be TabPFN + random subspace projection or TabPFN + trust regions. Without these comparisons, it's unclear whether the gains come from gradient-informed subspace discovery specifically, or just from any dimensionality reduction paired with TabPFN.\n- Appendix C (Figure 10) shows that GP+GI subspace performs poorly compared to TabPFN+GI. The authors hypothesize this is due to \"poorly estimated GP hyperparameters\" and \"unreliable gradient estimates.\" However, in my eyes this suggests that the success is driven by TabPFN's quality as a surrogate, not by the GI subspace mechanism itself. If GI subspaces are theoretically principled (per Assumption 3), why do they fail with GPs? This casts doubt on the generality of the approach\n- Beyond invoking \"No Free Lunch,\" the paper does not deeply investigate why GIT-BO fails on Rover, Styblinski-Tang, etc. Are these problems outside TabPFN's pre-training distribution? Do they lack low-dimensional structure? A failure mode analysis would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)