BoreaRL: A Multi-Objective Reinforcement Learning Environment for Climate-Adaptive Boreal Forest Management
Authors: Kevin Bradley Dsouza
Enoch Ofosu
Daniel Chukwuemeka Amaogu
Jérôme Pigeon
Richard Boudreault
Pooneh Maghoul
Juan Moreno-Cruz
Yuri Leonenko
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Objective Reinforcement Learning (MORL)
RL environments
Climate-adaptive forest management
Boreal forests
Permafrost thaw
Carbon sequestration
Process-based simulator
Preference-conditioned policies
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: BoreaRL is a physics-based multi-objective RL environment and benchmark for climate-adaptive boreal forest management that includes carbon plus thaw objectives and analyses MORL baselines.

Abstract: Boreal forests store 30-40\% of terrestrial carbon, much in climate-vulnerable permafrost soils, making their management critical for climate mitigation. However, optimizing forest management for both carbon sequestration and permafrost preservation presents complex trade-offs that current tools cannot adequately address. We introduce BoreaRL, the first multi-objective reinforcement learning environment for climate-adaptive boreal forest management, featuring a physically-grounded simulator of coupled energy, carbon, and water fluxes. BoreaRL supports two training paradigms: site-specific mode for controlled studies and generalist mode for learning robust policies under environmental stochasticity. Through evaluation of multi-objective RL algorithms, we reveal a fundamental asymmetry in learning difficulty: carbon objectives are significantly easier to optimize than thaw (permafrost preservation) objectives, with thaw-focused policies showing minimal learning progress across both paradigms. In generalist settings, standard gradient-descent based preference-conditioned approaches fail, while a naive site selection approach achieves superior performance by strategically selecting training episodes. Analysis of learned strategies reveals distinct management philosophies, where carbon-focused policies favor aggressive high-density coniferous stands, while effective multi-objective policies balance species composition and density to protect permafrost while maintaining carbon gains. Our results demonstrate that robust climate-adaptive forest management remains challenging for current MORL methods, establishing BoreaRL as a valuable benchmark for developing more effective approaches. We open-source BoreaRL to accelerate research in multi-objective RL for climate applications.

Published as a conference paper at ICLR 2026
BOREARL: A MULTI-OBIECTIVE REINFORCEMENT LEARNING ENVIRONMENT FOR CLIMATE-ADAPTIV BOREAL FOREST MANAGEMENT
Kevin Bradley Dsouza'
Enoch Ofosu
Daniel Chukwuemeka Amaogu
Jérôme Pigeon2
Richard Boudreault 1.
Pooneh Maghoul?
Juan Moreno-Cruz
Yuri Leonenko*
University of Waterloo
Polytechnique Montréal
Université de Sherbrooke
1 [k5dsouza, e2ofosu, juan. mor.mooooo-,,, leenenko)euxateerloo..oo (fanneel-chukwweeekk... jerome.pigeon) wwwpeeggggeuudneooooggggggeeeaee www.ettww.ppm....o.mmmm 2pooneh. @oooodhhmmmhhulll@oollyttt..... RRi "Richard. Buareeecooooaaaetteeeeeerrrrreee
jerome.
ABSTRACT
Boreal forests store 30-40% of terrestrial carbon, much in climate-vulnerable per- mafrost soils, making their management critical for climate mitigation. How- ever, optimizing forest management for both carbon sequestration and permafrost preservation presents complex trade-off's that current tools cannot adequately ad- dress. We introduce BoreaRL, the first multi-objective reinforcement learning en- vironment for dlimate-adaptive boreal forest management, featuring a physically- grounded simulator of coupled energy, carbon, and water fluxes. BoreaRL sup- ports two training paradigms: site-specific mode for controlled studies and gener- alist mode for learning robust policies under environmental stochasticity. Through evaluation of multi-objective RL algorithms, reveal a fundamental asymmetry in learning difficulty: carbon objectives are significantly casier to optimize than thaw (permafrost preservation) objectives, with thausfocused policies showing minimal learning progress across both paradigms. In generalist settings, standard gradient-descent based preference-conditiones approaches fail, while a naive site selection approach achieves superior performance by strategically selecting train- ing episodes. Analysis learned strategies reveals distinct management philoso- phies, where carbon-focused policies favor aggressive high-density coniferous stands, while effective multi-objective policies balance species composition and density to protect permafrost while maintaining carbon gains. Our results demon- strate that robust rent MORL methods, establishing BoreaRL as a valuable benchmark for develop- ing more effective approaches. We open-source BoreaRL to accelerate research in multi-objective RL for climate applications.
adaptive forest management remains challenging for cur-
INTRODUCTION
Boreal forests are one the largest terrestrial biomes, circling the Northern Hemisphere and storing an estimated 30-40% of the world's land-based carbon, much of it in permafrost soils (Bradshaw & Warkentin 2015). These ecosystems overlay vast regions of permafrost, carbon-rich frozen ground that is highly susceptible to climate warming (Schuur al. 2015). The release of this permafrost carbon pool poses a significant risk of a positive feedback to global warming. As such, the steward- ship of boreal regions is not merely regional concern but global climate imperative.

---

Published as a conference paper at ICLR 2026
Afforestation has emerged important nature-based climate solution (Drever et al. 2021 with the potential to sequester substantial atmospheric carbon while providing co-benefits for biodiver- sity and ecosystem services. In boreal regions specifically, afforestation strategies can play a dual role in climate mitigation: directly removing carbon dioxide from the atmosphere through enhanced vegetation growth, and indirectly protecting vast soil carbon stores by preventing permafrost thaw (Heijmans et al., (2022) Stuenzi et al. 2021). The unique characteristics of boreal ecosystems: their extensive permafrost coverage, extreme seasonal variability, and dominance by coniferous and de- ciduous species with contrasting biogeophysical properties, make them particularly promising tar- gets for strategic afforestation efforts that can optimize for multiple objectives.
In the context of afforestation, a complex trade-off exists between maximizing carbon sequestration and maintaining permafrost stability through the surface energy balance, which is strongly mod- ulated by forest structure through multiple interconnected pathways (Dsouza et al. 2025). Dense coniferous forests excel at carbon uptake due to their year-round photosynthetic activity, but their dark evergreen canopies have low albedo, absorbing more solar radiation during the growing season while intercepting winter snowfall. This creates competing effects: reduced ground snowpack allows cold winter air to penetrate the soil (benefiting permafross), but large summer energy gains can over- whelm these winter benefits (Heijmans et al. 2022). In contrast, deciduous forests allow deep snow- pack formation that insulates the soil (potentially accelerating thaw), yet their higher albedo when leafless reduces energy absorption during spring periods (Heijmans et al.] (022). These counter- acting biogeophysical effects create a complex optimization landscape where the ideal management strategy depends on interactions between climate, species composition, stand density, and manage-
ment timing, and the relative weighting of carbon versus permafrost objectives (Bonan) 2008).
Developing prescriptive strategies for this multi-objective problem requires tools that can discover optimal policies and not just predict outcomes. Reinforcement learning (RL) is uniquely suited to this challenge because it learns sequential policies through ridl--nd-error interaction with complex environments (Sutton et al., 1998), handling the long-term consequences and delayed multi-decadal rewards inherent in forest management. Agents must navigate a non-convex landscape of conflict- ing objectives (carbon vs. thaw), generalize across diverse stochastic site conditions, and solve challenging long-horizon credit assignment problem where early management decisions deter- mine permafrost outcomes decades later. Moreover, the inherently multi-objective nature of forest management, where carbon sequestration, permafrost preservation, and potentially other goals like biodiversity or economic returns must be simultaneously optimized, makes this an ideal application domain for Multi-Objective Reinforcement Learning (MORL) (Roijers et al. 2013 Liu al. 2014).
However, the application of RL to climate-adaptive forest management has been hindered by the lack of suitable training environments that combine modern RI. algorithms with realistic physics simulation and explicit multi-objective optimization capabilities. Existing forest models are de- signed for prediction rather than policy optimization, while previous RL approaches have relied on oversimplified growth models that cannot capture the complex biogeophysical trade-offs critical to climate adaptation. Our contributions address this gap and are fourfold:
1. We introduce BoreaRI., a configurable multi-objective RL environment and framework for bo- real forest management. BoreaRL provides a carbon, and water fluxes with modular components for physics, rewards, agents, and evaluations, enabling the first systematic study of afforestation trade-offs in a MORL setting.
simulator of coupled energy,
2. We design and validate two distinct training paradigms: site-specific mode for controlled studies and generalist mode for robust policy learning under environmental stochasticity. We reveal a fundamental asymmetry in learning difficulty, where carbon objectives are easier to optimize than thaw objectives, with standard preference-comee seeference-conditionne conditioned approaches failing in generalist settings.
3. We demonstrate that even naive baselines like adaptive episode selection outperforms standard MORL approaches in generalist settings, achieving superior empirical trade-off coverage. Our analysis of emergent strategies learned by the RL policies reveals distinct approaches to density and species composition management that reflect the trade-offs between carbon and permafrost.
4. Our framework provides a principled testbed for MORL in physically isolates asymmetric difficulties, generates testable scientific hypotheses, and offers a high-impact platform for addressing the existential threat of climate change.
domains that
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Multi-Objective Reinforcement Learning (MORL): MORL extends the RL framework to handle problems that involve multiple conflicting objectives by learning policies that can navigate these trade-offs (Roijers et al.) 2013 Liu et al. 2014; Roijers et al. 2018). A common approach is linear scalarization, where the vectorized reward is reduced to a scalar using fixed weights, which requires training such as preference-conditionee RI., aim to learn a single, generalist policy π x(a)a 8, w) that can adapt Abels al., 2019). Recent work has also adapted modern policy gradient methods like Proximal Policy Optimization (PPO) (Schulman al. 2017) for multi-ojjective settings (Hayes et al.
agent for every desired trade-off (Vamplew et al. 201I). More advanced methods,
its behavior based on given preference vector (Mu et al.] 2025
2021
We demonstrate the utility
of our BoreaRL environment using multiple methods from these families as baselines.
Reinforcement Learning for Environmental Management: RL has been applied to forest man- agement and environmental conservation before (Malo et al. [021]] Bone & Dragićevid 2010 Over- weg et al. 2021 Lapeyrolerie al. 2022), however, these approaches relied on simplified growth models, static datasets, or adjacent domains. BoreaRI, implements more realistic environment for training RI. algorithms using coupled energy- water-cbar- flux simulator with implicit energy balance equations, detailed snow dynamics, and climate-driven uncertainty, enabling more accurate representation of complex ecological trade-offs in boreal systems.
Simulating forest dynamics has evolved into sophisticated process- based ecosystem models like CLASSIC (Melton et al.) 2020) and the Community Land Model (CLM5) (Fisher et al., 2019). which represent vegetation responses to biogeochemical forcing through parametric controls that govern plant physiology, carbon allocation, and nutrient cycling. The Canadian Forest Service's Carbon Budget Model (CBM-CFS3) extends this paradigm by plicitly incorporating forest management decisions and natural disturbances (Kurz et al., 2009). Our work combines process-based modeling with learning management decisions under one roof, cre- ating the first modular, scientifically credible RL environment that seamlessly integrates detailed ecosystem simulation with modern RL frameworks.
Process- Based Forest Models:
3 THE BOREARL ENVIRONMENT AND FRAMEWORK
BoreaRL is designed as modular, configurable framework with plug-in components for different aspects of the multi-objective forest management problem (Fig. [T]. The core architecture con- sists of a physically grounded simulator (BoreaRL-Sim) and a flexible MORL environment wrapper (BoreaRL-Env) that supports multiple training paradigms and evaluation protocols, conforming to the mvo-yymasiim API standard (Felten al. 2023).
Our framework provides compartmentalized modules for: (1) Physics simulation with selectable backend and temporal resolution: (2) Reward specification with customizable objective functions and normalization schemes; (3) Agent interfaces supporting both single-policy and multi-policy MORL algorithms; (4) Environmental stochasticity with controllable weather generation and pa- rameter sampling: and (5) Evaluation protocols with standardized metrics for multi multi-objective as- sessment. The framework also allows users to override default parameters for physics simulation, reward shaping, observation space structure, and training protocols. We envision that this design will enable researchers to solve a wide array of forest management problems and build custom agents.
3.1 BOREAL FOREST SIMULATOR (BOREARL-SIMA
The first part of our framework is process-based simulator that models coupled energy, water, and carbon fluxes on a n-minute time step. The simulator incorporates several key components: multi-node energy balance model spanning canopy, trunk, snow, and soil layers; a dynamic carbon cycle; a comprehensive water balance that includes snow dynamics; and stochastic modules for fire and insect disturbances that are conditioned on climate and stand state. The model uses standard. validated physical formulations found in major Land Surface Models like CLM5 (Fisher et al. 2019 Lawrence et al. 2019) and CLASSIC (Melton et al. 2020). The simulator operates in two distinct modes: generalist mode, where each H -yyear episode is driven by unique, stochastically generated weather sequences with site parameters sampled from realistic ranges to ensure policy robustness
3

---

Published as a conference paper at ICLR 2026
Inputs
2h
Site Characteristies
BoreaRL-SIM
Other Metrics
Carhon
Sequestered
Weather and Climate
Carbon
Reward
Annual
Shaping
Ground
Energy Flux
Natural Disturbances
Energy
Carbon Reward
Thaw Reward
Other Rewards
Historical Information
Water
Species Mix
RL
Agents
Stand Density
Figure I: BoreaRL environment. physics-aware boreal forest simulator (BoreaRL-SIM) con- sumes site characteristics, weather & climate, natural disturbances, and historical information, re- turning annual carbon and ground energy flux metrics. Reward shaping converts simulator outputs into learning signals. RL agents act on these rewards to learn annual policies of stand density and species mix that maximize long-term carbon while limiting permafrost thaw.
across diverse conditions; and site-specific mode, which uses deterministic weather patterns and fixed site parameters for reproducible, location-targeter optimization. For detailed descriptions of the simulator's 's physics, management, and disturbance implementations, see Appendix B
3.2 OULTI-OBIECTIVE RL ENVIRONMENT (BOREARL-ENV)
We formalize the management task Partially Observable Markov Decision Process for MORL,, defined by the tuple (S, P,
Observation Space (O): In the generalist mode, the agent receives an observation vector designed for training robust policies across diverse forest sites. In site-specific mode observation dimen- sionality reduced, with fixed weather patterns, and deterministic initial conditions, suitable for location-specific policy optimization. See observation space details in Table
Action Space (A)): The discrete action space encodes two management dimensions into single value: (1) Stand Density Change {-100.- -50, O, +500 +100) stems ha-1 represent- ing thinning or planting, and (2) Species Mix Change representing the target conifer fraction {0.0, 0.25, 0.5, 0.75, 1.0]. The transition function takes action at year updates the stand prop- erties accordingly, and the simulator runs for one year to produce the next state at +
Vector Reward Function (R): To handle the multi-objective problem, the environment returns reward vector at each step t, R, = [re,r Tr,.), where components are normalized to [-1, 1].
The Carbon Reward (re,)) incentivizes carbon sequestration while penalizing ecological violations:
AC
Te.t clip clip
-1,1
+ Sb + hb Plimit Pdensity Pinnfff -1,1
(1)
2.0
where AC, is the net ecosystem carbon change (kg C m 2 yr stock and HWP storage. Penalties include Pismit for exceeding realistie biomass (> 15 kg C m m-2)
) sy and h2 are bonuses for total
4

---

Published as a conference paper at ICLR 2026
Component
Dimensions
Description
Current Ecological State
4
Year, stem density, conifer fraction, carbon stock Latitude, mean annual temperature, seasonal ampli- tude, phenological dates, growing season length History of disturbances, management actions, and carbon flux trends spanning recent years
Site Climate Parameters
6
Historical Information
17
Age Distribution
10
Fraction of stems in each age class (seedling-ole old) for both coniferous and deciduous species
Carbon Stock Details
2
Normalized biomass and soil carbon pools Indicators for ecological limit violations and man- agement inefficiencies
Penalty Information
3
Preference Input
1
Weight we E [0, 1] for the carbon objective, abling preference-conditione
policies Episode-level site parameters sampled from physics model ranges (generalist mode only)
Site Parameter Context
62
Table 1: Observation space components in BoreaRL's generalist mode (105 dimensions) and site- specific mode (43 dimensions). The generalist mode includes episode-level site parameters for bust policy learning across diverse forest sites, while site-specific mode uses fixed parameters for location-targetdd optimization.
or soil carbon (> 20 kg C m) pools, Pdensity for exceeding maximum stand density (> 2000 stems/ha), and Pineff for ineffective actions (c.g.. thinning empty stands).
The Thaw Reward (rt,)) is designed to protect permafrost by minimizing deep soil warming. It is calculated an asymmetric function of the conductive heat flux the deep soil layer:
Tt, clip fooot ---wwan fworm -1,1
40.0
where Sooot and Swarm are the cumulative annual cooling and warming heat fluxes (MJ m m2) across the permafrost boundary. The factor principle that permafrost degradation is often irreversible. We deliberately chose this physically- grounded reward based soil heat flux rather than simpler proxies (like air temperature) ture the complex, often delayed thermal inertia of permafrost soils. Both carbon and thaw rewards constructed using existing common knowledge from literature about how these fluxes operate. Preference-conddiionee training is supported through a preference weight input we E [0. in the observation space. For details construction of the RL environment, see Appendix
2.5 penalizes warming, reflecting the precautionary
cap-
3.3 TRAINING PARADIGMS AND COLTI-OBJECTIIN FORMULATION
Site-specific vs. generalist settings: Let Φ E & denote a vector of episode-level site parame- ters that parameterize the transition kernel P& (st+1 8t, at) and the vector reward (s a;) the site-specific setting we fix MDP/POMDP. In the generalist setting we sample á ~ Dance at the start of each episode from known distribution over sites and climates; optionally, context vector (b()) is appended to the observation (Table TT. The resulting objective is mixture-of-MIDP::
[Reatvon-
Rawwl IT.
é., so the agent optimizes single
((T) = Eng
Dane popp pd
120
where pd is the trajectory measure induced by and Pa.
We write the user preference as À = (we, Wp) A with we E [0. 1] and up =1 WC. Linear scalarization produces scalar reward
= A Ro 4x) = WC Reathon,
wc) Rahaw.t-
Two regimes are useful here: (i) fixed weight À = (constant across training and evaluation), and (ii) sampled weight where X is provided as input the policy.
5

---

Published as a conference paper at ICLR 2026
3.4 MULTI-OBJECTIVE RL BASELINE ALGORITHMS
As a starting point, we implement and evaluate some simple multi-objective RL approaches that handle carbon-thaw trade-off differently.
Fixed Lambda EUPG (Expected Utility Policy Gradient): Fixed Lambda EUPG trains single policy (a 0) on a scalarized reward using fixed preference weight throughout training. The scalarized reward is rž = A Rearbon,t + (1 - -) Rehaw,t Following EUPG (Roijers et al. 2018 the policy input is augmented with the per-objective accrued return vector, enabling Expected Utility of the Returns (ESR)-consisten credit assignment. The objective is:
Jeuro(O; X) ESnDan B
Σ rdt r 120
Variable Lambda EUPG (Adaptive Preference Learning): Variable Lambda EUPG trains a single policy (a o) that learns to adapt to weights by sampling X DA for each episode:
(0) EANDA Ba E
Σ 2 +20
The policy receives the preference weight as part of its observation space (Table I and learns to adjust its behavior accordingly, making it preference-conditiones This approach enables a single policy to handle multiple trade-offs without retraining.
PPO Gated (Proximal Policy Optimization with Action Masking): PPO Gated implements a standard PPO algorithm with a policy ao(a o) and gated architecture that separates planting actions (positive density changes) from non-planting actions (thinning or no change), with separate neural network heads for each action type. The gating mechanism ensures that only valid actions considered based on the current forest state, such as preventing planting when at maximum density or thinning when no old trees are available, The objective follows standard PPO.
EXPERIMENTS
4.1 EXPERIMENTAL DESIGN
For site-specific mode, we train 5 agents with different random site seeds. For generalist mode, single agent is trained, and each episode utilizes unique random seed to sample diverse weather conditions and site parameters, ensuring robustness. Reported results for generalist mode are aver- ages over these 100 evaluation episodes. We vary the number of training steps depending on whether the experiment is site-specific or generalist, and whether its weight compare against heuristic baselines (zero density change, +100 density increase with 0,5 conifer fraction, target density of 1000 stems/ha, and conifer restoration of 100% conifer) and evaluate performance using learning curves, reward metrics, and learned strategies of the RI. agents.
sampled-weighe We
4.2 ASYMMETRIC LEARNING DIFFICULTY IN CARBON VS. THAW OBJECTIVES
We find that there is an asymmetry learning difficulty between the two objectives across both site- specific and generalist settings. Fixed-weight agents demonstrate that carbon objectives are easier to learn than thaw objectives, with thaw-preferred policies showing minimal or no learning progress in many cases. This asymmetric learning landscape emerges from the underlying physics: carbon rewards provide clear, immediate feedback through biomass accumulation, while thaw rewards de- pend on complex seasonal energy balance dynamics with delayed and noisy signals. This is exacer- bated by the necessary risk-averse formulation of the thaw objective, which reflects that permafrost degradation is often irreversible and more damaging than equivalent cooling is beneficial.
Fig. 2 shows the scalarized reward learning curves for both generalist and site-specific settings, demonstrating this asymmetric learning pattern. In the generalist setting (Fig. 22), carbon-focused

---

Published as a conference paper at ICLR 2026
A=11.00000
N=I0-55,0211
A=18255,7751
10160.110
0.10
4era Ganee: 411
1200
Gantler Frudton
0.50
000
-0.0
600
0.50
103
1505
1590
2000
253
1023
003
1070
1000
19
30
23
00
Figure 2: Asymmetric learning difficulty between carbon and thaw objectives. (a,b) Carbon-focused policies (A (1.0, 0.0)) achieve rapid learning while thaw-focused policies (0.0, 1.0)) show minimal improvement in both generalist and site-specific settings. (c) Carbon strategies favor ag- gressive density increases (1280 stems/ha) while thaw strategies remain conservative (1000-1020 stems/ha). (d) Species composition shows complex patterns: carbon policies maintain status quo, mixed policies achieve highest conifer fractions, and thaw policies promote deciduous dominance.
policies (A (1.0, 0.0)) achieve rapid learning, while thaw-focused policies (A (0.0, 1.0)) show minimal improvement, remaining near baseline performance. The site-specific setting (Fig, 2p) exhibits similar patterns. Nonetheless, distinct forest management strategies emerge from these periments. Stem density evolution over training episodes (Fig. 22 shows carbon-focused policies aggressively increase density 1280 stems/ha, while thaw-focused policies maintain conservative densities around 1000-1020 stems/ha. Species composition strategies on the other hand do not fol- low simple carbon-thaw dichotomy (Fig-2tt, While purely carbon-focused policies show minimal conifer fraction change, purely thaw-focused policies promote deciduous dominance. Though the emergence of these qualitatively distinct strategies suggests that agents develop preference-specinee management approaches, no strong conclusion can be drawn about the thaw policies as they may be a conservative result of lack of learning. Nevertheless, we will later show that well balanced policies hallmark of agents that do learn well.
4.3 CURRICULUM SITE SELECTION AND PERFORMANCE GENERALIST MODE
Given that thaw rewards are harder to learn (Section 4.2 and are majorly influenced by chosen sites (see Appendix D.I. Fig. 5). we wanted check the impact simple site selection. To do so, we implement adaptive episode selection. Our Curriculum baseline allows the agent to ignore these destabilizing sites, and consolidate policy on "safer" subset (see Table 12
Curriculum PPO (Adaptive Episode Selection): Curriculum PPO implements two-level deci- sion process that combines episode selection with action selection. Like the other baselines, Cur- riculum PPO is also cnafffrnnce-onditioonnnn (the preference weight included the observation space), but differs in its training mechanism through adaptive episode selection. The agent first de- cides whether to train on given episode using a curriculum selection network ((sitt)) - [o, I] that evaluates the episode's potential learning value based on site characteristics. Episodes are se- lected based on an adaptive threshold, and then they proceed with standard PPO action selection. The combined objective is:
JCurriulamm(P. EA~DA EonDar Egelect
E ptt
Σn rerr 220
select
where the expectation is taken only over selected episodes. The curriculum selection network fo an untrained random projection that provides a consistent ordering of the site space, while the selection threshold is adaptively adjusted based performance to expand contract the training distribution (see Appendix C.5 for details on the mechanism).
We consider preference- con.eniccooooiiiinneeee.eeeeeedd generalist mode to be our benchmark setting, and evaluate three algorithms (Variable EUPG, PPO Gated, and Curriculum PPO) in gener- alist mode assess their ability to learn policies under environmental stochasticity. For detailed

---

Published as a conference paper at ICLR 2026
Scalarized Reward
8
Variable EUPG
10
PPO Gated
Scalarized Reward
Curriculum PPO
4
Variable EUPG
PPO Gated
Curriculum PPO
1000
2000
3000 Episode
4000
5000
6000
Method
(a)
(b)
8.0
Thaw Objective
Thaw Objective
Fixed Agents
7.8
06..5
PPO Gated
Variable EUPG
7.6
0H0300
Curriculum PPO
Zero Density Baseline
+100 Density Baseline
7.4
Target Density
Conifer Restoration
Carbon Objective
Carbon Objective
(c)
Figure 3: Algorithm performance comparison in generalist mode. (a) Learning curves reveal Cur- riculum PPO's rapid convergence and stable performance versus others. (b) Scalarized evaluation reward demonstrates Curriculum PPO's dominance, whereas Variable EUPG has near-zero per- formance. Error bars represent the standard deviation over 100 evaluation episodes (per preference weight), each with a unique random seed. (c) Trade-off analysis shows the relationship between evaluation carbon and thaw objectives for different methods versus baselines. d) Curriculum PPO empirical trade-off coverage achieves superior spread with lesser X-monotonicity violations com- pared to other methods. Shows mean over 100 evaluation episodes with unique random seeds. See Appendix D.2 for fronts of other methods. All rewards and objectives summed over 50 steps.
runtime analysis, see Appendix C.9 Curriculum PPO baseline outperforms other methods across
training (Fig. 33) and evaluation (Fig. 3D) metrics (scalarized rewards). Moreover, Curriculum PPO produces the most comprehensive trade-off coverage (Fig. Bd.d), while others show poor perfor- mance across the preference space with higher X-monotonicity violations (defined the failure of an objective's return to increase with its preference weight; see Appendix D.2)
Table compares RL agents against heuristic baselines. While these fixed strategies achieve moderate scalarized rewards, they fail to capture the multi-objective front. We use hypervolume and sparsity key metrics to quantify the quality and uniformity of the learned trade-off fronts, where Curriculum PPO demonstrates superior coverage (Hypervolume: 84.3). We also experiment with altenate thaw reward formulations. Table 3 reveals that while the Asymmetric Thaw formulation is the most challenging to optimize due to its risk-averse nature, Contrast Thaw and Raw Degree Days yield only marginally better scalarized rewards and lack the necessary penalty for irreversible per- mafrost degradation. Moreover, the Asymmetric formulation forces strong avoidance of warming, whereas symmetric formulations allow small warming trade-offs (see Table 10 All three formula- tions create conflicts with the carbon objective, the physical mechanisms that benefit carbon often harm permafrost (see Appendix D.3 Table II for some mechanistic evidence). omparison of Thaw Reward Formulations and Agent Behavior. For detailed breakdown of carbon and thaw objectives with error estimates across all RL baselines, see Table 13 in Appendix [D.4]
Curriculum PPO's success highlights that the existence of curriculum (selective exposure) already helps; while an optimal ordering of that curriculum may further improve performance, our results show that simple adaptive threshold provides significant benefits. This serves naive
8

---

Published as a conference paper at ICLR 2026
Table 2: Main performance comparison of multi-objective RI. algorithms and baselines. Rewards are averaged and other metrics are computed over 100 evaluation episodes per preference weight. Scalarized reward is the primary training objective. Hypervolume (reference point: [-2, 22 and Sparsity measure the quality and uniformity of the trade-off front.
Method
Scalarized Reward
Hypervolume (t)
Sparsity (.)
RL Algorithms (Generalist Mode) Curriculum PPO
8
84.3
0.12
PPO Gated
4.7 6.0
23.6
0.09
Variable À EUPG
1.7
14.2
0.07
Heuristic Baselines
Target Density (1000 stems/ha)
3.4
20.6
N/A
Conifer Restoration (100% Conifer)
4.1 2.9
21.4
N/A
Zero Density Change
2.5 2.4
11.3
N/A
+100 Density Change
3.2 2 6.1
18.5
N/A
Table 3: Impact of thaw reward formulation on PPO Gated performance. We compare the default Asymmetric Thaw (risk-averse) against Contrast Thaw (symmetric) and Raw Degree Days (linear). All three formulations are hard to optimize with naive baseline, pointing to the difficulty of mas- tering the thaw regime. Asymmetric Thaw is the hardest, but also ecologically safer.
Formulation
Scalarized
Carbon
Thaw
Hypervol. (1)
Sparsity (L)
Asymmetric (Default)
4.7 + 6.0
7.8 + 2.5
1.5 1.5+1 1.2
23.6
0.09
Contrast
4.9 + 3.2
7.6
2.1 + 2.4
25.4
0.08
Raw Degree Days
5.2 3.7
2.4
2.3
26.1
0.08
baseline to demonstrate that effective episode and site selection is one way to succeed in preference- conditioned learning in our BoreaRL environment. It also points to the fact that, certain sites and settings are inherently bad for planting when multiple objectives are important, no matter what the management decision are, and that smart site selection is crucial. That said, believe that this benchmark is far from saturated and various other properties of the physics, objectives, simulation, and real-world can be exploited to train better RL management agents.
4.4 COMPARATIVE ANALYSIS OF MANAGEMENT STRATEGIES
To understand the learned behavior of different algorithms, we analyze their strategies (Fig. Three distinct philosophies emerge from analysis. PPO Gated develops aggressive carbon- maximizing approach, achieving the highest conifer fractions when averaged across all evaluation episodes and weights (Fig. 44) through rapid early planting followed by natural thinning (Fig. 4p). This strategy concentrates forest outcomes in high-density, high-conifer regions (Fig. 4p. In con- trast, Curriculum PPO learns balanced approach, showing steady species optimization while main- taining moderate density strategies. Variable > EUPG adopts the most conservative strategy, main- taining baseline species composition and growth, suggesting limited learning (Fig. (bc.).
The environmental implications of these management strategies are revealed through the correlation between growing season length and thaw rewards (Fig. 4+). The difference in actions reflects fundamental difference in the quality of the local optima found by each algorithm (see Table PPO Gated falls into a local optimum aggressive carbon farming (high density/conifer), which boosts carbon but fails to protect permafrost. Curriculum PPO maintains moderate densities and mixed species, allowing for longer growing seasons (see Fig. Id), which enhances canopy shading and transpiration, thereby reducing soil heating and preventing thaw. We observe strong correlation between growing season length and thaw protection (r +0.65, see Table g. Mechanistically. longer growing seasons maintain high Leaf Area Index (L.II) for more days, increasing transpiration

---

Published as a conference paper at ICLR 2026
0.5
3000
Mouce: LIIS
PD sate aniel
55.55
RPF (bviater
(a)
(d)
Figure 4: Comparative analysis management strategies. Averaged across all evaluation episodes and weights. (a) Species composition evolution shows PPO Gated achieving highest conifer frac- tions, Currieulum PPO demonstrating steady improvement, and Variable À EUPG being conserva- tive. (b) Density evolution reveals rapid early growth for PPO Gated and Curriculum PPO versus linear growth for Variable EUPG, (c) Forest composition shows PPO Gated favoring high-density coniferous stands, Curriculum PPO achieving balanced mid-range strategies, and Variable À EUPG not converging to any approach. (d) Growing season vs. thaw reward correlation shows how longer growing seasons can enhance permafrost protection. Mechanisms like increased shading and evap- otranspiration may play a role, see Table g for more.
cooling (r +0.82) and canopy shading (r -0.75). Therefore, strategically managed forests can serve as buffers against climate-induced permafrost degradation.
Moreover, these distinct physical strategies driven by specific algorithmic failure modes. PPO Gated suffers from consistent gradient from the Carbon objective overpowering the noisy, sparse Thaw signal, leading the agent to ignore the latter. Variable À EUPG sees conflicting gradients from changing preference weights, leading to risk-averse inaction (low density, minimal changes). Curriculum PPO overcomes these issues through gradient filtering: by adaptively selecting sites where it is making progress, it avoids "trap" sites for thaw, allowing it to learn cooling strategies that standard methods miss. See Table for more information these mechanisms.
DISCUSSION AND FUTURE WORK
We introduced BorcaRL, multi-objective reinforcement learning framework for climate-adaptive simulator, it captures complex boreal forest management. By integrating a preservation. Our benchmarking highlights trade-offs between carbon sequestration and permafrost the challenge of asymmetric multi-objective optimization and demonstrates how adaptive episode selection helps avoid conflicting gradient "trap sites", yielding novel ecological insights into per- mafrost protection through appropriate management. BoreaRL offers a principled testbed for MORL and impact platform for managing boreal carbon sinks address climate change.
Despite these advances, several limitations constrain the current framework's 'ss scope. BoreaRL is designed a messiccall-grounndee simulator enable controlled experimentation and generate restable scientific hypotheses (e.g., density-thaw relationships), not as a deployment-ready decision support system. Real-world deployment would require rigorous calibration against field data, expert validation, and integration with existing planning tools. Additionally, the long time horizons (50 years) and delayed rewards inherent in this domain create a challenging credit assignment problem that our benchmark isolates for study. While site selection demonstrates improved learning. MORL agents still struggle with the full complexity of environmental stochasticity, indicating that novel ap- proaches needed to fully realize the potential of RL-based climate- forest management.
Several concrete directions for future research emerge (see Appendix
1. Multi-objective Extensions incorporating economic (timber revenues, management costs) and biodiversity metrics.
2. Advanced MORL Algorithms utilizing meta-learning, non-linear scalarization, and con- tinuous preference space optimization for asymmetric objectives.
3. Real-world Validation and Deployment using historical data and spatially explicit grid- wise environments for continuous decision support.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDDEMENTS
We thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for funding and supporting this research through the Alliance Mission grant ALLRP 577126-2022 and the NSERC PDF program.
7 REPRODUCIBILITY STATEMENT
We provide all components needed to replicate our results. The main paper specifies the environment and simulator design (Table I the multi-objective formulation and baselines (Sections 3.2 3.4). and the experimental protocol with evaluation metrics used across figures. The Appendix documents the physics and implementation details of the simulator (Appendix B). the environment API and training paradigms (Appendix[C) C and the parameter sampling ranges and other configuration choices needed recreate experiments; Source code associated with this project is attached as anonymous sup- plementary zip, which includes the complete environment and simulator, training/evaluation scripts, configuration for both site-specific and generalist settings, and seeds for all reported runs. Figures in the paper reference the exact methods and settings compared so results can be matched to the corresponding configs and scripts.
REFERENCES
Axel Abels, Diederik Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. In International conference on machine learning, pp. 11-20. PMLR, 2019.
Gordon Bonan. Forests and climate change: forcings, feedbacks, and the climate benefits of forests. science, (445888):1444-4 1449, 2008
Christopher Bone and Suzana Dragióević. Simulation and validation of a reinforcement learning agent-based model for suulii-ttakehodder forest management. Computers, Environment and Urban Systems, 34(2):162-1744 2010.
Corey JA Bradshaw and Ian G Warkentin. Global estimates of boreal forest carbon stocks and flux. Global and Planetary Change, 128:24-306 2015.
Ronnie Drever, Susan Cook-Patton, Fardausi Akhter, Pascal Badiou, Gail L Chmura, Scott Davidson, Raymond L Desjardins, Andrew Dyk, Joseph E Fargione, Max Fellows, et al. Natural climate solutions for canada. Science advances, 7(23))ebbd6034, 2021.
Kevin B. Dsouza, Enoch Ofosu, Jack Salkeld, Richard Boudreault, Juan Moreno-Cruz, and Yuri Leonenko. Assessing the climate benefits of afforestation in the canadian northern boreal and southern arctic. Nature Communications, 16(1):1964, 2025.
Florian Felten, Lucas N Alegre, Ann Nowe, Ana Bazzan, El Ghazali Talbi, Grégoire Danoy, and Bruno C da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforce- ment learning. Advances in Neural Information Processing Systems, 01-23671-23700 2023.
Rosie A Fisher, William R Wieder, Benjamin Sanderson, Charles Koven, Keith W Oleson, Chonggang Xu, Joshua B Fisher, Mingjie Shi, Anthony P Walker, and David M Lawrence. Para- metric controls on vegetation responses to biogcochemical forcing in the clm5. Journal of Ad- vances in Modeling Earth Systems, (1(9)287992888) 2019.
Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathicu Reymond, Timothy Verstracten. Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. arXiv preprint arXiv:2103.09568 2021.
Monique MPD Heijmans, Rúna Í Magnússon, Mark Lara, Gerald V Frost, Isla Myers-Smin Smith, Jacobus van Huissteden, M Torre Jorgenson, Alexander N Fedorov, Howard E Epstein, David M Lawrence, et al. Tundra vegetation change and impacts on permafrost. Nature Reviews Earth & Enviroument, 3(1):68 84, 2022.
11

---

Published as a conference paper at ICLR 2026
WA Kurz, CC Dymond, TM White. G Stinson, CH Shaw, GJ Rampley, C Smyth, BN Simpson, ET Neilson, JA Trofymow, et al. Cbm-cfs3: model of carbon- -obbomdd in forestry and land-use change implementing ipec standards. Ecological modelling, 220(4):480-500) 2009.
Marcus Lapeyrolerie, Melissa S Chapman, Kari EA Norman, and Carl Boettiger. Deep reinforce- ment learning for conservation decisions. Methods in Ecology and Evolution, (211))2226626266 2022.
David M Lawrence, Rosie A Fisher, Charles D Koven, Keith W Oleson, Sean C Swenson, G Bonan, et al. The community land model version 5: Description of new features, benchmarking. and impact of forcing uncertainty. Journal of Advances Modeling Earth Systems, 11(1)):4245-
Chunming Liu, Xin Xu, and Dewen Hu. Multiobjective reinforcement learning: A comprehensive overview. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(3):385 (388,-98,,0 2014.
Pekka Malo, Olli Tahvonen, Antti Suominen, Philipp Back, and Lauri Viitasaari. Reinforcement learning in optimizing forest management. Canadian Journal of Forest Research, 51(10):1393- 1409, 2021.
Joe R Melton, Vivek K Arora, Eduard Wisernig- Cojoc, Christian Seiler, Matthew Fortier, Ed Chan, and Lina Teckentrup. Classic vl. O: the open-source community successor to the canadian land surface scheme (class) and the canadian terrestrial ecosystem model (ctem)-part part I: Model frame- work and site-level performance. Geoscientific Model Development, 13(6):2825-28) 2850, 2020.
John L Monteith. Solar radiation and productivity in tropical ecosystems. Journal of applied ecol- ogy, 93):7477776, 1972.
Ni Mu, Yao Luan, and Qing-Shan Jia. Preference-based multi-objective reinforcement learning. IEEE Transactions on Automation Science and Engineering, 2025.
Hiske Overweg, Herman NC Berghuijs, and Ioannis N Athanasiadis. Cropgym: reinforcement
learning environment for crop management. arXiv preprint arXiv:2104.04326, 2021.
CHB Priestley and RJ Taylor. On the assessment of surface heat flux and evaporation using large- scale parameters. Monthly weather review, 100(2):81-92 1972.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley, A survey of multi- objective sequential decision- makkin. Journal of Artificial Intelligence Research, 48:67-113, 2013.
Diederik M Roijers, Denis Steckelmacher, and Ann Nowé. Multi-objective reinforcement learn- ing for the expected utility of the return. In Proceedings of the Adaptive and Learning Agents workshop at FAIM, volume 2018, 8, 2018
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707.07777. 2017.
Edward AG Schuur, A David McGuire, Christina Schädel, Guido Grosse, Jennifer W Harden, Daniel J Hayes, Gustaf Hugelius, Charles D Koven, Peter Kuhry, David M Lawrence, et al. Cli- mate change and the permafrost carbon feedback. Nature, 120-7546; 2015.
Simone Maria Stuenzi, Julia Boike, William Cable, Ulrike Herzschuh, Stefan Kruse, Luidmila A Pestryakova, Thomas Schneider von Deimling, Sebastian Westermann, Evgenii S Zakharov, and Moritz Langer. Variability of the surface energy balance in permafroot-unddelain boreal forest. Biogeosciences, 18(2):343-365. 2021.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Donald F Swinehart. The beer-lambert law. Journal of chemical education, 39(7):333, 1962.
Peter Vamplew, Richard Dazeley, Adam Berry, Rustam Issabekov, and Evan Dekker. Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning, 84 (1):51-80, 2011.
JH Van't 't Hoff and Robert Alfred Lehfeldt. Lectures on theoretical and physical chemistry. 1900.
12

---

Published as a conference paper at ICLR 2026
A
SUBMISSION DETAILS
A.1 SOURCE CODE
Source code associated with this project can be found the BoreaRI. GitHub Repository
A.2 USE OF LARGE LANGUAGE MODELS
We used large language models (LLMs) in the following scoped, tuman-supervised ways: (i) Writ- ing polish. Draft sections were refined for clarity, structure, and tone; all technical claims, numbers, and citations were authored and verified by us, and every LLM-suggested edit was line-reviewed to avoid introducing errors unsupported statements. (ii) Retrieval & discovery. We used LLMs to craft and refine search queries to find related work and background resources; candidate papers were then screened manually, with citations checked against the original sources to prevent hallu- cinations. (iii) Research ideation. We used brainstorming prompts to surface alternative baselines, ablation angles, and failure modes; only ideas that survived feasibility checks and pilot experiments were adopted. (iv) Coding assistance (via Cursor) We used Cursor's inline completions and chat for boilerplate generation (tests, docstrings, refactors); all code reviewed and benchmarked before inclusion. Across all uses, we ensured that LLM outputs never replaced human analysis, reproducibility artifacts, or empirical validation.
B FOREST SIMULATOR
B. SIMULATOR PRINCIPLES AND FLOW
The BoreaRL. benchmark is built upon a process-based simulator, BoreaRL-Sim,, which is designed to be scientifically credible while remaining computationally efficient for reinforcement learning. The design adheres to several key principles:
Time-Scale Separation: There is a clear separation between the agent's decision-making timescale and the physical simulation timescale. The RL agent performs management action once per year. In response, BoreaRL-Sim resolves the energy, water, and carbon fluxes at n- minute time-step (tunable) for the entire 365-day period, capturing the fine-grained diurnal and seasonal dynamics that govern the ecosystem.
To ensure that learned policies are robust and not overfit to single deterministic future, each H-year episode is driven by a unique, stochastically generated climate. At the start of episode, a site latitude is sampled, which parametrically determines the mean climate characteristics (e.g., annual temperature, seasonality, growing season length). Then, daily weather (temperature, precipitation) is generated with stochastic noise, forcing the agent to learn strategies that are adaptive to climate variability.
Stochasticity and Robustness:
Multi-Objective Core:
The simulator is fundamentally multi-objective. It tracks the two primary reward components: carbon component that includes normalized carbon change with stock bonuses and limit penalties, and an asymmetric thaw component derived from conductive heat flux deep soil (permafrost proxy). These are returned reward vector R, [Rearbon,t- Rebawo s] where Rthaw,t penalizes warming more than it rewards cooling. This allows for the training of both specialist and generalist multi-objective agents.
The agent's actions (changing stand density and species com- position) directly modulate the core physical parameters of the simulation, such the Leaf Area Index (LAI), canopy albedo (Acan), snow interception efficiency, and aerodynamic roughness. This creates tight feedback loop where management decisions have ressically-gronndee conse- quences on the surface energy balance.
Management-Physste Coupling:
The simulation flow for a single RL step (one year) is orchestrated by the ForestEny environment, which wraps the BoreaRI-Sim instance. The sequence is detailed in Appendix B.5
13

---

Published as a conference paper at ICLR 2026
B.2 CORE PHYSICAL MODEL: ENERGY BALANCE
The core of BoreaRL-Sim is a multi-node thermodynamic model that solves the energy balance for the key components of the forest stand. The model uses standard, validated physical formulations found in major Land Surface Models like CLM5 (Fisher et al. 2019: Lawrence et al. 2019) and CLASSIC (Melton et al., 2020). For example, the state of the system is defined by the temperatures of five primary nodes: Canopy can), Trunk (Itrank), Snowpack (I (oon) Surface Soil (I sur f), and Deep Soil (T deee). The temperature evolution of each node is governed by the net energy flux, following the principle:
C. d.. dl
- Σ Fout.
where C, is the heat capacity of the node and represents an energy flux in Watts per square meter (W m -2) The main flux equations for each node are detailed below.
B.2.1 CANOPY NODE (Tean)
The canopy energy balance is the most complex, as it involves radiative, turbulent, and physiological fluxes. Unlike the soil and snow nodes which have significant thermal mass, the canopy has relatively low heat capacity and equilibrates rapidly with the atmosphere. Therefore, we treat it as diagnostic variable that reaches steady-state equilibrium each timestep, rather than prognostic variable with memory. Its temperature is solved iteratively to satisfy the condition where the net flux zero:
= Ront.con
Hoon
LEcon
Gphoto
4 mmelt.con
Q ettrank
The components are:
Bnet,can Net radiation, defined as the balance of incoming shortwave (Qabe) and longwave (Lin) radiation against emitted longwave radiation (Lou)). We use the Beer-Lambert Law for radiation extinction (Swinehart 1962):
Raet.can Qsotar (1 acon) + Ecan (Ldown,atm + Lup.ground) - 2E 0T can
H H Sensible heat flux, representing convective heat exchange with the air, governed by an aero- dynamic conductance hean:
Hean hean (Tean T Tair)
LEcan: Latent heat flux from transpiration, modeled using the Priestley-Taylor formulation (Priest- ley & Taylor 1972) modified by environmental stress factors for Vapor Pressure Deficit (fVPD) and soil water content (fswc):
A
LEcon aPT
Bnet,can fVPD fswc
A +
where apr is the Trestley-TTyllo coefficient, is the slope of the saturation vapor pressure curve, and 7 is the psychrometric constant.
G :hotto: The energy sink used for photosynthesis. This directly coupled to the carbon model via a light-use efficiency parameter (LU F), ensuring energy and carbon are conserved: Gphoto
GPP Sper where Jper.gc is the energy cost to fix one gram of carbon.
O man Energy sink for melting intercepted snow on the canopy, active only when Toan 2 273.15K and canopy snow exists.
t Pe.trunk: Conductive heat flux between the canopy and the tree trunks.
B.2.2 TRUNK, SNOW, AND SOIL NODES
The energy balances for the grvund-level nodes are primarily driven by their own radiation balance, turbulent exchange with the sub-canopy air, and conduction between adjacent nodes.
Trunk Node (T (Trrnnk):
TTrank dt
Cirunk
Herunk + T De.con + P.ground
14

---

Published as a conference paper at ICLR 2026
where Htrunk is the sensible heat flux to the air, and & P.ground represents conduction to either the snowpack or the surface soil, depending on snow cover.
Snowpack Node (T
Canow dl snow Snot,
Hnoww snow Po,soi + Pettrunk
Here, anew is the net radiation at the snow surface, which is high in albedo (Amnow ~ 0.8). mell..now is the energy sink for melting, active when the net energy flux is positive and T snow 273.15K. The thaw haw-degree-day reward component (Rthaw,) () is derived from the energy flux out of the deep soil layer into the permafrost boundary, providing a physically-baset metric of permafrost degradation.
Surface Soil Node (osiit,surf))
surr dfooosleurss Reet,soil
soii LEgoit
De,deep
+
& c, Pe,trunk
dt
This balance is active when no snow is present. includes latent heat of evaporation from the soil (Lo)) and conductive flux to the deep soil layer (Ro.deep).
Deep Soil Node (T doopp)
dT eToost dess
Csoit deep dt
T c.surrf Pe. c.booundryy
The deep soil node integrates heat from the surface layer and loses heat to a fixed-temperature deep boundary, representing the top of the permafrost table. The permafrost thaw objective, Rthaw,t. is calculated as an asymmetric function of the cumulative annual positive and negative energy fluxes across this boundary, penalizing warming more than it rewards cooling.
B.3 KEY SIMULATOR SUB-MODELA
Layered on top of the core energy balance are modules for the water balance, carbon cycle, and stand dynamics. The simulator's 's realism achieved through several key subsystems that work together to capture the complex dynamics of boreal forest ecosystems:
B.3.1 AGE-STRUCTURED DEMOGRAPHY
The simulator implements an age-structured population model with five age classes: seedling (0- 5 years), sapling (6-20 years), young (21-40 years), mature (41-100 years), and old (101+ years). Each age class has distinct canopy factors and light-use efficiency scaling, with natural transitions occurring annually. Thinning operations are preferentially applied old trees simulate realistic harvesting practices, while planting adds seedlings with the specified species mix. This age struc- ture enables realistic representation of forest development trajectories and management constraints, as younger trees have different physiological properties and respond differently to environmental conditions.
B.3.2 ADVANCED WEATHER GENERATION
The weather module generates latitude-dependent climate parameters including temperature- precipitation relationships that vary by season. Summer precipitation is positively correlated with temperature (reflecting convective rainfall patterns), while winter precipitation shows complex tem- perature dependencies (snow vs. rain thresholds). The system also models rain-induced suppres- sion of diurnal temperature amplitude, where precipitation events dampen daily temperature swings through increased cloud cover and latent heat effects. This creates realistic weather patterns that in- fluence forest dynamics through both direct physiological effects and indirect impacts on the energy balance.
B.3.3 DISTURBANCE MODELING
The simulator includes stochastic models for fire and insect outbreaks. Fire probability is condi- tioned drought index (accumulated temperature and precipitation deficit), temperature thresholds
15

---

Published as a conference paper at ICLR 2026
(fires more likely during hot periods), and species flammability (conifers susceptible than de- ciduous). Insect outbreaks depend on winter temperature (warmer winters increase overwintering survival) and stand density (denser stands facilitate spread), with coniferous species being more susceptible to infestations. Both disturbances cause fractional mortality and route carbon appropri- ately: fire combusts biomass (releasing atmosphere), while insect kill transfers dead biomass to soil carbon pools.
B.3.4 HARVESTED WOOD PRODUCTS (HWP) ACCOUNTING
When thinning occurs, the simulator tracks carbon sequestration in harvested wood products rather than treating all removed biomass immediate emissions. By default, most of the removed biomass carbon (typically 70-80%) is stored as HWP (contributing positively to the carbon objective), while smaller percentage lost during harvest operations (representing processing waste and immediate emissions). This creates an additional management incentive for sustainable harvesting practices and reflects the reality that forest products can provide long-term carbon storage.
B.3.5 WATER BALANCE
The simulator tracks water in three main reservoirs: canopy-intercepted water, the snowpack on the ground, and soil water content (SWC).
Snowpack (SWE): Snow Water Equivalent on the ground (SW Eground) and the canopy (SW Ecan) are tracked. The change in the ground snowpack is given by:
ASW Eground Psnou, throuhhhrough.all Mgrouudd
where Psnow,th throughfall is snow that passes through the canopy and M Mground is melt, calculated from the energy balance. Canopy snow interception is function of LAI and species type (conifers intercept more).
Soil Water Content (SWC): The soil is treated single bucket model. Its water content changes according to:
SSWC Proinnttroughhffl + Moon + M Mground ET Rofs
where inputs are rain and meltwater, and outputs are evapotranspiration (ET, coupled to the energy balance) and runoff (Rofs), which occurs when SWC exceeds its maximum capacity.
B.3.6 CARBON CYCLE
The simulator tracks two primary carbon pools: living biomass (Chimmass) and soil organic carbon (Csoa)) The net change in total carbon is a key reward component.
Gross Primary Production (GPP): Carbon uptake is calculated at each n-minute time-step using the Light Use Efficiency (LUE) model (Monteith) 1972), where GPP is proportional to absorbed photosynthetic radiation (P ARais) and is down-regulated environmental stressors:
GPP = PA Rats LUF (DVVD)))))))) ((SCC
Respiration (R): Carbon losses via respiration from both biomass (autotrophic, R.,) and soil (heterotrophic, R₁). Both are modeled as a function of temperature using the standard Q10 response curve (Van't Hoff & Lehfeldt, 1900):
Q (T-Tr.:)/10)
Roose
where Rbase is base respiration rate reference temperature Tref-
Pool Dynamics: The biomass and soil carbon pools are updated annually based on the integrated fluxes. The net change in biomass is Net Primary Production (NPP GPP R.,) minus losses to litterfall and mortality. The net change in soil carbon is the sum of inputs from litterfall and mortality minus losses from heterotrophic respiration.
AC Chioaass NPP
-
-Laall Mfsve
Minsoet
Mnat
ACsail
Lifali M Mdeadwoodd Rr.
16

---

Published as a conference paper at ICLR 2026
B.3.7 STAND DYNAMICS AND DISTURBANCES
The simulator includes modules for natural population changes and stochastic disturbances that impact forest structure and carbon.
Natural Demography: the end of each year, the model calculates background mortality as function of stand density (self-thinning) and a base stochastic rate. It also calculates natural recruit- ment (new seedlings), which is limited by available space.
Fire Module: A stochastic fire event can occur during the summer. The probability is conditioned on the stand's species composition (conifers are more flammable) and a running drought index. If fire occurs, causes fractional mortality and combusts portion of the biomass carbon, releasing it from the system.
An annual check for insect outbreak is performed. The probability is conditioned on the mean winter temperature (warmer winters increase survival) and stand density. An outbreak causes fractional mortality, primarily targeting coniferous species, with the dead biomass being transferred to the soil carbon pool.
Insect Module:
B.4 PERFORMANCE OPTIMIZATIONS
The simulator includes several performance optimizations for efficient computation:
JIT Compilation: The canopy energy balance solver uses Numba JIT compilation for significant speedup of computationally intensive components.
Memory Management: The environment implements automatic memory management for history tracking, preventing memory leaks during long training runs.
Configurable Physies Backend: The simulator supports two physics backends: a pure Python implementation for compatibility and Numba JIT-compiled backend for improved performance during training.
B.5 ANNUAL SIMULATIOON TIMELINE
The BoreaRL-Sim instance evolves over one-year RL time step following a precise sequence of events. This ensures that management actions and natural processes occur logical order.
1. Management Implementation: At the beginning of the year (t=0), the agent's action is imme- diately implemented. Stems are added (planting) or removed (thinning) according to the action's 's density and species mix specifications. If thinning occurs, the corresponding carbon is removed from the Chiomes pool, representing harvested timber. The age distribution of the forest is up- dated accordingly.
Physical Parameter Update: Based on the new stand structure (density, species mix, age), all physical parameters are recalculated. This includes LAI, canopy area, albedo, roughness length, and interception efficiencies. This step ensures the subsequent physical simulation uses properties that reflect the management action.
3. Sub-Annual Physics Loop:
The simulator runs for 365 days, with time-steps per day depending on step resolution (n). In each time-step:
The weather forcing (temperature, radiation, precipitation) is updated based on the daily and diurnal cycles, with added stochastic noise.
The full energy, water, and carbon balance equations (see Appendix B.2 and B.3 are solved for the current state.
The temperatures of all nodes, snow water equivalent (SWE), and soil water content (SWC) are updated.
A check for stochastic fire event is performed if conditions are met (summer, high drought index).
The dynamic carbon pools (Chiomass- Coost) are updated based on the GPP and respiration fluxes of the time-step.
17

---

Published as a conference paper at ICLR 2026
4. Mad-of-Year Bookkeeping: After the 365-day loop completes, several annual processes resolved:
A stochastic check for an insect outbreak is performed.
Natural mortality (background and density-dependent and recruitment are calculated and applied the stand's age distribution.
All trees in the age distribution are aged by one year.
Final carbon pool values and the final stem density are calculated.
Return Metries: The simulator calculates the final annual metrics needed for the RL re- ward. This includes the net change in total ecosystem carbon, positive and negative energy fluxes across the permafrost boundary, and various carbon pool states. These raw metrics are returned to the ForestEnv wrapper, which processes them into the normalized reward vector R. = [Rearbon.t, 3.
B.6 PARAMETERIZATION
The BoreaRL-Sim parameterized with comprehensive set of physics-based parameters to ensure that trained agents are robust to environmental uncertainty and can generalize across diverse boreal forest conditions. In generalist mode, key parameters are sampled from uniform distributions the start of each episode, while site-specific mode uses fixed parameter values. The parameterization spans climate forcing, soil properties, vegetation characteristics, and disturbance regimes, enabling realistic representation of boreal ecosystem variability.
The parameterization strategy ensures that trained agents encounter realistic environmental vari- ability while maintaining physical consistency. Climate parameters are sampled from ranges rep- resentative of boreal forest latitudes, with temperature and precipitation patterns that capture the seasonal dynamics of northern ecosystems. Soil properties vary within ranges typical of boreal soils, including thermal conductivity and water-holding capacity. Vegetation parameters reflect the contrasting characteristics of coniferous and deciduous species, with different maximum leaf area indices and albedo values. Carbon cycle parameters are sampled from literature- bassd ranges for boreal ecosystems, ensuring realistic carbon fluxes and turnover rates. Disturbance parameters cap- ture the stochastic nature fire and insect outbreaks, with probabilities and impacts calibrated to boreal forest conditions. This comprehensive parameterization enables the environment to serve as a robust testbed for multi-objective forest management under climate uncertainty.
Table 4: Comprehensive parameter sampling ranges for the BoreaRL- Sim.
Parameter
Description
Sampled Range
Climate Forcing Latitude
Site latitude ("N)
[56.0, 65.0] [-10.0, -5.0]
Mean Ann. Temp. Offset
Climate warming/cooling offset (CC)
Seasonal Amplitude
Seasonal temperature swing
[20.0, 25.0]
(CC)
Diurnal Amplitude
Daily temperature variation (*C)
[4.0, 8.0]
Peak Diumal Hour
Hour of maximum daily
[3.0, 5.0]
temperature
Daily Noise Std
Temperature stochasticity
[1.0, 2.0]
(CC)
Relative Humidity
Mean atmospheric humidity
[0.6, 0.8]
Precipitation Patterns
Summer Rain Prob
Daily summer precipitation
10. 0.20]
probability
Summer Rain Amount Winter Snow Prob
Summer rainfall (mm/day)
[10.0, 20.0] [0.15, 0.30]
Daily winter snowfall prob- ability
18

---

Published as a conference paper at ICLR 2026
Table 4 continued from previous page
Parameter
Description
Sampled Range
Winter Snow Amount
Winter snowfall (mm/dayy
[3.0, 8.0]
Soil Properties Soil Conductivity
Thermal
conductivity
[0.8, 1.6]
(Wm//K)
Max Water Content Stress Threshold
Soil water capacity (mm)
[100.0, 200.0]
Water stress threshold (frac- tion)
[0.3, 0.6]
Deep Boundary Temp
Permafrost boundary tem-
[268.0, 272.0]
perature (K)
Vegetation Characteristics Max LAI Conifer
Maximum leaf area index (conifers)
[3.0, 5.0]
Max LAI Deciduous
Maximum leaf area index (deciduous)
[4.0, 6.0]
Base Albedo Conifer Base Albedo Deciduous
Canopy albedo (conifers)
[0.07. 0.11] [0.15, 0.20]
Canopy albedo (deciduous)
Carbon Cycle
Base Respiration
Biomass respiration
rate
[0.30, 0.40)
(kgC/mP/yr)
Soil Respiration
Soil
respiration rate
[0.4, 0.6]
(kgC/mPlyr)
Q10 Factor
Temperature sensitivity of
[1.8, 2.3]
respiration
Litterfall Fraction
Annual biomass rate
turnover
[0.03, 0.04]
Demography Natural Mortality
Annual mortality rate Annual recruitment rate Maximum stand density (stems/ha)
[0.02, 0.03]
Recruitment Rate
[0.005, 0.015)
Max Natural Density
[1500, 2000]
Disturbances
Fire Drought Threshold
Drought index for fire igni- tion
[20, 40]
Fire Base Probability Insect Base Probability
Annual fire probability
[0.0001, 0.0005]
Annual insect outbreak
[0.02, 0.05]
probability
Insect Mortality Rate
Mortality rate during out- [0.02, 0.05]
breaks
Phenology
Growth Start Day Fall Start Day
Day of year for growth onset
[130, 150] [260, 280]
Day of year for senescence onset
Growth Rate
Spring phenology rate
[0.08, 0.15] [0.08, 0.15]
Fall Rate
Autumn phenology rate
C MULTI-OBJECTIVE RL ENVIRONMENT
The BoreaRL environment implements a multi-objective reinforcement learning framework that wraps the physics simulator with standardized interfaces for training and evaluation. The environ- ment conforms to the mo- moogyymmasium API standard and supports both site-specific and generalist training paradigms.
19

---

Published as a conference paper at ICLR 2026
C.I ENVIRONMENT ARCHITECTURE
The environment consists of several key components:
The environment provides rich observation vector that captures the current ecological state, historical information, and environmental context. The observation space varies between operational modes: generalist mode includes episode-level site parameters for robust pol- icy learning, while site-specific mode uses reduced observation space with fixed parameters for location-targeted optimization.
Observation Space:
The environment implements discrete action space that encodes two management dimensions: stand density changes (thinning or planting) and species composition targets. Ac- tions encoded single discrete values representing unique combinations of density change and conifer fraction targets, enabling efficient policy learning while maintaining interpretable manage- ment decisions.
Action Space:
The environment returns a 2-dimensional reward vector [Rearbon,ts Rehow a at each step. Both reward components are normalized the range [-1, 1] per step to ensure compara- ble scales for optimization.
Reward Function:
Carbon Reward (Rearbon,:): Normalized by factor of 2.0 kg Cm Cm-22r" It includes the net carbon change plus stock bonuses, with penalties for exceeding realistic carbon pools (> 15 kg Cm= biomass, 20 kg C m sool).
Thaw Reward (Rthaw,): Normalized by a factor of 40.0 degree-days yr It calculated asymmetric function of conductive heat flux to deep soil: Rthaw X (cooling flux) a X (warming flux), where a = 2.5 a penalty factor that heavily penalizes warming.
Despite the comparable numerical ranges [-1, 1, 1]. the thaw objective is significantly harder to opti- mize due to this asymmetric penalty a and the conflicting physics of the domain (e.g., snow insula- tion vs. albedo effects), rather than a difference in reward magnitude. A theoretical optimal return for both objectives over 50-year episode is estimated approximately 50.0.
Episode Structure: Each episode consists of 50 annual management decisions, with each decision followed by full 365-day physical simulation.
C.2 TRAINING PARADIGMS
The environment supports two distinct training paradigms:
Site-Specific Mode: Designed for controlled studies and Docation-targette optimization, this mode uses deterministic weather patterns, fixed site parameters, and reduced observation dimensional- ity. The environment uses a fixed weather seed, zero temperature noise, and deterministic initial conditions, providing reprodueible results for systematic ablation studies.
Generalist Mode:
Designed for robust policy learning under environmental stochasticity, this mode samples unique weather sequences and site parameters for each episode. The environment includes episode-level level site parameters in the observation space, enabling policies to conditions and climate variability.
adapt to diverse forest
C.3 PREFERENCE CONDITIONING
The environment supports preference-condiiionee training through a preference weight input in the observation space. This enables training single policies that can adapt to different objective weight- ings without retraining. Both fixed preference training (for controlled studies) and randomized pref- erence sampling (for robust generalist policies) are supported.
C.4 RL HYPERPARAMETERS
We evaluate three multi multi-objective RL algorithms across different training paradigms. The agents trained using the hyperparameters listed in 5
20

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameters for Multi-Ohjective RI. Agent Training
Hyperparameter
Variable À EUPG
PPO Gated
Curriculum PPO
Framework
morl-baselines
Custom PyTorch
Custom PyTorch
Learning Rate
1 x 10-3
3 10-4
10-4
Discount Factor (?)
1.0
0.99
0.99
Network Architecture
[128, 64]
[64, 64]
[64, 64]
GAE Lambda
N/A
0.95
0.95
Clip Coefficient
N/A
0.2
0.2
Rollout Steps
N/A
2048
2048
Batch Size
N/A
64
64
Update Epochs
N/A
10
10
Curriculum Threshold
N/A
N/A
0.5
Plant Gate
N/A
Enabled
Enabled
Total Timesteps (Generalist)
3 x 10
3 x 10
3 x 10
Total Timesteps (Site-specific)
1 x 105
1 x 10
x 10
C.5 CURRICULUM PPO MECHANISM
The Curriculum PPO agent employs an adaptive episode selection mechanism to stabilize learning in the generalist setting. The mechanism consists of two components:
1. Episode Selector Network: A fixed, randomly initialized neural network fo : O
(0, 1] that projects site features scalar score. This network is not trained via gradient descent but provides a consistent hashing of the site space. We employ a fixed projection establish a minimal baseline for curriculum efficacy, demonstrating that the adaptive thresholding mechanism itself is sufficient for stabilization without the added complexity learning site-value function. While an optimal ordering of the curriculum could poten- tially yield further improvements, our results show that even this random ordering provides significant benefits.
2. Adaptive Threshold: dynamic threshold that determines whether to accept an episode for training (fe(s) > r). The threshold is updated based on the relative performance of selected versus skipped episodes. If the agent performs better on selected episodes (indicating mastery of the current subset), the threshold is decreased (T < 0.999) to expand the training distribution. If performance on selected episodes is worse than skipped ones, the threshold is increased (r +- x 1.001) confract the curriculum to smaller, more manageable subset.
This approach creates automatic "breathing" curriculum that expands and contracts the effective training distribution based on the agent's 's current competence.
C.6 OBSERVATION SPACE DETAILS
The observation space structure varies between operational modes. generalist mode, the observa- tion vector contains 105 dimensions detailed in Table while site-specific mode uses reduced observation space of 43 dimensions that excludes variable site parameters. The observation vector is designed to provide information about the current ecological state, historical trends, and environ- mental context to enable effective policy learning.
Table 6: Detailed breakdown of the observation space structure (gener-
alist mode).
Index Description
Normalization
Category 1: Preference Input (1 dimension)
21

---

Published as a conference paper at ICLR 2026
Table continued from previous page
Index
Description
Normalization
0
Carbon Preference Weight (wc)
[0, 1] (no change)
Category 2:
Current Ecological State (4 dimensions) Year
1
year/50
2
Stem Density (stems ha ha-))
density/1500
3
Conifer Fraction
[0, 1] (no change)
4
Total Carbon Stock (kgC m-)
(biomassc + soilc)/50
Category 3:
Site Climate Purameters (6 dimensions)
5
Latitude (?N)
(lat 50)/20
6
Mean Annual Temperature (*C)
(Tmeon + 10)/20
7
Seasonal Temperature Amplitude ("C)
Tamp/30
8
Growth Start Day (DOY)
growth.day
9
Fall Start Day (DOY)
fall_day /35
10
Growing Season Length (days)
(fall day growth
day) growth.day) /00
Category 4: Disturbance History (6 dimensions)
11-12
Fire Mortality Fraction (last 2yr)
0, 1] (fraction) 0, 1] (fraction) index/100 1000
13-14
Insect Mortality Fraction (last 2yr)
15-16
Drought Index (last 2yr)
Category 5: Carbon Cycle Details (7 dimensions)
17
Recent Biomass C Change
(change 0.5)/1.0
18
Recent Soil C Change
(change + +..2/0044
19
Recent Total C Change
(change 0.7)/11.4
20
Recent Natural Mortality
mortality/0.5
21
Recent Litterfall
lilter
22
Recent Thinning Loss
(loss + 0.5)/1.0 hwpp/055
23
Recent HWP Stored
Category 6: Management History (4 dimensions)
24
Recent Density Action Index
action /4
25
Recent Mix Action Index
action/ / (change + 100)/200 change (no change)
26
Recent Density Change
27
Recent Mix Change
Category 7:
Age Distribution (10 dimensions)
28-32
Conifer Age Fractions (5 classes)
0, 1 (fraction) 0, (fraction)
33-37
Deciduous Age Fractions (5 classes)
Category 8:
Carbon Stocks (2 dimensions)
38
Normalized Biomass Stock
biomass/50 soil/50
39
Normalized Soil Stock
Category 9:
Penalty Information dimensions)
40
Biomass Limit Penalty
penalty/O0 00.5
Soil Limit Penalty Max Density Penalty
penalty/O. 00.5 penally/1.0
42
Category 10: Site Parameter Context (62 dimensions, generalist only) 43-104
Site-specific physics parameters
Normalized to [0, 1] ranges
The normalization strategy ensures that all components are scaled to approximately [0, 1] ranges for stable learning, while preserving the relative magnitudes and relationships between different ecological variables. The preference input enables preference-comnditooee policies to adapt their behavior based on the desired objective weighting.
22

---

Published as a conference paper at ICLR 2026
C.7 ACTION SPACE ENCODING DETAILS
The environment uses discrete action space with 25 unique actions (5 density actions x conifer fractions) detailed in Table Actions encoded single integer values where the density action index and conifer fraction index are combined using integer division and modulo operations.
Table 7:
Action Space Encoding for Forest Management
Action Index
Density Change (stems/ha) Conifer Fraction
0-4
-100
0.0, 0.25, 0.5, 0.75, 1,0
5-9
-50
0.0, 0.25, 0.5, 0.75,
10-14
0
0.0, 0.25, 0.5, 0.75, 1.0 0.0, 0.25, 0.5, 0.75,
15-19
+50
20-24
+100
0.0, 0.25, 0.5, 0.75, 1.0
Management constraints include thinning restrictions to maintain minimum density of 150 stems/ha, with thinning operations removing oldest trees first (101+ years) when available. Plant- ing operations add scedlings (0-5 years) to the stand, with maximum density of 2000 stems/ha that cannot be exceeded. If planting is attempted at maximum density, penalty is applied. Species composition is controlled by the conifer fraction parameter, allowing for mixed-species management strategies.
Carbon Accounting: Net carbon change calculations include biomass, soil, and HWP carbon components. Carbon limits are enforced with penalties rather than hard caps, with biomass carbon limited 15.0 kg C/m² and soil carbon limited 20.0 kg C/m², Excess carbon beyond these limits incurs proportional penalties to discourage unrealistic carbon accumulation.
Age Distribution Management: The system tracks five age classes: seedling (0-5), sapling (6- 20). young (21-50), mature (51-100), and old (101+ years). Natural mortality and recruitment oc- cur annually, while management actions modify the age distribution based on species preferences. Age-weighted canopy factors affect light use efficiency and growth rates, creating realistic stand dynamics.
C.8
REWARD FUNCTION MATHRMATICAL FORMULATION
The reward function returns two-dimensional vector [rearbon: Tthaw] with the following compo- nents:
C.8.1
CARBON REWARD (r:))
Fc = clip(c, + 86 + hb pt Pa Ps. 1.0,, 1.0)
Where:
AC
en clip 1.0, 1.0)
2.0
88 0.0 x clip( C₁
0.0, 1.0)
50.0
h
h6 0.0 x clip(
0.0, 1.0)
1.0
Pt P6 + Ps Eb x 0.5
15.0 6,
x 0.5
20.0
[1.0
if d > 2000
Pd
0.0
otherwise
23

---

Published as a conference paper at ICLR 2026
C.8.2
THAW REWARD (rt)
Fr clip(
-1.0, 1.0)
40.0"
Where:
a: = fn 2 2.5 x fp
The carbon reward re consists of normalized net carbon change Cn. (where AC' is the net carbon change including HWP in kg C/m²/yr), stock bonus 8, is based total carbon stock C. and HWP sales bonus ha is based on HWP carbon stored h. Penalties include total limit penalties pi (sum of biomass penalty ps and soil penalty Ps. where eb and e, are carbon excesses beyond limits), density penalty Pa (applied when stem density d exceeds 2000 stems/ha), and ineffective action penalties P;- The thaw reward re based on asymmetrie thaw a, -eegree-daysyy),. which combines positive heat flux fp and negative heat flux fn to deep soil (permafrost proxy) with 2.5:1 penalty ratio for warming versus cooling.
Regarding validation, both carbon and thaw rewards are constructed using existing common knowl- edge from literature about how these fluxes operate. Growth of carbon stock, carbon capacity of forest and soil, overplanting, excessive thinning, etc common ways to think about the health of forest in forest management. Thaw degree days and fluxes into and out of the soil are common ways to calculate permafrost thaw. Apart from this existing base, more components can be added to these depending on user preference and is subjective. Therefore, in some sense, these reward formulations already validated from existing literature.
C.9 COMPUTATIONAL COMPLEXITY AND RUNTIME ANALYSIS
Computational cost per episode is 1-22-1 seconds per 50-year episode on a standard CPU (Intel 17/19 AMD Ryzen, single core). With Numba JIT it takes about ~5-7 seconds. The variance de- pends on the number of disturbances (fire/insect events trigger additional computation) and whether the canopy energy balance solver converges quickly (dependent on weather conditions).
In Generalist Mode, the total timesteps is 300,000 (6,000 episodes x 50 steps/episode) and training time is ~8-12 hours on a standard CPU workstation. In Site-Specific Mode. the total timesteps is 100,000 (2,000 episodes x 50 steps/episode) and training time is ~-34 hours on standard CPU workstation. These estimates include forward simulation (physics + reward computation), PPO policy/value network updates, Logging and checkpointing.
The primary computational cost is the sub-daily physies loop. For each of the 50 annual timesteps, the simulator runs 365 days x (1440 minutes / 30-minute resolution) 17,520 physics steps. The canopy energy balance solver (iterative (ewton-Raphson) accounts for ~60-70% of this cost. Train- ing throughput scales well with CPU cores. With 16 parallel workers, generalist training can be completed in 1-2 hours.
Per-environment memory footprint is ~50-100 MB. Training single PPO agent (network param- eters, replay buffer) is ~2200-50 MB. Total for 16 parallel envs + agent ~23 GB RAM, easily feasible on modern workstations.
In absolute terms, BoreaRL is trainable on commodity hardware (laptop or workstation). full train- ing costs $1 in cloud compute (AWS EC2 c5, 4xlaxlarg.). Relative to other poysicallyggroundde simulators, BoreaRL. is efficient, achieving a balance between physical realism and RI. tractability. We will be exploring JAX/GPU acceleration in future versions.
D ADDITIONAL RESULTS
This section provides additional analysis supporting the claims in the main paper.
D. SITE INFLUENCE ON THAW PERFORMANCE
Site characteristics strongly influence thaw performance and learning stability. Certain sites enable much higher thaw objective values than others. demonstrating that site selection fundamentally de- termines achievable performance regardless of management decisions. The high volatility in thaw
24

---

Published as a conference paper at ICLR 2026
reward learning across algorithms supports the importance of curriculum basedd approaches for sta- ble learning.
Thaw/Reward
Thaw Objective
Site Seed 123
789
Site Seed 456
Site Seed 42
0.0
-8-
-10
0.2
Variable
PPO Galed
-12
-0.4
Curricutum
200
1000
Objective
(Last 1000)
(a)
(b)
Figure 5: Site influence on thaw performance. (a) Thaw objective clustering by site characteristics.
(b) Training volatility thaw reward learning across algorithms.
D.2
EMPIRICAL TRADE-OFF COVERAGE METHODS
Figure 6 shows the carbon- ttaw trade-offss achieved by different algorithms.
Se0.44
1.40
5
Thaw Objective
1.35
Thaw Objective
4-
1.30
1.25
2.
1.20
1+0.78
-0.3
-0.2
00.1
0.0
0
Carbon Objective
Carbon Objective
(a)
(b)
S-630
Thaw Objective
Thaw Objective
2
0
320.00
-2
7.4
6
8
9
4
6
10
Carbon Objective
Carbon Objective
(c)
(d)
Figure 6: Empirical trade-off coverage for different algorithms. We plot for all weights (averaged across 100 episodes each; summed over 50 steps) to visualize the density and range of learned policies. The rewards are (a) weight Composition (b) Variable À BUPG (c) PPO Gated (d)
Curriculum PPO.
Lambda Monotonicity Analysis: Curriculum PPO achieves the best preference control with 50% monotonicity violations, while other methods show poor preference adherence: Fixed-Weight
25

---

Published as a conference paper at ICLR 2026
(75%). Variable À EUPG (66.7%), and PPO Gated (100% violations). Curriculum PPO provides the most reliable trade-off behavior for practical applications.
D.3 MECHANISTIC EVIDENCE AND ANALYSIS
This section provides detailed mechanistic evidence supporting the claims made in the main text, linking agent behaviors to physical processes.
Table 8: Correlation between Growing Season Length and Thaw Protection Mechanisms, Longer growing seasons are strongly correlated with better thaw protection, driven by increased transpira- tion cooling and reduced ground radiation.
Variable
Correlation (r)
Interpretation
Thaw Reward
+0.65
Longer season - Better Thaw protection
Latent Heat (LE)
+0.82
Longer season + More transpiration cooling
Ground Radiation
-0.75
Longer season -> Less solar heat on soil
Table 9: Causal Chain of Learned Strategies. Different algorithms converge to distinct local optima with specific physical mechanisms, outcomes, and algorithmic causes.
Agent
Action Strategy
Physical
Outcome
Algorithmic Cause & Evidence
Mechanism
PPO Gated
Carbon Farming (High Density, High Conifer)
High Biomass:
High Carbon ( 9.0)
Gradient Dominance:
Maximizes Carbon.
Dense Carbon signal
Albedo:
High Warming
overpowers noisy Thaw signal. Evid: 100% X-violations Thaw -5.0.
Conifers absorb heat.
(z -5.0)
Variable
Thaw Avoidance (Low Density, Deciduous)
High Albedo:
Low Carbon (ss 1.0) Max Cooling
Policy Collapse: Conflicting gradients Iead to risk-averse "inaction".
Reflects sunlight.
Low Biomass:
Minimizes Carbon.
(+= 8.0)
Evid: Lowest Reward (1.7); Hypervol 14.2.
Curriculum Balanced Cooling (Mod. Density, Mixed)
High Transpiration:
Good Carbon (ze 8.0) Good Cooling (Av 6.0)
Gradient Filtering: Removes "trap" sites, enabling complex learning. Evid: High Hypervol (84.3): Sparsity 0.12.
Cools air.
Mod. Albedo: Mixed reflection.
Table 10: Comparison of Thaw Reward Formulations and Agent Behavior. The Asymmetric formu- lation forces strong avoidance of warming, whereas symmetric formulations allow small warming trade-offs.
Formulation
Warming Penalty
Agent Behavior
Resulting Warming Flux
Raw DD Contrast
Symmetric (1.0) Ratio-based
Accepts small warming Accepts warming Cooling high
5.0 MJ
2.0
Asymmetric
Strong (2.5x)
Avoids Strongly
2 0.1 MJ
26

---

Published as a conference paper at ICLR 2026
Table 11: Physical Conflicts between Carbon and Permafrost Objectives.
Physical Variable
Effect on Carbon (C)
Effect on Permafrost (7)
Conflict?
Stem Density
(More Biomass)
J. (Interception) /
Yes (Complex)
(Absorption)
Conifer Fraction
t (Higher Density)
J. (Lower Albedo + Warmer)
Yes (Direct)
LAI
(More Growth)
1 (Shading/Cooling)
No (Synergy)
Table 12: Curriculum Selection Statistics: Characteristics of Accepted vs. Rejected Sites. curriculum filters out sites with high warming potential.
The
Site Category
Avg. Latitude
Avg. Potential Warming Flux
Accepted Sites
High (> N)%N)
Low (es 0.5 MJ)
Rejected Sites
Low (< 60%N)
High (+2 8.0 MJ)
D.4 DETAILED OBJECTIVE ANALYSIS
Table provides detailed breakdown of the Carbon and Thaw objectives achieved by RL base- lines. Curriculum PPO baseline consistently achieves high values in both objectives, demonstrating superior trade-off management. In contrast, PPO Gated achieves high Carbon scores but suffers significant penalties in the Thaw objective, often resulting in negative values due the asymmetric warming penalty, Variable EUPG fails to learn effective strategies for either objective, clustering near zero.
Table 13: Detailed breakdown of Carbon and Thaw objectives for RL baselines. Values correspond to means and errors represent standard deviation over 100 evaluation episodes.
Method & Preference (A)
Carbon Objective
Thaw Objective
Curriculum PPO
À = 0.00
2.5 = 1.5
7.8 + 1.2 8.1 1.4
À 0.25
7.0 1.8
A 0.50
8.5 = 2.0
7.6 1.5
À 0.75
9.5 1.5
7.7 1.3
À 1.00
10.0 + 1.0
7.3 + 1.6
PPO Gated
À = 0.00
8.8 = 2.0
-.2 + 2.5
À 0.20
7.2 = 2.0
2.5
A = 0.40
7.7
3.1
À 0.60
8.4 1.5
2.0
A = 0.80
7.6 1.5
1.0
À 1.00
7.1 1.0
- + 1.5
Variable EUPG
À 0.00
-0.7 1.5
2.8
À = 0.33
1.7 17=15
2.2220
X 0.44
0.5
5.8
A = 0.67
1.9
1.1
A 0.78
-0.8
-0.2
A 0.89
-1.8
1.0 1.5
27

---

Published as a conference paper at ICLR 2026
D.5 TRAINING DYNAMICS AND STRATEGY EVOLUTION
Figure 7 shows training dynamics for conifer fraction and stem density, revealing distinct algo- rithmic learning patterns. PPO Gated pursues aggressive early carbon strategies with high conifer fractions and rapid density growth followed by decline. Curriculum PPO shows steady learning with moderate improvements in both metrics. Variable EUPG maintains conservative species composition but exhibits sustained density growth throughout training. PPO Gated shows strong generalization from training evaluation (see Fig. (),, while Variable EUPG exhibits potential overfitting with poor generalization in density management. Curriculum PPO demonstrates stable learning across both phases.
0.625
Variable À EUPG (Training)
1200
PPO Gated (Training)
Coniter Fraction
0.6000
Curriculum PPO (Training)
Stem Density (ha
1000
0.575
800
0.550
0.525
Variable EUPG (Training)
PPO Gated (Training)
0.500
400
Curriculum PPO (Training)
0.475
0
10
20
30
40
50
10
20
30
40
50
Step
Step
(a)
(b)
Figure 7: Training dynamics of algorithm strategies. (a) Conifer fraction evolution during training.
(b) Stem density evolution during training.
D.6 MULTI-OBJECTIVE TRADE-OFFS AND CLIMATE ADAPTATION STRATEGIES
Figure sJsooss algorithm performance under varying environmental conditions and objective trade- offs. Curriculum PPO achieves superior multi-objective performance with high rewards in both carbon and thaw objectives. PPO Gated prioritizes carbon performance, while Variable À EUPG shows inconsistent exploration.
D.7 INDIVIDUAL OBJECTIVE PERFORMANCE ANALYSIS
Figure shows individual carbon and thaw objective performance for each algorithm. PPO Gated and Curriculum PPO achieve superior carbon performance, while Curriculum PPO dominates thaw optimization. Variable À EUPG shows poor performance in both objectives. Curriculum PPO's balanced multi-objective approach explains its superior scalarized performance.
D.8 FOREST DEMOGRAPHICS AND COMPOSITIONAL DYNAMICS
This section provides an analysis of forest demographic trajectories and compositional preferences, offering detailed insights into how different algorithms manage forest structure and species compo- sition over time. Figure I0 presents two complementary analyses: age-class specific stem density trajectories and conifer fraction distributions during training and evaluation.
Age-Class Trajectory Analysis: The age-class trajectories (Panels a-e) reveal distinct manage- ment strategies across different forest developmental stages. For younger age classes (seedling, sapling, young), all algorithms show initial high stem densities followed by rapid decline, reflect- ing natural mortality and competition processes. However, the recovery patterns differ: Variable EUPG demonstrates sustained high densities in mature and old age classes, particularly for decidu- ous trees, suggesting a strategy focused on long-term forest productivity and carbon storage. PPO Gated shows strong early growth in conifer age classes but experiences more pronounced declines, indicating a strategy that prioritizes rapid establishment but may sacrifice long-tern stability, Cur- riculum PPO exhibits intermediate patterns, balaneing growth and sustainability across age classes.
28

---

Published as a conference paper at ICLR 2026
Variable X EUPG
0.4
PPO Gated
Curriculum
Thaw Thaw Reward
0.0
0.0
Thaw
-0.2
-0.2
Variable A EUPG
0.6
Curriculum PPO
-0.6
-0.8
-0.6
-0.4
-0.2
0.2
0.4
-0.6
-0.4
-0.2
0.0
0.2
0.4
Carbon Reward
Carbon Reward
(a)
(b)
PPO Gated
PPO Gated (Evaluation)
Curriculum PPO
Curriculum PPO (Evaluation)
Total Stem Density (ha -1
1200
0.2
Roward
0.0
800
400
-0.6
60
100
1i0
120
0.3
0.9
10
Growing Season (days)
Conifer
(c)
(d)
Figure 8: Algorithm performance analysis. (a) Carbon thaw rewards during training. (b) Carbon vs. thaw rewards during evaluation. (c) Growing season vs. thaw reward during evaluation. (d) Forest demographics during evaluation.
These trajectory differences explain the observed carbon and thaw performance variations, as ma- ture and old forests contribute significantly to both carbon sequestration and permafrost protection through their insulating effects.
The conifer fraction distributions (Panels f-g) reveal funda- mental differences in species composition preferences. During training (Panel f), Variable À EUPG shows narrow, peaked distribution around 0.45-0.5 conifer fraction, indicating lack of special- ized strategy. PPO Gated exhibits a broader distribution with preference for higher conifer fractions (0.6-0.9), suggesting a strategy that promotes conifer dominance. Curriculum PPO shows interme- diate preferences, with broader distribution centered around 0.55-0.6, indicating adaptive compo- sitional management. The evaluation distributions (Panel g) confirm these training preferences, with Variable A EUPG maintaining moderate conifer fractions (0.35-0.65). PPO Gated strongly favor- ing high conifer fractions (0.75-0.95), and Curriculum PPO showing balanced preferences around 0.6-0.65. These compositional strategies directly impact both carbon and thaw objectives: higher conifer fractions generally support carbon sequestration through increased biomass, while balanced compositions may better support permafrost protection through modified energy and water fluxes.
Compositional Strategy Analysis:
E EXTENDED FUTURE WORK
Additional Multi-Objective Approaches: Future work will incorporate evolutionary algorithms (e.g., NSGA-II, MOEA/D) as population-basee alternatives, hypervolume-base hasse methods directly
29

---

Published as a conference paper at ICLR 2026
PPO Gated
Carricukm 191
0.150
0.125
Rowald
Veriable EUPS
0.075
Why
0.050
0.025
-0.05
Eploode
Estsode
(a)
(b)
0.25
0.20
0.05
0.00
0.00
-0.5
-0.05
0.10
0.10
-0.15
Varabie EUPO
PRO Gardd
Carticttm PPO
Variattee EUPO
PRo Gamed
Cartcattm PPD
Mechod
Machod
(d)
Figure 9: Individual objective performance analysis. (a) Carbon learning curves during training.
(b) Thaw learning curves during training. (c) Carbon performance during evaluation. (d) Thaw performance during evaluation.
optimize trade-off quality, and model-based MORL to improve efficiency in long-horizon permafrost dynamics.
Spatial and Temporal Scaling: Develop hierarchical action spaces for landscape scaaee manage- ment, incorporating spatial interactions between stands, temporal coordination of management tivities, and integration with regional climate models for improved environmental forecasting.
Computational Acceleration: Enable massive parallelization GPUs/TPUs with JAX/PyTorch. This would allow for end-to-end vectorization, significantly faster training times, and the ability to scale to much larger populations of agents or more complex environmental simulations.
30

---

Published as a conference paper at ICLR 2026
Vy trabic EUPC
140
PRO Cuted Coett
200
250
Durtakin PPO Deciiiess
Stam 04000 (he
29
Vatasa CUPS Carde
Uniible EURS Certer
Vetson doead
EURG Decitss
40
PPO Dockteus
Carresun FPO Conde
100
Cuntoalm P80 cetfer
cand Decdoous
Carrouu FPO Cecdioss
Pa0 Deckanu
Stp
(a)
Vor EUPO. DOC
16-
Veribte EUPO DesiRess
PRO Sated Conder
PRO Oxced Colfar
Durikkk PPO Deciiiess
Donisstr IPP Dax
wwm.batt
Siep
(d)
(e)
6-
Variable À EUPG (Training) (Training)
Variable EUPG (Evaluation)
PPO Gated (Evaluation)
Curriculum PPO (Training)
5-
Curriculum (Evaluation)
4.
Donoty
Density
2-
14
0.6
0.8
0.2
0.4
0.6
0.8
1.0
Coniier Fraction
Contfer Fraction
(f)
Figure 10: Detailed forest demographics and compositional analysis. (a) Seedling trajectory show- ing stem density evolution over 50 simulation steps. (b) Sapling trajectory illustrating early forest de- velopment patterns. (c) Young trajectory demonstrating intermediate growth dynamics. (d) Mature trajectory revealing long-term forest structure management. (c) Old trajectory showing late-stage forest dynamics.
(f) Conifer fraction distribution during training phase, revealing compositional ex- ploration strategies. (g) Conifer fraction distribution during evaluation phase, showing learned com- positional preferences. All panels illustrate how different algorithms (Variable À EUPG in purple, PPO Gated in green, Curriculum PPO in brown) manage forest structure and species composition across developmental stages.
31


Reviews:
[{"review_id": "desBXeuoYW", "reviewer": "Reviewer_qus2", "summary": "This paper introduces BoreaRL, a multi-objective RL environment for boreal forest management. BoreaRL consists of two components: (1) BoreaRL-Sim, a physics-based simulator running at tunable minute-scale resolution, and (2) BoreaRL-Env, a wrapper around BoreaRL-Sim that implements the mo-gymnasium API and accepts annual agent actions. The “multi-objective” aspect of BoreaRL refers to the dual objectives of carbon sequestration and permafrost preservation. Each episode has 50 time steps, representing 50 years, and the agent’s actions control the tree density as well as ratio between deciduous and coniferous tree species. Experiments across 4 different RL algorithms (fixed-weight EUPG, variable-weight EUPG, PPO Gated, Curriculum PPO) find that Curriculum PPO achieves the best pareto frontier in the multiobjective optimization.", "questions": "Q1) Besides RL, what other tools do forest policy managers traditionally use? Is there any way to compare against such a non-RL baseline?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**S1) Clarity of writing**\n\nThe paper is well-written and generally easy to follow. The motivation is clear, and the conclusions are straightforward to follow.\n\n**S2) Novel multi-objective RL testbed**\n\nThe paper presents a novel, and real-world-motivated multi-objective RL problem. Furthermore, the tested algorithms show a large spread in performance, suggesting that the benchmark is useful for testing RL algorithm performance.", "weaknesses": "**W1) Questionable claim that permafrost preservation objective is hard to optimize**\n\nA central conclusion from the paper is that the permafrost preservation objective is harder to optimize than the carbon objective. However, I am not fully convinced of this claim. To make this claim stronger, the paper should clarify the scale and range for each of the two reward terms $R_{carbon,t}$ and $R_{thaw,t}$. For instance, if $R_{thaw,t}$ has a range [0, 1] whereas $R_{carbon,t}$ as a range [0,100], then naturally we would expect it to be challenging for an RL algorithm to optimize the permafrost objective.\n\nAlso, is there any estimate of what the optimal return might be for the permafrost objective?\n\n**W2) Conclusions can be made stronger with more repeated trials**\n\nIt is not clear to me which of the results reported are averaged over multiple runs, vs. a single result. For example, Figure 3b shows error bars, but I am confused by what the error bars are supposed to represent. (Is it showing standard deviation or standard error? Over how many random seeds? And do the random seeds only affect the test episodes, or was the RL algorithm trained multiple times with different random seeds?)\n\nAlso, the monotonicity plots (e.g., Figure 3d and elsewhere in the appendix) could benefit from runs with multiple seeds.\n\n**W3) Importance of contribution to AI community is a bit unclear**\n\nWhile I recognize that BoreaRL is “the first multi-objective reinforcement learning environment for climate-adaptive boreal forest management,” I am not entirely sure how to evaluate the importance of this contribution for an AI conference like ICLR. The paper evaluates several RL algorithms, but it is unclear how BoreaRL compares to other multi-objective RL benchmarks. Suppose some researcher discovers an optimal RL algorithm for BoreaRL; what would be the impact of that?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ryDnc3yE7A", "reviewer": "Reviewer_f77E", "summary": "This paper considers the application of multi-objective reinforcement learning (MORL) methods to the problem of boreal forest management. The main contributions are (1) formulating boreal forest management as two variants of a MORL problem based on physical simulations; (2) implementing these problems as a simulator and environment compatible with a common MORL benchmarking library; (3) experimentally comparing the performance of multiple different MORL approaches on these problems and discussing the results.", "questions": "Reward Design and Validation: Could you provide more detail on how the reward components (especially the thaw-related one) were designed and validated? Did you consider alternative reward formulations, and if so, what were the results?\n\nEvaluation Metrics:  You mention λ-monotonicity and show scalarised rewards, but have you evaluated more standard multi-objective metrics, such as hypervolume or sparsity, to better capture the quality and diversity of Pareto fronts? If not, could you elaborate on why these were not used?\n\nDefinition and Interpretation of Pareto Front: In Figures such as 3(d), you label plots as Pareto fronts but they appear to include both dominated and non-dominated solutions. Can you clarify your definition of the Pareto front and explain how the solutions were identified as non-dominated?\n\nSimulator and Environment Use: Given the level of detail in the simulator, how do you envision users might extend or adapt BoreaRL? For example, can users easily modify the reward components, action spaces, or add new ecological processes within your framework?\n\nGeneralizability of the Findings: The paper mainly focuses on preference-conditioned MORL. Have you tested or do you plan to test your benchmark environment with other types of multi-objective learners?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Interesting approach to a relevant application problem. This is the first work to propose the use of MORL for forest management.\n\n2. Judging from a brief scan of the supplementary material, the underlying physical modelling appears thorough.\n\n3. Implementation as a modular extension of an existing benchmarking library.", "weaknesses": "1. Not convinced by the focus of the paper. The authors mention that this problem has never been formulated as an MORL problem before, and not in such detail as a single-objective problem. As such, the modelling and formulation of the problem itself seems like a relevant contribution, yet little discussion is devoted to this in the main paper. More consideration should be given to the design of rewards, particularly as the validation experiments show that the thaw reward is difficult to optimise for even when used as the only objective. Different rewards could be explored and compared experimentally.\n\n2. Following from the first comment, the comparison of different algorithms and design of a new solution approach seems premature. (Perhaps this work could even be split in two: one part focussed on the design of the problem and the description of the implementation, and another on improving algorithmic approaches.) Too much of the problem formulation is only discussed in the appendix.\n\n3. Scalarised reward may not be the best metric to evaluate algorithmic performance, given that the reward functions appear to operate on different (effective) intervals. For example, the two solutions for $\\lambda = 0.75$ and $\\lambda = 1.0$ in Fig. 3(d) are both on the Pareto front, but one would return a higher scalarised reward than another. Multi-objective metrics (e.g. hypervolume, sparsity) would be more suitable for evaluation.\n\n4. Some terms used in the paper appear to be undefined, e.g. λ-monotonicity, or misused, such as the Pareto front. Figures 3(d) and 6 (a-d), for example, are all described as showing Pareto fronts (i.e. the set of non-dominated solutions), but appear to draw no distinction between dominated and non-dominated solutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f3MfxN67pv", "reviewer": "Reviewer_vnbU", "summary": "The paper proposes BoreaRL, a physics-based multi-objective RL environment that simulates how climate-adaptive boreal forest management decision impacts on the forest. It is a physically-grounded simulator. Its observation space include components such as ecological state, site climate parameters, historical information and episode-level site parameter context. Its action space include two discrete action: stand density change and species mix change. The reward function is a weighted sum of normalized net carbon change and thaw reward, with the weight set by the preference weight parameter. The paper experiments and compares with the following RL algorithms: Fixed-$\\lambda$ EUPG, Variable-$\\lambda$ EUPG, PPO Gated and Curriculum PPO.\nMain contribution of the paper includes (1) a physics-grounded simulator that captures complex biogeophysical trade-offs (2) a modular MORL framework supporting site-specific and generalist training paradigms (3) benchmarking shows Curriculum PPO outperforms other algorithms (4) novel ecological insights for appropriate forest management.", "questions": "1. **Claim about superiority of Curriculum PPO**: The claim \"In generalist settings, standard preference-conditioned approaches fail entirely, while a naive curriculum learning approach achieves superior performance by strategically selecting training episodes.\" is questionable. My confusion is two-folded.\n     - To begin with, curriculum learning usually refers to gradually increasing task difficulty, shaping the distribution of episodes, selecting “easier” tasks first, which is not what the paper does. Therefore calling it \"curriculum learning\" is confusing. The implementation includes an episode selector network to decide whether to include a site-episode observation into training data, but claims it is for inference only. I am confused about at which step is the episode selector network trained.\n    - The preference weight input in the observation space of Curriculum PPO. Therefore the Curriculum PPO is also preference-conditioned. Standard preference-conditioned approaches here just refers to gradient descent method. Therefore the claim boils down to \"PPO with episode selection works better than gradient descent\".\n\n2. **Multi-objective balancing**: The idea of balancing between carbon and thaw objectives encourages studying Pareto-optimal behavior and decision-making under competing goals, which is both scientifically valuable and underexplored in RL benchmarks.\n\n3. **Validity concerns**: The path from \"trained policy\" to \"deployable forest management\" is unclear.\n\n4. Overall, the paper would be stronger if it either:\n    - Went deeper on methodology: Why does curriculum help? Can we design MORL algorithms specifically for this type of asymmetric objective difficulty?\n   - Went deeper on science: Validate against real forest data, partner with foresters, demonstrate actual deployment potential", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The major strength is a physically-grounded, process-based and open-source simulator with a wide range of configurations. The authors show a high level of expertise in the boreal ecosystem covering three distinct models: energy, water, and carbon fluxes on a n-minute time step. It captures real biogeographical trade-offs, such as albedo, rain and snow patterns and insect disturbance. It is a valuable contribution to the domain of real-world environmental policy and management research.", "weaknesses": "Section 3.4: The paragraph ends with \"The objective follows standard PPO:\" seems unfinished.\n\nSection B.2.1 The canopy energy balance equation should not be equal to 0.\n\n**Analysis of Learning Objectives**: Despite applying multiple learning algorithms, the RL objective optimize for the same cumulative reward, so simply comparing “actions” (stand density change and conifer fraction) across algorithms without deeper causal insight doesn’t illuminate how or why they differ.\n\n**Circular reasoning on fundamental asymmetry in learning difficulty:**: The author design a forest simulator which calculates annual carbon objective and thaw objective. The thaw objective is more complex than the carbon objective, and therefore the PPO algorithm struggles to optimize for the thaw objective. Instead of acknowledging the root cause of the difference in the learning difficulty is the reward shaping, the author claim this reveals something fundamental about the ecosystem management problem. But the difficulty is largely self-imposed through reward engineering choices.\n\n**Novelty concerns**: Creating a simulator and running off-the-shelf RL agents—is relatively commonplace in applied RL research. The main contribution is domain-specific realism, not algorithmic innovation. In addition, there exists papers in the environmental science space that does something very similar.\n- Overweg, H., Berghuijs, H.N.C., Athanasiadis, I.N. (2021). CropGym: a Reinforcement Learning Environment for Crop Management.\n- Lapeyrolerie, M., Chapman, M.S., Norman, K.E.A., Boettiger, C. (2021). Deep Reinforcement Learning for Conservation Decisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LLADDPzMDK", "reviewer": "Reviewer_3zMh", "summary": "The authors propose a novel RL environment modelling the management of boreal forests. The environment tasks RL algorithms with two objectives: maximizing carbon sequestration and  minimizing permafrost thaw. The environment is built on a complex thermodynamic simulator that models the energy balance of different parts of the forest stand. The simulator also models the carbon cycle, water balance and various weather patterns. Finally, the environment allows for training only on specific site or from a site/weather sampled from some distribution.\n\nTo test their environment, the authors test both expected utility policy gradient (with different preference weights) and PPO. Experimentally, they observe that it is easier to optimize for carbon objectives than thaw objectives. To address the difficulty, they use curriculum learning and show that it outperforms other methods and learns to improve performance on thawing objectives.", "questions": "- How close is the model of possible actions to the actual decisions humans need to make? It feels like there is a lack of granularity (though I am unfamiliar with the intricacies of forest management).\n\t- Similarly, the long time horizon between actions is interesting. I'm guessing most actions are taken over long time horizons for forest managements?\n- You mention the use of numba and jit to improve performance. Are there any plans to create a Jax version of the environment (given the recent trend towards GPU simulation of RL environments)?\n\t- How expensive is the environment to run/training the PPO methods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well-written and motivated, it does a good job of presenting the subject as a valuable application area for ML.\n- The work is novel and appears to address a gap in the literature.\n- The environment involves a significant amount of work and attention to detail to model a large amount of aspects of the real world in the physical simulation.\n- The authors test various RL methods on the environment and carefully analyze their performance.", "weaknesses": "My main concerns are as follows. I am willing to increase my score if these are at least somewhat addressed. Essentially, I need to be convinced that (a) the simulator is a reasonable model of real world dynamics (b) the forest management problem requires complex decision making that benefits from reinforcement learning methods.\n\n- Looking at the appendix, the underlying simulator seems quite complex. I do not have the necessary expertise to judge how realistic it is. Could you point me to any evidence/papers to show this model is reasonable and grounded in real-world dynamics? The paper in general could benefit from additional justification of some of these decisions.\n- You mention you believe the benchmark is far from saturated. To provide evidence of this, it would be helpful to provide additional baselines based on heuristics or simpler methods (e.g. what would be a reasonable baseline policy that a human would implement)? If the RL methods noticeably outperform this it would provide further evidence of the value of RL here.\n\t- The +100 density/0 density baselines are a step in the right direction but I'm guessing there are stronger baselines?\n- RL performance plots would benefit from error bars if these are not too expensive to compute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)