Title: Novel Exploration via Orthogonality
Venue: NeurIPS 2025
Accepted: yes
Keywords: Laplacian

Novelty

Reinforcement Learning

Exploration

Eigenvectors

Spectral Methods
Abstract: Efficient exploration remains one of the most important open problems in reinforcement learning. Discovering novel states or transitions requires policies that efficiently direct the agent away from the regions of the state space that are already well explored. We introduce Novel Exploration via Orthogonality (NEO), an approach that automatically uncovers not only which regions of the environment are novel but also how to reach them by leveraging Laplacian representations. NEO uses the eigenvectors of a modified graph Laplacian to induce gradient flows from states that are frequently visited (less novel) to states that are seldom visited (more novel). We show that NEO's modified  Laplacian yields eigenvectors whose extreme values align with the most novel regions of the state space. We provide bounds for the eigenvalues of the modified Laplacian; and we show that the smoothest eigenvectors with real eigenvalues below certain thresholds provide guaranteed gradients to novel states for both undirected and directed graphs. In an empirical evaluation in online, incremental settings, NEO outperformed related state-of-the-art approaches, including eigen-options and cover options, in a large collection of undirected and directed environments with varying connectivity structures.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce Novel Exploration via Orthogonality (NEO), a method to improve exploration in Reinforcement Learning (RL) through option discovery based on spectral graph properties. NEO builds on Laplacian-based approaches by incorporating sink-weights into the graph Laplacian, where sink weights are derived from state visitation counts to encode novelty. The resulting modified Laplacian yields eigenvectors whose gradients guide the agent toward novel regions of the state space. Unlike previous work, NEO extends naturally to both undirected and directed graphs, supported by theoretical guarantees regarding eigenvector properties and gradient flows. The method is evaluated in several RL environments, showing that NEO outperforms existing option discovery baselines, including eigenoptions and cover options.\n\nSTRENGTHS AND WEAKNESSES:\n1. Quality:\n    - Strengths:\n        - The paper presents a technically sound approach, building on solid spectral graph theory foundations.\n        - The paper is generally well structured and well written\n        - The theoretical contributions are well-developed, including $\\lambda$-bounds, gradient guarantees, and proofs of eigenvector properties.\n        - The empirical evaluation is extensive, covering eight domains, both directed and undirected.\n    - Weaknesses:\n        - While the theoretical results are compelling, the practical implications for large, continuous domains are not fully addressed.\n        - Most of the empirical evaluation focuses primarily on discrete, synthetic environments.\n2. Clarity:  \n    - Strengths:\n        - The formalism around the Source Laplacian and the construction of the NEO options is clear and mathematically well-structured.\n        - Algorithm 1 is helpful in understanding the practical implementation of NEO.\n    - Weaknesses:\n        - In Figure 3, inconsistent coloring for baselines makes comparisons, particularly between directed and undirected versions of the same domain, unnecessarily difficult.\n        - Theorem 3.3 suffers from overloaded notation where $i$ is the index of the eigenvector value $f_i$ with $N(i) > \\lambda$ and nodes along a constructed path  $i = v_0 \\rightarrow \\dots \\rightarrow v_k$, which can confuse readers\n        - Minor but noticeable formatting and typographical issues detract from polish, including:\n            - State action pair $(s,a)$ (l.27)\n            - Missing spacing for $\\gamma$ in (l. 95) \n            - max(f(.)) in (l.140)\n            - Buggy reference in (l.216)\n        - References to baseline models in Evaluation setup (l. 210 - 220) are missing which hinders reproducability \n3. Significance: \n    - Strengths:\n        - The empirical results suggest that NEO significantly improves learning efficiency, especially in directed and large-scale domains.\n        - Results seem impactful with NEO outperforming most other approaches in the evaluation\n        - Baselines in the evaluation seem well selected \n    - Weaknesses:\n        - The evaluation is limited to tabular, synthetic domains. Testing on more complex environments (e.g., Atari) would strengthen the practical impact.\n4. Originality: \n    - Strengths:\n        - The introduction of a modified Laplacian with visitation-based sink weights is a novel and meaningful conceptual contribution.\n        - Influences this work is based on are referenced appropriately.\n\nQUESTIONS:\n- The experiments are performed exclusively in deterministic settings, while the theoretical formulation of the MDP in the Foundations Section allows for stochastic transitions $\\mathcal{T}(s,a,s’)$. How robust is NEO to non-deterministic environments, where edge weights in the empirical graph reflect averaged, probabilistic transitions rather than deterministic ones?\n- Assumptions in Theorem 3.2: From my understanding, the proof relies on $\\lambda < \\max_i \\gamma_i$, which implies $\\gamma_i > \\lambda$ only under certain conditions, such as a uniform sink set. Additionally, to establish the maximality argument, the existence of at least one source node appears necessary. Could the authors clarify these assumptions explicitly, or correct any misunderstanding?\n- Have the authors considered applying NEO in deep RL environments, where state spaces are continuous and high-dimensional? Even conceptual discussion or preliminary experiments would be valuable to understand potential extensions.\n- The construction of sink weights involves a scaling function $F(.)$ with hyperparameters $\\delta$ and $k$. How sensitive is NEO’s performance to these choices? Can these be linked to the assumptions that eigenvalues $\\lambda$ exists where $N(i) > \\lambda$? \n- Computing eigenvectors for large graphs can be computationally expensive. Can the authors provide runtime comparisons or estimates of computational overhead relative to baselines?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents NEO, an exploration method that leverages the orthogonality of the graph Laplacian's smoothest eigenvectors to extract exploration options. Unlike prior methods which assume an undirected graph, NEO is made compatible with strongly connected directed graphs by limiting eigenvectors to those with a real eigenvalue below a certain novelty threshold. Compared to prior option papers, NEO's options explicitly drive the agent to more novel regions. Experiments on various undirected and directed domains demonstrate that NEO outperforms exploration baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The visualizations and toy experiments are compelling case studies that clearly demonstrate that the field of the smoothest eigenvectors corresponds to the more novel regions, illustrating how the method can be used for exploration.\n2. The paper provides solid theoretical support for the proposed method. With small insight on the eigenvalues and modification of the graph Laplacian, the method is extendable to directed graphs, making it more practical.\n\n\n\nNeutral\n\n1. While the writing overall is clear, it is hard to pinpoint what are the specific contributions of the paper. It would be beneficial to highlight them in the introduction. \n\n\n\nWeaknesses\n\n1. There is a lack of analysis for why the baselines perform poorly relative to NEO. For instance, \"Eigen\" gets very low performance despite being intuitively similar to NEO, as it also extracts options from eigenvectors of the graph Laplacian.\n2. The method is not compatible with continuous environments, limiting its usefulness. Furthermore, the requirement of having an approximation of the state transition graph presents a chicken-and-egg problem, where you need good exploration to approximate this graph and you need a good approximation of the graph to perform exploration via NEO. Older works [1] that obtain options from the successor representation for exploration already had some mechanism to bootstrap the approximation of this graph as well as some initial work on extensions to continuous domains.\n\n\n\nFormatting errors impacting readability:\n\n216: Citations for Shortest Path Novelty is broken.\n\n219: Missing dash for Cover options.\n\n221: Epoch is capitalized.\n\n226, 235: figure 2 is not capitalized.\n\n255: figure 3 is not capitalized.\n\n272: \"show\" should be \"shown\".\n\n283: \"cature\" should be \"captured.\" Space between \"atleast\".\n\nFigure 3: the symbols and colors for the methods changes across environments, making it hard to compare.\n\n\n[1] Tomar et al. Successor Options: An Option Discovery Framework for Reinforcement Learning. IJCAI 2019.\n\nQUESTIONS:\nQuestions\n\n1. \"Count-based exploration methods suffer from lag in exploration and eigenoptions can drive back to frequently visited states\" - does this materialize in experiments?\n2. How is the novelty-based threshold chosen? Does it vary across environments?\n3. Is there a more principled method for choosing the hyperparameters for the raw visitation count scaling function?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes a technique to improve exploration in reinforcement learning by using global signals from structures like the eigenspectrum of the graph to guide exploration. The key idea is to use the eigenvectors to construct options by following the gradient of the eigenvector to move in the state space. Starting with this intuition the proposed approach takes into account visitation counts to make the options more effective with respect to exploration. Extensive evaluation across multiple domains and other spectral baselines bear out this improvement in exploration performance.

STRENGTHS AND WEAKNESSES:
+ Conceptually the method is quite elegant and simple
+ Seems to work very well in practice
+ The paper does a good job of spelling out all the implementation nuances in sufficient detail
+ The paper has a strong set of theoretical results showing that the proposed approach will take us towards more novel states

Weaknesses

- It would be interesting to explore how the method might generalize to scenarios beyond tabular RL and high dimensional inputs, and scenarios where the state transition graph might not be known in advance and have to be constructed e.g. in a partially observable setting. Would be interesting if the paper discussed some of these practical issues that practitioners often care about in practice.

QUESTIONS:
- See weaknesses above.

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 3
DATE: 2025-07-08