Title: RL4SBDD: Reinforcement Learning for Preference Alignment in Structure-based Drug Design
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Structure-based Drug Design, Preference Alignment, Reinforcement Learning

Abstract: Generative models have been proven effective in designing novel ligands conditioned on protein target structures, which is a fundamental task in drug discovery. Recent research has explored aligning generative models with practical requirements for certain molecular properties through gradient guidance or direct preference optimization. In this paper, we introduce RL4SBDD, a reinforcement learning framework that unifies existing alignment methods and analyzes their limitations, including biased value estimation, inefficient data utilization, and off-policy distribution shift. Building on our framework, we propose RL4SBDD-M, a novel approach to address these problems and facilitate preference alignment in structure-based drug design. We introduce a policy model conditioned on rewards to avoid value estimations and fully leverage training data, and perform iterative reinforcement learning to mitigate off-policy distribution shift. Moreover, we introduce a sampling process with mixed guidance and noise reduction to improve generation quality and efficiency. On the CrossDocked2020 benchmark, our model outperforms the state-of-the-art by 6.8\% in success rate while achieving 4.6 times faster sampling. Furthermore, when equipped with oracle evaluations, our model could propose at least one desirable molecule for over 93\% of novel pockets within 50 generations, underscoring its potential in practical scenarios.

TL;DR: An RL framework for preference alignment in structure-based drug design with reward-conditioned policy, iterative reinforcement learning, and sampling with mixed guidance and reduced noise.