Title: Optimizing protein fitness using Bi-level Gibbs sampling with Graph-based Smoothing
Venue: NeurIPS 2023
Accepted: no
Keywords: protein design

discrete optimization

Gibbs sampling

protein engineering
Abstract: The ability to design novel proteins with higher fitness on a given task would be revolutionary for many fields of medicine. However, brute-force search through the combinatorially large space of sequences is infeasible. Prior methods constrain search to a small mutational radius from a reference sequence, but such heuristics drastically limit the design space. Our work seeks to remove the restriction on mutational distance while enabling efficient exploration. We propose $\textbf{Bi}$-level $\textbf{G}$ibbs sampling with $\textbf{G}$raph-based $\textbf{S}$moothing (BiGGS) which uses the gradients of a trained fitness predictor to sample many mutations towards higher fitness. Bi-level Gibbs first samples sequence locations then sequence edits. We introduce graph-based smoothing to remove noisy gradients that lead to false positives. Our method is state-of-the-art in discovering high-fitness proteins with up to 8 mutations from the training set.
We study the GFP and AAV design problems, ablations, and baselines to elucidate the results.

Abstract

The ability to design novel proteins with higher fitness on a given task would be revolutionary for many fields of medicine. However, brute-force search through the combinatorially large space of sequences is infeasible. Prior methods constrain search to a small mutational radius from a reference sequence, but such heuristics drastically limit the design space. Our work seeks to remove the restriction on mutational distance while enabling efficient exploration. We propose Bi -level G ibbs sampling with G raph-based S moothing (BiGGS), which uses the gradients of a trained fitness predictor to sample many mutations towards higher fitness. Bi-level Gibbs first samples sequence locations then sequence edits. We introduce graph-based smoothing to remove noisy gradients that lead to false positives. Our method is state-of-the-art in discovering high-fitness proteins with up to 8 mutations from the training set. We study the GFP and AAV design problems, ablations, and baselines to elucidate the results.

1 Introduction

In protein design, fitness is loosely defined as performance on a desired property or function. Examples of fitness include catalytic activity for enzymes [1, 20] and fluorescence for biomarkers [27]. Protein engineering seeks to design proteins with high fitness by altering the underlying sequences of amino acids. However, the number of possible proteins increases exponentially with sequence length, rendering it infeasible to perform brute-force search to engineer novel functions which often requires many mutations (i.e. at least 3 [11]). Directed evolution [3] has been successful in improving protein fitness, but it requires substantial labor and time to gradually explore many mutations.

We aim to find shortcuts to generate high-fitness proteins that are many mutations away from what is 22 known but face several challenges. Proteins are notorious for highly non-smooth fitness landscapes: 1 23 fitness can change dramatically with just a single mutation, and most protein sequences have zero 24 fitness [29]. As a result, machine learning (ML) methods are susceptible to learning noisy fitness 25 landscapes with false positives [18] and local optimums [6] which poses problems to optimization 26 and search. The 3D protein structure, if available, can help provide helpful constraints in navigating 27 the noisy fitness landscape, but it cannot be assumed in the majority of cases - current protein folding 28 methods typically cannot predict the effects on structure of point mutations [25]. 29

Our work proposes a sequence-based method that can optimize over a noisy fitness landscape and 30

efficiently sample large mutational edits. We introduce two methodological advances summarized in 31

Figure 1. The first is Graph-based Smoothing (GS) that regularizes the noisy landscape. We consider 32

it as a noisy graph signal and apply L 1 graph Laplacian regularization. This encourages sparsity and 33

local consistency in the landscape; most protein sequences have zero fitness, and similar sequences 34

BiGGS training

Goal

Graph-based

Train on

Smoothing

startingdata

fe(c)

Startingdata

Starting data

A.Noisyfitnesslandscape

BiGGS sampling

End

Step1:sampleresiduelocations

Vfe(α)

D

Current

Compute

Step2:samplemutationsatlocations

fitness

gradients

Start

D.IterativeExtrapolationwithBi-level Gibbssampling

E.Mutation trajectory

have similar fitness [42]. The effect is a smooth fitness landscape learned by the ML model on which 35 gradients accurately approximate the direction towards high-fitness sequences. To reach high-fitness 36 sequences requiring many mutations, we use the improved gradients in our second advancement, 37 Bi-level Gibbs (BiG), to approximate the proposal distribution in a Gibbs sampling procedure - as 38 inspired by Gibbs with Gradients (GWG) [12]. BiG uses bi-level sampling to propose up to 5 indices 39 to mutate simultaneously. Local improvements from the gradients help select beneficial mutations to 40 guide low-fitness sequences towards higher fitness while sampling allows exploration. Following the 41 intuition of directed evolution, we apply multiple rounds of sampling over clustered sequences in a 42 procedure we call Iterative Extrapolation (IE). 43

We find BiG and GS are complementary in enabling long-range exploration while avoiding the pitfalls 44 of a noisy fitness landscape; the combination of both is referred to as BiGGS. We introduce a set 45 of tasks using the Green Fluorescent Proteins (GFP) dataset [30] to simulate challenging protein 46 design scenarios by starting with low-fitness sequences that require many (5 or more) mutations to 47 the best fitness. We primarily study GFP because of (1) its difficulty as one of the longest proteins in 48 fitness datasets and (2) its comprehensive fitness measurements of up to 15 mutations. To assess the 49 generalizability of our method, we additionally study the Adeno-Associated Virus (AA V) dataset [7] 50 based on gene delivery fitness. We evaluate BiGGS and prior works on our proposed benchmarks 51 to show that BiGGS is state-of-the-art in GFP and AAV fitness optimization. Our contributions are 52 summarized as follows: 53

54

55

56

We develop a novel sequence-based protein fitness optimization algorithm, BiGGS, based on BiG to efficiently sample multiple mutations, GS to regularize the fitness landscape, and IE to progressively mutate towards higher-fitness (Section 2).

57

58

59

60

61

62

We study GFP by proposing a set of design benchmarks of different difficulty with varying starting sequence distribution (Section 3). While our focus is GFP, we develop benchmarks on AAV to evaluate a new fitness criteria (Appendix C).

Weshow BiGGS is state-of-the-art in GFP and AAV fitness optimization while exhibiting diversity and novelty from the training set. We analyze the contributions of BiG and GS towards successful fitness optimization over challenging fitness landscapes (Section 5).

2 Method 63

64

65

66

67

We begin with the problem formulation in Section 2.1. Our method uses two components bi-level Gibbs sampling (Section 2.2) and graph-based smoothing (Section 2.3). Together they are part of a iterative sampling method called iterative extrapolation (Section 2.4) as a way to progressively extrapolate towards novel sequences. The full algorithm, BiGGS, is presented in Algorithm 1.

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

2.1 Problem formulation

Let the starting set of length L protein sequences and their fitness measurements be denoted as D 0 = ( X 0 , Y 0 ) where X 0 ⊂ V L with vocabulary V = { 1 , . . . , 20 } and Y 0 ⊂ R . We use subscripts to distinguish sequences, x i ∈ V L , while a paranthetical subcript denotes the token, ( x i ) j ∈ V where j ∈ { 1 , . . . , L } . Note our method can readily be extended to other modalities, e.g. nucleic acids.

For in-silico evaluation, we denote the set of all known sequences and fitness measurements as D ∗ = ( X ∗ , Y ∗ ) . We assume there exists a black-box function g : V L → R such that g ( x ∗ ) = y ∗ , which is approximated by an oracle g φ . In practice, the oracle is a model trained with weights φ to minimize prediction error on D ∗ . The starting dataset only includes low fitness sequences and is a strict subset of the oracle dataset D 0 ⊂ D ∗ to simulate fitness optimization scenarios. Given D 0 , our task is to generate a set of sequences ˆ X = { ˆ x i } N samples i =1 with higher fitness than the starting set.

2.2 BiG: Bi-level Gibbs (with Gradients)

To generate new sequences, we propose a modified version of Gibbs With Gradients (GWG) [12]. The first step is to train a fitness predictor , f θ : V L → R , using D 0 to act as the learned unnormalized probability (i.e. negative energy) from sequence to fitness. We use the Mean-Squared Error (MSE) loss to train the predictor which we parameterize as a deep neural network. We found it beneficial to employ negative data augmentation since both the dataset and the range of fitness values are small. Specifically, we double the size of the dataset by sampling random sequences, x neg i ∼ Uniform ( V L ) , and assigning them the lowest possible fitness value, µ .

Our goal is to sample from log p ( x ) = f θ ( x ) -log Z where Z is the normalization constant. Higher fitness sequences will be more likely under this distribution while sampling over many mutations will induce diversity and novelty. GWG uses Gibbs sampling with locally informed proposals :

where d θ ( x ) ij is a first order Taylor approximation of the log-likelihood ratio of mutating the i th index of x to token j . Treating x, x ′ as one-hot, ( x ′ ) /latticetop d θ ( x ) = ∑ i ( x ′ i ) /latticetop d θ ( x ) i is the sum over the local differences where x ′ differs from x . The proposal q ( x ′ | x ) can be efficiently computed when H ( · ) is the 1-Hamming ball 2 : a single backward pass is needed to compute the Jacobian in eq. (1).

Sampling M > 1 mutations in the same fashion would require estimating the gradients for each 94 mutation individually resulting in exponentially more computations. Instead, we find a simple bi-level 95 sampling scheme to be effective. The first level samples mutation indices, /lscript m , with a categorical 96 tempered-softmax distribution over the column-wise maxima, d θ ( x ) i = max j ∈ { 1 ,...,L } d θ ( x ) ij . The 97 second level samples token-wise mutations ( x ′ ) /lscript m over the vocabulary the same way as the first level 98 using d θ ( x ) /lscript m j . 99

where τ is a temperature hyperparameter. Indices are sampled iid which means the same index may 100 get sampled twice. An improvement left for future work is to model conditional dependencies across 101

locations. Each proposed sequence is accepted or rejected using Metropolis-Hasting (MH) 102

103

104

105

106

To summarize, our method Bi-level Gibbs (BiG) first samples N prop sequences each with up to M mutations from eq. (2) then returns a set of accepted sequences, X ′ , according to eq. (3). Forcing BiG to make M mutations may make it skip sequences that are less than M mutations away. We found it best to run BiG over all values leading up to M . The full algorithm is provided in algorithm 2.

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

A concern is the accuracy of the 1st order Taylor approximation, d θ ( x ) ij , for M > 1 . We observed the performance of BiG is highly dependent on the performance of the predictor for gradients that correlate with higher fitness. The next two sections focus on the development of a robust predictor (Section 2.3) and an iterative framework to improve the Gibbs sampling approximation (Section 2.4).

2.3 GS: Graph-based smoothing

The efficacy of the gradients in BiG to guide sampling towards high fitness sequences depends on the smoothness of the mapping from sequence to fitness learned by the predictor. Unfortunately, the high-dimensional sequence space coupled with few data points and noisy labels results in a noisy predictor that is prone to sampling false positives [18] or getting stuck in local optima [6]. To address this, we use techniques from graph signal processing to smooth the learned mapping by promoting similar sequences to have similar fitness [42] while penalizing noisy predictions [17].

Suppose we have trained a noisy predictor with weights θ 0 on the initial dataset D 0 . To construct our graph G = ( V, E ) , we first construct the nodes V by iteratively applying pointwise mutations to each sequence in the initial set X 0 to simulate a local landscape around each sequence. We call this routine Perturb with a hyperparameter N perturb for the number of perturbations per sequence (see Algorithm 5). The edges, E , are a nearest neighbor graph with N neigh neighbors where edge weights are inversely proportional to their sequence distance, ω ij = ω (( v i , v j )) = 1 / dist ( v i , v j ) ; edge weights are stored in a similarity matrix W = { ω ij ∀ v i , v j ∈ V } .

The normalized Laplacian matrix of G is L = I -D -1 / 2 WD -1 / 2 where I is the identity and D is a diagonal matrix with i -th diagonal element D ii = ∑ j ω ij . An eigendecomposition of L gives L = U Σ U T where Σ is a diagonal matrix with sorted eigenvalues along the diagonal and U is a matrix of corresponding eigenvectors along the columns. An equivalent eigendecomposition with symmetric matrix B (and edges E arranged into an adjacency matrix) is

Next, we formulate smoothing as an optimization problem. For each node, we predict its fitness S = { f θ 0 ( v ) ∀ v ∈ V } , also called the graph signal , which we assume to have noisy values. Our goal is to solve the following where S is arranged as a vector and S ∗ is the smoothed signal,

Equation (4) is a form of graph Laplacian regularization that has been studied for image segmentation with weak labels [17]. B has eigenvalue weighted eigenvectors as rows. Due to the L 1 -norm ‖ B ˆ S ‖ 1 is small if ˆ S is primarily aligned with slowly varying eigenvectors whose eigenvalues are small. This term penalizes large jumps in fitness between neighboring nodes hence we call it smoothness sparsity constraint . The second term, ‖ ˆ S -S ‖ 1 , is the signal sparsity constraint that remove noisy predictions with hyperparameter γ . The L 1 -norm is applied to reflect that most sequences have zero fitness.

At a high level, eq. (4) is solved by introducing auxiliary variables which allows for an approximate solution by solving multiple LASSO regularization problems [34]. Technical details and algorithm are described in Appendix B. Once we have S ∗ , we retrain our predictor with the smoothed dataset D = ( V, S ∗ ) on which the learned predictor is smoother with gradients much more amenable for gradient-based sampling, BiG. We refer to our smoothing algorithm as Graph-based Smoothing (GS).

2.4 IE: Iterative Extrapolation

The 1st order Taylor approximation of eq. (1) deteriorates the more we mutate from the parent sequence. Inspired by directed evolution [3], we propose to alleviate this by performing multiple

rounds of sampling where successive rounds use sequences from the previous round. Each round 147 re-centers the Taylor approximation and extrapolates from the previous round. We first train a 148 predictor f θ using GS (Section 2.3). Prior to sampling, we observe the number of sequences may be 149 large and redundant. To reduce the number of sequences, we perform hierarchical clustering [22] and 150 take the sequence of each cluster with the highest fitness using f θ . Let C be the number of clusters. 151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

Each round r reduces the sequences from the previous round and performs BiG sampling.

One cycle of clustering, reducing, and sampling is a round of extrapolation,

where the initial round r = 0 starts with X ′ 0 = X 0 . After R rounds, we select our candidate sequences by taking the TopN samples sequences based on ranking with f θ . We call this procedure Iterative Extrapolation (IE). While IE is related to previous directed evolution methods [31], it differs by taking larger mutational edits on each round with BiG and encouraging diversity by mutating the best sequence of each cluster. The full candidate generation, Bi-level Gibbs with Graph-based Smoothing (BiGGS), with IE is presented in Algorithm 1.

.

X

3 Benchmarks

We use the Green Fluoresent Protein (GFP) dataset from Sarkisyan et al. [30] containing over 56,806 log fluorescent fitness measurements, with 51,715 unique amino-acid sequences due to sequences having multiple measurements . We quantify the difficulty of a protein fitness optimization task by introducing the concept of a mutational gap , which we define as the minimum Levenshtein distance between any sequence in the training set to any sequence in the 99th percentile:

A mutational gap of 0 means that the training set, D 0 may contain sequences that are in the 99th percentile of fitness. Solving such tasks is easy because methods may sample high-fitness sequences from the training set. Prior work commonly uses the GFP task introduced by design-bench (DB) evaluation framework [36] which has a mutational gap of 0 (see Appendix A). To compare to previous work, we include the DB task as "easy" difficulty in our experiments, but we introduce "medium" and "hard" optimization tasks which have lower starting fitness ranges in the 20-40th and 10-30th percentile of known fitness measurements alongside much higher mutational gaps. Our proposed difficulties are summarized in Table 1 and visualized in Figure 5.

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

The oracle in design-bench (DB) uses a Transformerbased architecture from Rao et al. [26]. When using this oracle, we noticed a concerning degree of false positives and a thresholding effect of its predictions. We propose a simpler CNN architecture as the oracle that achieves superior performance in terms of Spearman correlation and fewer false positives as seen in Figure 6. Our CNN

consists of a 1D convolutional layer that takes in a one-hot encoded sequence, followed by maxpooling and a dense layer to a single node that outputs a scalar value. It uses 256 channels throughout for a total of 157,000 parameters - 15 fold fewer than DB oracle.

Our experiments in Section 5 benchmark on GFP easy, medium, and hard with our CNN oracle. In Appendix C we summarize an additional benchmark using Adeno-Associated Virus (AAV) dataset [7] which focuses on optimizing a 28-amino acid segment for DNA delivery. We use the same task set-up and train our CNN oracle on AAV.

4 Related work

Optimization in protein design. Approaches in protein design can broadly be categorized in using sequence, structure or both [9]. Advances in structure-based protein design have been driven by a combination of geometric deep learning and generative models [37, 13, 39, 8]. Sequencebased protein design has been explored through the lens of reinforcement learning [2, 16], latent space optimization [32, 16, 19], GFlowNets [14], bayesian optimization [38], generative models [6, 5, 23, 21], and model-based directed evolution [31, 4, 24, 28, 35]. Together they face the common issue of a noisy landscape to optimize over. Moreover, fitness labels are problem-dependent and scarce, apart from well-studied proteins [5]. Our method addresses small amounts of starting data and noisy landscape by regularization with GS. We focus on sequence-based methods where we use locally informed Markov Chain Monte Carlo (MCMC) methods [40] method based on Gibbs With Gradients (GWG) [12] which requires a smooth energy function for strong performance guarantees. Concurrently, Emami et al. [10] used GWG to sample higher fitness sequences by optimizing over a product of experts distribution, a mixture of a protein language model and a fitness predictor. However, they eschewed the need for a smooth energy function which we address with GS.

Discrete MCMC. High-dimensional discrete MCMC can be inefficient with slow mixing times. GWG showed discrete MCMC becomes practical by utilizing learned gradients in the sampling distribution, but GWG in its published form was limited to sampling in a proposal window of size 1. Zhang et al. [41] proposed to modify GWG with langevin dynamics to allow for the whole sequence to mutate on every step while Sun et al. [33] augmented GWG with a path auxiliary proposal distribution to propose a series of local moves before accepting or rejecting. We find that BiGGS with bi-level sampling is simpler and effective in achieving a proposal window size beyond 1.

5 Experiments

We study the performance of BiGGS on the GFP tasks from Section 3. Furthermore, to ensure that we did not over-optimize to the GFP dataset, we benchmark BiGGS using AAV benchmark in Appendix C. In the subsequent sections, we outline our experiments on GFP, while corresponding results for AAV are in Appendix C. Section 5.1 compares the performance of BiGGS on GFP to a representative set of baselines while Section 5.2 performs ablations on components of BiGGS. Finally, Section 5.3 analyzes BiGGS's performance.

BiGGS training and sampling. Following section 3, we use the oracle CNN architecture for our predictor (but trained on different data). To ensure a fair comparison, we use the same predictor across all model-based baselines. We use the following hyperparameters as input to Algorithm 1 across all tasks: N prop = 100 , τ = 0 . 01 , M = 5 , N neigh = 500 , N perturb = 1000 N samples = 128 R = 3 , C = 500 . We were unable to perform extensive exploration of hyperparameters. Reducing the number of hyperparameters and finding optimal values is an important future direction. Training is performed with batch size 1024, ADAM optimizer [15] (with β 1 = 0 . 9 , β 2 = 0 . 999 ), learning rate 0.0001, and 1000 epochs using a single A6000 Nvidia GPU. Initial predictor training takes 10 minutes while graph-based smoothing takes around 30 minutes depending on convergence of the

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

numerical solvers. Training with the smoothed data takes 4 to 8 hours. Sampling takes under 30 minutes and can be parallelized.

Baselines. We choose a representative set of prior works with publicly available code: GFlowNets (GFN-AL) [14], model-based adaptive sampling (CbAS) [6], greedy search (AdaLead) [31], bayesian optimization with quasi-expected improvement acquisition function (BO-qei) [38], conservative model-based optimization (CoMs) [35], and proximal exploration (PEX) [28].

Metrics. Each method generates N samples = 128 samples ˆ X = { ˆ x i } N samples i =1 to evaluate. Here, dist is the Levenshtein distance. We report three metrics:

(Normalized) Fitness = median ( { ξ (ˆ x i ; Y ∗ ) } N samples i =1 ) where ξ (ˆ x ; Y ∗ ) = g φ (ˆ x i ) -min ( Y ∗ ) max ( Y ∗ ) -min ( Y ∗ ) is the min-max normalized fitness.

Diversity = mean ( { dist ( x, ˜ x ) : x, ˜ x ∈ ˆ X , x = ˜ x } ) is the average sample similarity.

/negationslash

Novelty = median ( { η (ˆ x i ; X 0 ) } N samples i =1 ) where η ( x ; X 0 ) = min ( { dist ( x, ˜ x ) : ˜ x ∈ X ∗ , ˜ x = x } ) is the minimum distance of sample x to any of the starting sequences X 0 .

We use median for outlier robustness. Diversity and novelty were introduced in Jain et al. [14]. We emphasize that higher diversity and novelty is not equivalent to better performance. For instance, a random algorithm would achieve maximum diversity and novelty.

5.1 Results

All methods are evaluated on 128 generated candidates, as done in design-bench. We run 5 seeds and report the average metric across all seeds including the standard deviation in parentheses. Results using our GFP oracle are summarized in table 2. Results using the DB oracle are in appendix C.

BiGGS substantially outperforms other baselines on the medium and hard difficulties, consistently navigating the mutational to achieve high fitness, while maintaining diversity and novelty from the training set. The unique extrapolation capabilities of BiGGS on the hardest difficulty level warranted additional analysis, and we investigate this further in Section 5.3. Adalead overall performed secondbest, matching the performance of BiGGS on the easy difficulty with PEX only slightly worse. Notably, both Adalead and PEX suffer from a low novelty in the medium and hard settings.

Regarding the other baselines, GFN-AL exhibits subpar performance across all difficulty levels. We were unable to reproduce their published results. 3 Its performance notably deteriorates on medium and hard difficulty levels, a trend common amongst all baselines. CbAS explores very far, making on average 46 mutations, resulting in poor fitness. BO-qei is unable to extrapolate beyond the training set, and CoMs presents instability, as indicated by their high standard deviations, and collapse. 4

We further analyze the distribution of novelty and fitness among CbAS, Adalead, and our method, BiGGS, in Figure 2. Adalead tends to be conservative, while CbAS is excessively liberal. BiGGS, on

/negationslash

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

Easy

Easy

Easy

AdaLead

BiGGS

CbAs

0.8

0.4

0

Medium

Medium

Medium

AdaLead

BiGGS

CbAS

Fitness

0.8

0

Hard

Hard

Hard

AdaLead

BiGGS

CbAS

0.8

0.4

0

012345678910012345678910012345678910

Novelty

the other hand, manages to find the middle ground, displaying high fitness in its samples while also effectively exploring across the mutational gap at each difficulty level.

5.2 Ablations

We perform ablations on each component of BiGGS on the hard difficulty task. In the first ablation, we replace BiG with GWG but use an equivalent number of samples by running R = 15 of IE for a fair comparison. The second ablation removes GS and starts sampling after initial predictor training. The last ablation removes iterative extrapolation by setting R = 1 , M = 15 , N sample = 300 which maintains the number of samples but without iterative rounds. Our results are shown in Table 3. We

see GS is crucial for BiGGS on the hard difficulty level. Additional analysis is provided in section 5.3. Removing IE also results in a large decrease in performance. Unsurprisingly, GWG greatly benefits from GS and IE due to its similarity with BiG. However, using BiG results in improved fitness. We conclude each component of BiGGS contributes to its performance.

5.3 Analysis

We analyze BiGGS in the hard GFP task and demonstrate that (1) GS results in gradients from BiG that point towards higher fitness sequences and (2) BiG's ability to sample large mutations ( M ≥ 3) enables efficient traversal of large mutational distances in a high dimensional space.

Figure 3A, B shows how GS leads to a smooth fitness landscape, enabling BiG to sample high-fitness mutations. Often but not always, GS allows BiGGS to assign high probability to higher-fitness mutations that are low-probability without GS. We use the GFP wildtype (WT) as a representative of high-fitness sequences in the 99th percentile. The smoothed f θ (fig. 3A) assigns high probability to the mutation that changes the current residue to the WT residue at a given proposed position, while

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

A

B

SmoothedPredictor

Non-SmoothedPredictor

BiGGSAcceptedMutations

6

6

Non-WT Residues

Non-WTResidues

700

M=1 Mutations

5

WTSequence

5

WTResidue

600

M=4Mutations

4

4

500

Density

Density

Count

400

3

3

300

2

2

200

1

1

100

0

0

0

0.0

ProposedMutationProbability

0.2

0.4

0.6

0.8

1.0

0.0

ProposedMutationProbability

0.2

0.4

0.6

0.8

1.0

2

-1

OracleDifference

0

1

2

3

giving low probability to other (lower fitness) mutations. The non-smoothed predictor proposes to mutate the current residue to the WT residue only slightly more often than other mutations (fig. 3B).

In Figure 3C, we show that BiGGS's ability to consider large mutations ( M ≥ 3) facilitates efficient exploration. We use the oracle to analyze all single ( M = 1) and quadruple ( M = 4) mutations accepted during the course of running BiGGS. We choose M = 4 as it represents the largest portion of BiGGS-accepted mutations among large mutations ( M ≥ 3) . According to the oracle, the largest quadruple mutation fitness increases are bigger than the largest single mutation fitness increases. Quadruple mutations also result in a greater number of substantial fitness increases. We note a somewhat larger count of substantially negative mutations for quadruple mutations vs. for single mutations. This is expected given BiGGS's stochasticity, and the tendency of large mutations to be more deleterious than small ones. Similar analysis for M up to 5 is in Appendix D.

6 Discussion

In this work, we presented BiGGS, a method for optimizing protein fitness by incorporating ideas from MCMC, graph Laplacian regularization, and directed evolution. We outlined a new benchmark on GFP that introduces the challenge of starting with poor-fitness sequences, many edits from the top fitness sequences. BiGGS discovered higher fitness sequences than in the starting set, even in the hard difficulty of our benchmark where prior methods struggled. We analyzed the two methodological advancements, Graph-based Smoothing (GS) and Bi-level Gibbs (BiG) (which includes Iterative Extrapolation), as well as ablations to conclude each of these techniques aided BiGGS's performance.

There are multiple extensions of BiGGS. The first is to improve BiG by removing the independence assumption across residues and instead modeling joint probabilities of epistatic interactions. One possibility for learning epistatic interactions is to incorporate 3D structure information (if available) to bias the sampling distribution. Secondly, the effectiveness of GS in our ablations warrants additional exploration into better regularization techniques for protein fitness predictors. Our formulation of GS is slow due to the nearest neighbor graph construction and its L 1 optimization. Lastly, investigating BiGGS to handle variable length sequences, multiple objectives, and multiple rounds of optimization is of high importance towards real protein engineering problems. Our code is included in the supplementary data and will be publicly available upon acceptance.

References

Dave W Anderson, Florian Baier, Gloria Yang, and Nobuhiko Tokuriki. The adaptive landscape of a metallo-enzyme is shaped by environment-dependent epistasis. Nature Communications , 12(1):3867, 2021.

Christof Angermueller, David Dohan, David Belanger, Ramya Deshpande, Kevin Murphy, and Lucy Colwell. Model-based reinforcement learning for biological sequence design. 2020.

Frances H Arnold. Design by directed evolution. Accounts of chemical research , 31(3):125-131, 314 1998. 315

Frances H Arnold. Directed evolution: bringing new chemistry to life. Angewandte Chemie 316 International Edition , 57(16):4143-4148, 2018. 317

Surojit Biswas, Grigory Khimulya, Ethan C Alley, Kevin M Esvelt, and George M Church. 318 Low-n protein engineering with data-efficient deep learning. Nature methods , 18(4):389-396, 319 2021. 320

David Brookes, Hahnbeom Park, and Jennifer Listgarten. Conditioning by adaptive sampling 321 for robust design. In International conference on machine learning , pages 773-782. PMLR, 322 2019. 323

324

325

326

Drew H Bryant, Ali Bashir, Sam Sinai, Nina K Jain, Pierce J Ogden, Patrick F Riley, George M Church, Lucy J Colwell, and Eric D Kelsic. Deep diversification of an aav capsid protein by machine learning. Nature Biotechnology , 39(6):691-696, 2021.

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-based protein sequence design using proteinmpnn. Science , 378(6615):49-56, 2022.

Wenze Ding, Kenta Nakai, and Haipeng Gong. Protein design via deep learning. Briefings in bioinformatics , 23(3):bbac102, 2022.

Patrick Emami, Aidan Perreault, Jeffrey Law, David Biagioni, and Peter St John. Plug & play directed evolution of proteins with gradient-based discrete mcmc. Machine Learning: Science and Technology , 4(2):025014, 2023.

Mahan Ghafari and Daniel B. Weissman. The expected time to cross extended fitness plateaus. Theoretical Population Biology , 129:54-67, 2019. ISSN 0040-5809. doi: https://doi.org/ 10.1016/j.tpb.2019.03.008. URL https://www.sciencedirect.com/science/article/ pii/S0040580918301011 . Special issue in honor of Marcus Feldman's 75th birthday.

Will Grathwohl, Kevin Swersky, Milad Hashemi, David Duvenaud, and Chris Maddison. Oops i took a gradient: Scalable sampling for discrete distributions. In International Conference on Machine Learning , pages 3831-3841. PMLR, 2021.

John Ingraham, Max Baranov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujie Wang, Vincent Xue, Fritz Obermeyer, Andrew Beam, et al. Illuminating protein space with a programmable generative model. bioRxiv , pages 2022-12, 2022.

Moksh Jain, Emmanuel Bengio, Alex Hernandez-Garcia, Jarrid Rector-Brooks, Bonaventure F. P. Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, Lena Simine, Payel Das, and Yoshua Bengio. Biological sequence design with GFlowNets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 9786-9801. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/jain22a.html .

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Minji Lee, Luiz Felipe Vecchietti, Hyunkyu Jung, Hyunjoo Ro, Meeyoung Cha, and Ho Min Kim. Protein sequence design in a latent space via model-based reinforcement learning.

Zhiwu Lu, Zhenyong Fu, Tao Xiang, Peng Han, Liwei Wang, and Xin Gao. Learning from weak and noisy labels for semantic segmentation. IEEE transactions on pattern analysis and machine intelligence , 39(3):486-500, 2016.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 359 Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 360 2017. 361

Natalie Maus, Haydn Jones, Juston Moore, Matt J Kusner, John Bradshaw, and Jacob Gardner. 362 Local latent space bayesian optimization over structured inputs. Advances in Neural Information 363 Processing Systems , 35:34505-34518, 2022. 364

365

366

Stanislav Mazurenko, Zbynek Prokop, and Jiri Damborsky. Machine learning in enzyme engineering. ACS Catalysis , 10(2):1210-1223, 2019.

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems , 34:29287-29303, 2021.

Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint arXiv:1109.2378 , 2011.

Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transformers and inference-time retrieval. In International Conference on Machine Learning , pages 16990-17017. PMLR, 2022.

Vishakh Padmakumar, Richard Yuanzhe Pang, He He, and Ankur P Parikh. Extrapolative controlled sequence generation via iterative refinement. arXiv preprint arXiv:2303.04562 , 2023.

Marina A Pak, Karina A Markhieva, Mariia S Novikova, Dmitry S Petrov, Ilya S Vorobyev, Ekaterina S Maksimova, Fyodor A Kondrashov, and Dmitry N Ivankov. Using alphafold to predict the impact of single mutations on protein stability and function. Plos one , 18(3): e0282689, 2023.

Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S Song. Evaluating protein transfer learning with tape. In Advances in Neural Information Processing Systems , 2019.

S James Remington. Green fluorescent protein: a perspective. Protein Science , 20(9):15091519, 2011.

Zhizhou Ren, Jiahan Li, Fan Ding, Yuan Zhou, Jianzhu Ma, and Jian Peng. Proximal exploration for model-guided protein sequence design. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 18520-18536. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/ v162/ren22a.html .

Philip A Romero and Frances H Arnold. Exploring protein fitness landscapes by directed evolution. Nature reviews Molecular cell biology , 10(12):866-876, 2009.

Karen S Sarkisyan, Dmitry A Bolotin, Margarita V Meer, Dinara R Usmanova, Alexander S Mishin, George V Sharonov, Dmitry N Ivankov, Nina G Bozhanova, Mikhail S Baranov, Onuralp Soylemez, et al. Local fitness landscape of the green fluorescent protein. Nature , 533 (7603):397-401, 2016.

Sam Sinai, Richard Wang, Alexander Whatley, Stewart Slocum, Elina Locane, and Eric D Kelsic. Adalead: A simple and robust adaptive greedy search algorithm for sequence design. arXiv preprint arXiv:2010.02141 , 2020.

Samuel Stanton, Wesley Maddox, Nate Gruver, Phillip Maffettone, Emily Delaney, Peyton Greenside, and Andrew Gordon Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. arXiv preprint arXiv:2203.12742 , 2022.

Haoran Sun, Hanjun Dai, Wei Xia, and Arun Ramamurthy. Path auxiliary proposal for mcmc in discrete space. In International Conference on Learning Representations , 2022.

Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological) , 58(1):267-288, 1996.

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

Brandon Trabucco, Aviral Kumar, Xinyang Geng, and Sergey Levine. Conservative objective models for effective offline model-based optimization. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 10358-10368. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/trabucco21a.html .

Brandon Trabucco, Xinyang Geng, Aviral Kumar, and Sergey Levine. Design-bench: Benchmarks for data-driven offline model-based optimization. CoRR , abs/2202.08450, 2022. URL https://arxiv.org/abs/2202.08450 .

Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiv , 2022.

James T Wilson, Riccardo Moriconi, Frank Hutter, and Marc Peter Deisenroth. The reparameterization trick for acquisition functions. arXiv preprint arXiv:1712.00424 , 2017.

Jason Yim, Brian L Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277 , 2023.

Giacomo Zanella. Informed proposals for local mcmc in discrete spaces. Journal of the American Statistical Association , 115(530):852-865, 2020.

Ruqi Zhang, Xingchao Liu, and Qiang Liu. A langevin-like sampler for discrete distributions. International Conference on Machine Learning , 2022.

Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf. 433 Learning with local and global consistency. Advances in neural information processing systems , 434 16, 2003. 435

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a sampling method for directed protein evolution. The method includes three parts: Bi-level Gibbs sampling (BiG), graph-based smoothing (GS), and iterative extrapolation (IE). BiG improves sampling efficiency for multi-point mutant sequences, GS helps to sample toward high fitness sequences, and IE reduces the Taylor approximation error. Experiments results show that the proposed method could achieve competitive performance.\n\nSTRENGTHS:\n- The presentation is clear and the idea is reasonable. The figures are beautiful and The authors clearly formalize the problem and key operations.\n- The addressed problem is important. The relevant methods may be applied in different protein design scenarios.\n- The experiments are convincing. I appreciate the authors' efforts in revealing the effect of each module via ablation studies.\n\nWEAKNESSES:\n- The proposed approach is somewhat heuristic and lacks theoretical assurance.\n- There is a lack of relevant prior research on protein design [1-7].\n- The proposed method does not compare with strong baselines such as the diffusion-based method [6,7]. Therefore, it is challenging for reviewers to verify its effectiveness and potential.\n\n[1] Gao, Zhangyang, Cheng Tan, and Stan Z. Li. \"Alphadesign: A graph protein design method and benchmark on alphafolddb.\" arXiv preprint arXiv:2202.01079 (2022).\n\n[2] Hsu, Chloe, et al. \"Learning inverse folding from millions of predicted structures.\" International Conference on Machine Learning. PMLR, 2022.\n\n[2] Gao, Zhangyang, Cheng Tan, and Stan Z. Li. \"PiFold: Toward effective and efficient protein inverse folding.\" The Eleventh International Conference on Learning Representations. 2022.\n\n[3] Ingraham, John, et al. \"Generative models for graph-based protein design.\" Advances in neural information processing systems 32 (2019).\n\n[4] Tan, Cheng, et al. \"Global-Context Aware Generative Protein Design.\" ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023.\n\n[5] Jing, Bowen, et al. \"Learning from protein structure with geometric vector perceptrons.\" arXiv preprint arXiv:2009.01411 (2020).\n\n[6] Stanton, Samuel, et al. \"Accelerating bayesian optimization for biological sequence design with denoising autoencoders.\" International Conference on Machine Learning. PMLR, 2022.\n\n[7] Gruver, Nate, et al. \"Protein Design with Guided Discrete Diffusion.\" arXiv preprint arXiv:2305.20009 (2023).\n\nQUESTIONS:\n- Q1: How do you define $\\text{dist}(v_i, v_j)$ when constructing the graph? Does the distance function consider the fitness score?\n\n- Q2: When constructing the graph, are the sequences mutated randomly, or is there some guiding knowledge used to direct the mutations?\n\n- Q3: Could you compare the proposed method with deep learning ones? Is it because there is a lack of work related to deep learning in this area?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work applies gibbs sampling methods for generating random sequences of proteins to find mutants with high fitness. The model is compared against a set of sampling methods and the results are evaluated on GFP by fitness, diversity, and novelty.

STRENGTHS:
- The problem formulation and model construction are presented clearly. 
- The experiments were evaluated comprehensively from different perspectives.

WEAKNESSES:
- The motivation for random sampling the whole sequence for a protein in the context of is unclear. 
- As AAV has been mentioned more than once in Introduction, it is preferred to put them in the main text. 
- While the proposed solution to the problem is a sampling method, other baseline methods that have been used to solve the same problem might also be compared, such as the ESM-series.

QUESTIONS:
- (line 37) "to reach high-fitness sequences requiring many mutations..." is not rigorous. It is not true that modifying more sites will always return a better mutant. 
- (line 49) GFP (in [30]) only has 238 amino acids, which is not "one of the longest proteins".
- (line 86) “Sampling random sequences and assigning them the lowest possible fitness value” might be risky, as it is hard to tell if a random sequence will actually lose its functionality. Also, at least for the GFP dataset, there is already a large portion of negative samples (i.e., mutants that perform worse than the wild-type), why it is essential to add more negative samples?
- (line 240) If diversity and novelty do not have a preferred direction or optimal value to reach, why are they reported in the tables and how should one make an analysis from them?

LIMITATIONS:
The authors did not analyze the potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
This paper proposes a novel method to optimize protein fitness by using bi-level Gibbs sampling with graph-based smoothing. Experiments on real datasets are used to verify the effectiveness of the proposed method.

STRENGTHS:
1. The studied problem about designing novel protein is interesting and challenging. 

2. The proposed method seems to be novel.

3. Experiments show that the proposed method can outperform other baselines to achieve the state-of-the-art performance.

WEAKNESSES:
The writing can be improved. There exist many typos and grammatical errors. Line 106, where is Algorithm 2? Furthermore, the organization can also be improved. For example, Section 3 can be integrated into “Experiment”. It is better to move Section 4 to be just after “Introduction” or just before “Conclusion”.

QUESTIONS:
In Table 3, the fitness is zero without GS. Can you give more explanation about this? Does it mean that other parts of the proposed method have no ability to model the problem?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the problem of optimization over a noisy fitness landscape of sequences. There are two main contributions: 1) Graph-based smoothing (GS) --- considering the landscape as a noisy landscape and applying L1 graph Laplacian regularization. 2) Bi-level Gibbs (BiG) --- using bi-level sampling to propose 5 indices to mutate simultaneously.\n\nTo evaluate the performance of the proposed method, the authors used design benchmarks based on oracles trained on the GFP and AAV datasets, and showed improved sequence optimization on the synthetic design benchmarks.\n\nSTRENGTHS:\nThis work proposes a new sampling method (Bi-level Gibbs with Graph-based Smoothing) in the protein sequence optimization problem. The proposed method is well-motivated. In particular, graph-based smoothing is an interesting approach to address the non-smoothness of protein fitness landscapes. The authors gave a clear description of the proposed method and presented detailed empirical results on simulated data based on real-world experimental data. In the evaluation, the authors took care to avoid overfitting the algorithm and included two datasets (GFP and AAV) to study two different protein fitness landscapes with different properties. The ideas in this work could inspire future work around this important topic.\n\nWEAKNESSES:\nThe main weakness is in the empirical evaluation. The evaluation is based on machine learning models as oracles. Even though the oracle models are trained on experimental data, the fitness landscape in the oracle models might have very different properties compared to real protein fitness landscapes. For example, for graph-based smoothing to work well, it seems like some amount of smoothness in the landscape is needed, and machine learning models might present artificially smoother landscapes than real fitness landscapes. Therefore, one major concern is that the empirical results are tied to the oracle models.\n\nThat being said, it is notoriously challenging to evaluate protein sequence design algorithms as doing so rigorously would require collecting new data points, so this is not a unique weakness to this particular work at hand. In short of performing experimental evaluation, it could relatively strengthen the paper to study more fitness landscapes in the literature and demonstrate evidence that they do have the similar characteristics as the oracle landscapes.\n\nQUESTIONS:\nSee \"weaknesses\" and \"limitations\".\n\nLIMITATIONS:\nThe discussion section does mention limitations around handling variable length sequences and computational speed. It would also be helpful to include a detailed discussion on what types of fitness landscapes BiG would be well-suited for, and counter-examples of fitness landscapes where BiG fails to find optimal solutions. In particular, is it always a good idea to mutate many positions simultaneously? are there situations where it would be preferable to do one mutation at a time?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a Gibbs sampling algorithm with graph signal smoothing that can generate high fitness protein sequence. The proposed method, BiGGS, constructs a smoothed space between protein sequence nodes, and use the gradient of a trained fitness predictor to guide the sampling of both new locations and its corresponding mutations. The authors also provided benchmark dataset with different level of difficulties to validate the efficacy of BiGGS, as well as another dataset that is of different fitness measure to showcase the generalization capability.\n\nSTRENGTHS:\nThe problem of valid protein sequence generation is popularly studied in both computational biology and machine learning community. The proposed method is a combination of traditional sampling techniques, neural predictor for protein sequence, and graph signal smoothing. Overall the paper is well-written with minor typos that don't affect my reading. The problem is well-stated with each component of the method clearly described in details. The authors also find the issues with existing benchmark datasets, which are labelled as \"easy setting\" in this paper based on the distance between highest fitness sequence and the initial set of sequences, and proposed more challenge version of it. In the experimentation section, the authors provided extensive comparison with different baseline methods and BiGGS exhibits significant performance improvement. The ablation study also provide insights on which component brings the most improvement.\n\nWEAKNESSES:\nBesides the strengths mentioned above, there are several areas that can be improved:\n- The hyperparameters seems to be chose arbitrary and the hyperparameter tuning seems difficult. The justification of choice is also missing.\n- The complexity of method is quite high, especially the retraining step after smoothing, which could be a major bottleneck of optimizing the methods in other use cases. \n- The intuition behind graph smoothing is somewhat contradicting to the \"non-smooth\" assumption in protein sequence fitness space, where changing one position will greatly shift the fitness score. Using single mutation neighbors are a bit worrying due to this reason. \n- Lack of comparison with different predictors, or evidence of the existing oracle won't work with the framework.\n\nQUESTIONS:\nThe proposed method seems like a greedy approach, yet it managed to find high fitness samples outside of the initial set. Is there any explanation on why the smoothing and gradient descent/ascend will help avoid local optima and reach higher value samples?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and I don't a potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces BiGGS, a new sequence-based protein fitness optimization algorithm. BiGGS employs bi-level Gibbs sampling for efficient mutation sampling, graph-based smoothing to regularize the fitness landscape, and iterative extrapolation for progressive mutation towards higher fitness. The algorithm's effectiveness is demonstrated through benchmarks on Green Fluorescent Proteins and Adeno-Associated Virus, where BiGGS exhibits state-of-the-art performance in optimizing protein fitness.\n\nSTRENGTHS:\n1. The introduction of BiGGS, a novel sequence-based protein fitness optimization algorithm that incorporates ideas from MCMC, graph Laplacian regularization, and directed evolution.\n\n2. The authors establish a novel benchmark on Green Fluorescent Proteins which presents the unique challenge of initiating the process with sequences of low fitness that require numerous edits to reach peak fitness.\n\n3. The experimental results convincingly demonstrate that BiGGS surpasses other recently proposed methods in terms of both fitness and novelty, marking a significant advancement in the field.\n\n4. The authors provide comprehensive details about the datasets, methods, and evaluation metrics used in their experiments. The well-structured and documented code is an added advantage.\n\n5. The paper also discusses potential extensions of BiGGS, including improving BiG by removing the independence assumption across residues, exploring better regularization techniques for protein fitness predictors, and investigating BiGGS to handle variable length sequences, multiple objectives, and multiple rounds of optimization.\n\nWEAKNESSES:\n1. While the paper presents compelling results, it could be further enhanced by incorporating a case study that applies the proposed method in a real-world scenario. This would provide tangible evidence of the method's effectiveness in practical applications. For instance, the abstract asserts that \"Our method is state-of-the-art in discovering high-fitness proteins with up to 8 mutations from the training set\", but this claim lacks further substantiation within the body of the paper.\n\n2. The paper's evaluation methodology heavily depends on a fitness score, calculated through a trained, black-box neural network. This approach raises questions about the results' reliability and interpretability. It would be beneficial if the authors could explore strategies to increase the transparency and comprehensibility of the evaluation process.\n\n\n3. As someone unfamiliar with protein fitness optimization, I find it challenging to gauge whether this highly specialized task would be of interest to the broader NeurIPS community.\n\nQUESTIONS:\nCould the authors provide information on the prediction error of the trained oracle when applied to unseen sequences? How might this error impact the accuracy of the evaluation?\n\nLIMITATIONS:\nThe paper does not explicitly address any limitations. However, from my viewpoint, a significant limitation lies in the disparity between the evaluation pipeline employed in the paper and the realities of practical application scenarios. The authors could strengthen their work by acknowledging and discussing this gap.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}