# Biomolecular Interaction Prediction in the Pre- and Post-AlphaFold Era: The 8th CAPRI Evaluation

Marc Lensink, Nessim Raouraoua, Guillaume Brysbaert, Sameer Velankar, Shoshana J. Wodak, Alexandre M. J. J. Bonvin

► To cite this version:

Marc Lensink, Nessim Raouraoua, Guillaume Brysbaert, Sameer Velankar, Shoshana J. Wodak, et al.. Biomolecular Interaction Prediction in the Pre- and Post-AlphaFold Era: The 8th CAPRI Evaluation. Proteins - Structure, Function and Bioinformatics, 2025, Proteins: Structure, Function, and Bioinformatics, pp.1-14. 10.1002/prot.70018. hal-05303432

HAL Id: hal-05303432

https://lilloa.hal.science/hal-05303432v1

Submitted on 8 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

RESEARCH ARTICLE

OPEN ACCESS

# Biomolecular Interaction Prediction in the Pre- and Post-AlphaFold Era: The 8th CAPRI Evaluation

Marc F. Lensink<fnref n="1" /> | Nessim Raouraoua<fnref n="1" /> | Guillaume Brysbaert<fnref n="1" /> | Sameer Velankar² | Shoshana J. Wodak³ | Alexandre M. J. J. Bonvin⁴

**Correspondence:** Marc F. Lensink (marc.lensink@univ-lille.fr) | Alexandre M. J. J. Bonvin (a.m.j.j.bonvin@uu.nl)

**Received:** 26 May 2025 | **Revised:** 26 June 2025 | **Accepted:** 30 June 2025

**Funding:** This work was supported by Horizon 2020 Framework Programme, Université de Lille.

**Keywords:** artificial intelligence | biomolecular interaction | blind prediction | CAPRI | protein complexes | protein docking | scoring | structure prediction

## ABSTRACT

We report on the 8th CAPRI Evaluation period, capturing the assessment of CAPRI Rounds 47 to 55 (excluding the CASP and COVID-related Rounds), which have witnessed the transition to AI-driven prediction tools such as AlphaFold and related alternatives. The prediction Rounds in this evaluation are characterized by a high level of difficulty due to various factors, including the nature of the targets, the intricacy of the interfaces to be predicted, and conformational changes. A total of 11 targets encompassing 21 interfaces, mostly in the difficult prediction category, were evaluated. While a retrospective analysis reveals a strong performance of AlphaFold on those targets, human predictors still outperform AI on difficult targets, particularly those involving antibodies and nucleic acids. Almost 25 years after its birth, CAPRI remains a vibrant and collaborative initiative with active participation from approximately 50 predictor and scorer groups and 10 servers. Continued contributions from experimentalists providing targets to such blind experiments, and further advances in AI, sampling strategies, and improvement in scoring methods will be key to overcoming remaining structural prediction challenges in complex biomolecular systems.

## 1 | Introduction

Biomolecular interactions govern a wide range of cellular and intercellular activities. Characterizing these interactions at the atomic level is essential for understanding the mechanisms underlying health and disease and for developing new therapeutics. Experimental structural biology, with X-ray crystallography, Nuclear Magnetic Resonance (NMR) and cryo-electron microscopy (cryo-EM) and tomography (cryo-ET) is continuously providing atomic insights into the three-dimensional (3D) structure of those interactions [1]. Over the years, many other experimental methods have been providing pieces of this interaction puzzle. Similarly, bioinformatics approaches have

also evolved, from predicting binding sites to predicting intermolecular contacts from co-evolutionary signals. These puzzle pieces on their own are not sufficient to solve the 3D structure of a complex and must therefore be combined with computational methods, which has given rise to the field of integrative structural biology [2–4].

On the computational side, docking methods that are generating 3D models of biomolecular complexes from their free components have contributed since ~50 years to populating the interaction landscape. In order to provide an unbiased assessment of the field, CAPRI (Critical Assessment of PRedicted Interactions) was established in 2001 as a community-wide initiative, inspired

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

© 2025 The Author(s). *PROTEINS: Structure, Function, and Bioinformatics* published by Wiley Periodicals LLC.

<footnote n="1">¹University of Lille, CNRS UMR8576 UGSF, Unite de Glycobiologie Structurale et Fonctionnelle, Lille, France | ²European Molecular Biology Laboratory, European Bioinformatics Institute (EMBL-EBI), Wellcome Trust Genome Campus, Cambridge, UK | ³VIB-VUB-Structural Biology Research Center, Brussels, Belgium | ⁴Bijvoet Centre for Biomolecular Research, Faculty of Science - Chemistry, Utrecht University, Utrecht, the Netherlands</footnote>
by CASP (Critical Assessment of protein Structure Prediction) [5]. Its main goal is to assess the current state of the art and catalyze progress in computational methodology for modeling protein complexes. It has done so by offering participants the opportunity of testing their algorithms in blind predictions of experimentally determined 3D structures of protein complexes, the ‘targets’, provided to CAPRI prior to publication. CAPRI’s focus over the years has been mainly on protein–protein assemblies [6], but other types of complexes have been included, such as protein-nucleic acids, protein-peptides and protein-glycans [7–11].

Due to the paucity of available protein complexes that can be used as targets, a CAPRI prediction Round is initiated each time a target (or a few targets) becomes available and completed three to 6 weeks later. Targets are structures of protein complexes/assemblies offered in strict confidence to CAPRI by structural biologists prior to publication. Registered participants are invited to predict the 3D structure of the target protein assembly starting typically from sequence information alone. A Round also includes a scoring challenge in which the correct assembly models must be identified out of a pool of decoys (incorrect models) submitted by the participating predictors. Groups can participate in either or both challenges (as predictors and scorers). Servers can also participate in CAPRI and do so with a much shorter prediction time of at most 3 days.

Historically, most predictions in CAPRI originate from docking and related conformational sampling approaches, often using shape complementarity with various other energy-based terms to score the models [5, 12]. With the publication of AlphaFold2 (AF) in 2021 [13], artificial intelligence (AI) quickly entered the CAPRI prediction field. This catalyzed the development of many other AI-based prediction methods like for example, RoseTTafold [14], AF-multimer [15], OpenFold [16], and many others. The recent release of AlphaFold3 opened the route to the prediction of a variety of complexes including those with nucleic acids and small molecules [17], although with more restrictive access and licensing terms, and was quickly followed by the release of more permissive alternatives, such as Boltz-1 and Chai-1 [18, 19]. These AI methods have demonstrated excellent performance for protein–protein complex prediction in cases where coevolution can be used, but still often struggle with, for example, antibody and protein nucleic acids complexes. Some of these limitations can be lifted by massively increasing the sampling, which can be performed with tools like MassiveFold [20], but this comes at very high computational costs.

In the current manuscript, we describe CAPRI Rounds 47 to 55 (excluding the CASP Rounds 50 and 54, and the COVID effort—Round 51) which have, halfway, witnessed the rise of AI. As of Round 53, AI prediction methods have been integrated into most prediction pipelines of the participating groups. These Rounds were discussed at the 8th CAPRI Evaluation Meeting, which took place in Grenoble in February 2024. AlphaFold2 itself is now used by the CAPRI assessors to define a prediction accuracy baseline and assess the difficulty of a target. In the following, we first briefly present the targets and the assessment methodology and then discuss the overall results

and groups' performance, distinguishing between the pre- and post-AlphaFold era.

## 2 | The Targets

Information on the targets of these Rounds is summarized in Table 1. The targets are listed in chronological order. With one exception, all targets were deemed Difficult. The modeling tasks were various: prediction of domain organization, homo- and hetero-dimer interaction, loop modeling, and protein-DNA, protein-nanobody, and antibody-antigen interaction. The 10 targets are depicted in Figure 1.

Target **T160** consisted of the surface assembly domain of SAP, the surface layer protein from *B. anthracis*. The assembly domain folded into six discrete structural domains and was solved in complex with two (different) antibodies to high resolution by X-ray diffraction. The structure (now published as PDB ID 6HHU) [23] was offered to CAPRI by *Han Remaut* and *Antonella Fioravanti* from the VIB-VUB Center for Structural Biology in Brussels, Belgium. To facilitate the modeling task, the atomic backbone coordinates of the individual domains D1–D6 in random orientation and translation were supplied. The prediction challenge consisted of modeling the 3D structure of the SAP assembly domain, that is, the intramolecular contacts between the SAP domains and the conformations of the connecting loops, as well as the complex with the two nanobodies.

The two targets **T161** and **T162** from Round 48 stem from the same biological system, namely the *E. coli* toxin-antitoxin complex, RnlA-RnlB. T161 consists of the unbound form of the toxin, whereas T162 was the homo-dimer of the toxin protein bound to two antitoxin molecules. It was offered to CAPRI by *Remy Loris* and *Gabriela Garcia-Rodriguez* from the VIB-VUB in Brussels, Belgium, and is now published as PDB ID 6Y2P [24]. In the complex, the toxin homodimer forms a completely different dimer than in its unbound form. The challenge of T161 was to predict the bound form of the toxin homodimer; the challenge of T162 was to predict the structure of the full toxin-antitoxin complex.

Target **T163** is a heterocomplex with A2B2 stoichiometry, consisting of two small helical protein components SYCE2 and TEX12 of the synaptonemal complex. Offered by *Owen Davies* from Newcastle University, UK, now published as PDB ID 6R17 [25].

Target **T186** is a cryo-EM structure, representing a large protein complex that is part of the peripheral arm of the *E. coli* respiratory complex NADH:ubiquitine oxidoreductase. It consists of six polypeptides, one of which has a *ca.* 100 residue loop that is bound to a large pocket of the complex. The loop adopts a well-defined conformation with little secondary structure elements. An existing template from a related complex from *T. thermophilus* lacks the 100-residue loop. The challenge of this target was to correctly model the bound conformation of the loop.

The first “post-AlphaFold” targets were **T187** and **T188**, which both featured the same bacterial (*B. thuringiensis*) protein-DNA complex, representing a transposase homodimer in complex with 2 dsDNA fragments. Both targets required
<table><caption>TABLE 1 | The CAPRI targets of the 8th Evaluation period.</caption><thead><tr><th>Round</th><th>Target</th><th>Stoichiometry</th><th>#int, #tgt</th><th>Area/Å<sup>2</sup></th><th>Experimental technique</th><th>Resolution</th><th>PDB</th><th>Type</th><th>Difficulty</th></tr></thead><tbody><tr><td colspan="10">Pre-AlphaFold targets</td></tr><tr><td rowspan="3">R47</td><td rowspan="3">T160</td><td rowspan="3">A</td><td>6, 6</td><td>150–500</td><td>X-ray</td><td rowspan="3">2.7 Å</td><td>6HHU</td><td>Domain organization</td><td>Difficult</td></tr><tr><td>1, 1</td><td>950</td><td></td><td>6HUU</td><td>Nanobody binding</td><td>Difficult</td></tr><tr><td>A:nb</td><td>1, 1</td><td>600</td><td></td><td>6HUU</td><td>Nanobody binding</td><td>Difficult</td></tr><tr><td rowspan="2">R48</td><td rowspan="2">T161</td><td rowspan="2">A2</td><td>1, 1</td><td>7000</td><td>X-ray</td><td rowspan="2">2.64 Å</td><td>6Y2P</td><td>Toxin homodimer</td><td>Difficult</td></tr><tr><td>1, 1</td><td>1750</td><td>X-ray</td><td>6Y2P</td><td>Enzyme/inhibitor</td><td>Difficult</td></tr><tr><td>R49</td><td>T163</td><td>A2B2</td><td>5, 3</td><td>500–1400</td><td>X-ray</td><td>2.42 Å</td><td>6R17</td><td>Protein/protein</td><td>Difficult</td></tr><tr><td>R52</td><td>T186</td><td>ABCDEFG</td><td>3, 2</td><td>3750</td><td>cryo-EM</td><td>2.1 Å</td><td>7NZ1</td><td>Loop modeling</td><td>Difficult</td></tr><tr><td colspan="10">Post-AlphaFold targets</td></tr><tr><td rowspan="2">R53</td><td rowspan="2">T187</td><td rowspan="2">A2:DNA</td><td>1, 1</td><td>7900</td><td>cryo-EM</td><td rowspan="2">3.1 Å</td><td>7QD5</td><td>Protein/DNA</td><td>Difficult</td></tr><tr><td>1, 1</td><td>7900</td><td>cryo-EM</td><td>7QD5</td><td>Protein/DNA (bound)</td><td>Difficult</td></tr><tr><td rowspan="3">R55</td><td rowspan="3">T231</td><td rowspan="3">A:HL</td><td>1, 1</td><td>900</td><td>X-ray</td><td rowspan="3">1.16 Å</td><td>8RMO</td><td>Antibody/peptide</td><td>Difficult</td></tr><tr><td>1, 1</td><td>1500</td><td>cryo-EM</td><td>N/A</td><td>Protein/protein</td><td>Easy</td></tr><tr><td>A:B</td><td>1, 1</td><td>1650</td><td>X-ray</td><td>N/A</td><td>Antibody/antigen</td><td>Difficult</td></tr><tr><td>T233</td><td>T234</td><td>AB:HL</td><td>1, 1</td><td>1350</td><td>X-ray</td><td>N/A</td><td>N/A</td><td>Antibody/antigen</td><td>Difficult</td></tr></tbody></table>

Note: Targets are listed in chronological order. Rounds 50 and 54 were joint CASP-CAPRI rounds whose results are detailed elsewhere [21, 22]. #int. stands for the number of interfaces, and #tgt for the number of (sub)-targets. The difficulty level is assigned manually, based on template availability (pre-AlphaFold) or AlphaFold performance (post-AlphaFold).
<figure><img src="image_3.png" /><figcaption>**FIGURE 1** | The 10 CAPRI targets of the 8th Evaluation period. The 10 targets (see also Table 1) are labeled by their CAPRI target number. For T160 the individual domains D1–D6 are colored lightblue-cyan-green-yellow-orange-pink and the two nanobodies darkblue and red. For T186 the accessible surface area of the loop to be modeled is shown for visual clarity. For all targets (except T160) the larger entity is colored blue/cyan and the smaller entity red/orange. Images were prepared with PyMOL v2.3.0.</figcaption></figure>

the prediction of the full complex, but for T188 the bound conformation of a single dsDNA fragment was supplied. The bound transposase dimer features a significant conformational change with respect to its unbound dimeric structure. Modeling this conformational change was considered a near-impossibility in “pre-AlphaFold” times, but AlphaFold was found to predict the DNA-bound conformation of the transposase monomer. In addition, the DNA itself showed strong bending; hence for T188 a single dsDNA structure in its bound conformation was supplied. Targets T186 and T187/T188 were provided by Rouslan Efremov from VIB-VUB Brussels, Belgium, and are now published as PDB ID 7NZ1 and 7QD5, respectively [26, 27].

Lastly, Round 55 featured four targets. Target **T231** was an antibody-peptide target provided by Matti Pronker and Bert Janssen from Utrecht University, The Netherlands, and is now published as PDB ID 8RMO [28]. Target **T232**, featuring a phosphatase-regulator complex, was provided by Wei Huang from Case Western Reserve University, USA, and is not yet published. A low-resolution crystal structure of the same complex existed, but it was not found to fit the EM density map. Targets **T233** and **T234**, finally, were two antibodies bound to the human Major Histocompatibility Complex-I. They were offered to CAPRI by Josan Marquez from EMBL Grenoble, France, and are also not yet published.

### 3 | Participation Statistics

A total of 64 groups participated in the six CAPRI Rounds evaluated here. These can be categorized into 61 predictor and 53

scorer groups, with a significant overlap between the two. Table 2 lists the number of registered groups (A) and the number of submitted models (B), with a subdivision into pre- and post-AlphaFold Rounds.

A pronounced drop in registration can be observed between pre- and post-AlphaFold Rounds. This decrease is even more significant if one considers that 10 of the registered groups (8 human, 2 server) only registered for the post-AlphaFold Rounds.

In terms of submitted models, it should be noted that not all registered groups participate in all targets; typically, a target attracts between 20 and 30 predictor groups and between 15 and 20 scorer groups, resulting in a collection of 100–150 predictor models and 150–200 scorer models. Interestingly, these participation statistics do not change between pre- and post-AlphaFold Rounds.

We also note that the total number of submitted models is balanced between pre- and post-AlphaFold rounds, which, together with the lower number of registrants, shows an increase in the number of registered groups that actually submit models. The higher total number of models, which includes the predictor models that go into the Scoring Experiment (see caption of Table 2), is consistent with this analysis.

### 4 | Assessment Criteria and Performance Ranking

The classical CAPRI evaluation and ranking protocols were used to assess the quality of submitted models for most targets
<table><caption>TABLE 2 | CAPRI 8th Evaluation period participation statistics.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Overall</th><th colspan="2">Pre-AlphaFold</th><th colspan="2">Post-AlphaFold</th></tr><tr><th>Total</th><th>Human</th><th>Server</th><th>Human</th><th>Server</th><th>Human</th><th>Server</th></tr></thead><tbody><tr><td colspan="8">(A) Number of groups</td></tr><tr><td>Predictors</td><td>61</td><td>50</td><td>11</td><td>40</td><td>9</td><td>31</td><td>9</td></tr><tr><td>Scorers</td><td>53</td><td>43</td><td>10</td><td>38</td><td>8</td><td>23</td><td>7</td></tr><tr><td>All</td><td>64</td><td>53</td><td>11</td><td>45</td><td>9</td><td>32</td><td>9</td></tr><tr><td colspan="8">(B) Number of models</td></tr><tr><td>Predictors</td><td>1329</td><td></td><td></td><td>629</td><td></td><td>700</td><td></td></tr><tr><td>Scorers</td><td>1640</td><td></td><td></td><td>825</td><td></td><td>815</td><td></td></tr><tr><td>Total</td><td>18 542</td><td></td><td></td><td>8535</td><td></td><td>10 007</td><td></td></tr></tbody></table>

Note: (A) Number of registered predictor and scorer groups, separated into human and server groups. Note that the same group (identified by their PI name) can participate both pre- and post-AlphaFold, and can be both predictor and scorer. (B) Number of evaluated models for the predictor groups (max. 5 submitted models), Scorer groups (max. 10 submitted models) and the total, which includes the predictor models (max. 100 submitted models) that go into the scoring experiment.

[7, 29, 30]. The protocol was modified for the antibody-peptide target T231 of Round 55 as previously to account for the smaller interaction surface [11]. Briefly, to assess the quality of modeled complexes, the native contact recall ($f_{nat}$) was calculated on residue-residue contacts using a 5Å distance threshold (4Å for antibody-peptide). Two residues are considered to be in contact if any of their heavy atoms are within the defined distance cutoff of one another. This metric is complemented by two rmsd-based quantities: the interface rmsd i-rms (residues forming the interface are defined using a 10Å distance threshold (8Å for antibody-peptide)), and the ligand misorientation rmsd L-rms, calculated over the ligand main-chain atoms after optimally superimposing the receptor entities, with the receptor defined as the larger entity of the binding partners. Additional metrics are calculated as well, but these are not used to rank predictions.

The CAPRI assessment is interface centric. Individual interfaces are each assessed separately, with selected combination operators applied to assess the quality of a target as a whole. We use “target” to refer to any ground-truth structure submitted for blind prediction. When a target is too large to be evaluated as a single unit, we split it into smaller subtargets, each itself a target. For the purpose of ranking groups, multiple interfaces from larger targets may be grouped into these subtargets to avoid unduly penalizing performance on especially difficult to predict interfaces. These (sub)-targets were previously called “Assessment Units”, a term we decided to deprecate because it conflates the act of judging with the items being judged and fails to convey their relationship to the original targets; so we now use simply target (whole or fragment) and subtarget. When a target is split into subtargets, the target as a whole is never considered in the ranking.

For three pre-AlphaFold targets, this was the case. T163—a 2 by 2 association of long alpha-helices—was split into three sub-targets: T163.2 and T163.3 counted the best of the two homomeric and three heteromeric interfaces, respectively, whereas T163.1 model quality was determined through the average of all

five interfaces. T186—the loop modeling target—was also split into three subtargets: T186.1, comprising the interface between chains G (including the loop) and D; T186.2, assessing the intrachain interface between the loop and the rest of chain G; and T186.3, which was limited to the interface between the loop and chain D. Finally, T160, which counted no less than 8 different interfaces between the domains 1–6 and the two nanobodies, was split into as many subtargets: T160.1 and T160.2 for the interaction with the two nanobodies; T160.3–7 for each consecutive domain triple (D1,2,3—D2,3,4—D3,4,5—D4,5,6 and D5,6,3); and T168.8 as the best of all 6 domain-domain interfaces.

## 4.1 | Performance Ranking

The performance of predictors, servers, and scorers was ranked as previously described [11]. Briefly, the global score, Score_G, is calculated as follows:

$$ \text{Score}_G = \omega_1 N_{\text{ACC}} + \omega_2 N_{\text{MED}} + \omega_3 N_{\text{HIGH}} $$

Here, N represents the number of targets for which at least an acceptable, medium, or high-quality model was obtained. The weighting factors $\omega$ were taken as 1, 2, and 3, resp.

## 5 | Prediction Results

In the following, we summarize the prediction results for the different targets in chronological order and present the global performance ranking across groups and automatic servers. The performance of predictors and servers was ranked on the basis of the best model in the top five submissions for each target; for scorers, this was the top 10 submissions for each target. A full account of all the results for these targets is available at the official CAPRI website (https://www.capri-docking.org/assessment). The results are split into the pre- and post-AlphaFold era. For an overview of the CAPRI timeline, we refer to Figure 2.
## 5.1 | Results on the Pre-AlphaFold Targets

T160, the surface layer assembly domain of *B. anthracis*, was resolved in complex with two nanobodies by X-ray crystallography at 2.7 Å resolution. This turned out to be a difficult target in all aspects: only three of the six inter-domain interfaces were predicted by more than one predictor group, and none of the predictors nor scorers managed to generate or identify any correct model for the nanobody interfaces. The S-layer domain assembles in a planar S shape, with the C-terminal D6 domain curving back onto the structure, creating an extra interdomain interface (see Figure 3A). The largest interfaces between consecutive domains—interfaces 3 and 6—were, together with interface 8, the best predicted. Medium-quality submissions were made by the groups of Andreani/Guerois, Kozakov/Vajda, and Venclovas. Acceptable models were, in addition to these groups, created by Fernandez-Recio and the GalaxyPPDock server. Gray produced an acceptable model for interface 7. With so few acceptable or better models in the set, singling out the good models was a challenging task for the scorers. Venclovas correctly identified the best model by Andreani/Guerois and obtained medium quality for interfaces 3 and 8. This medium-quality model is depicted in Figure 3B. Fernandez-Recio, Kihara, Venclovas, Bates, and the server LZerD achieved acceptable quality for interface 6, whereas the server HDOCK managed an acceptable-quality model for the

smallest interface, interface 9. The target split-up resulted in a final score of 5/1** for Andreani/Guerois, 3/1** for Kozakov/ Vajda and Venclovas, and one acceptable target for Fernandez- Recio, Gray, and the GalaxyPPDock server. For scorers, Venclovas had 5/1**, and Kihara, Bates, and the LZerD and HDOCK servers had one acceptable target.

The two targets **T161** and **T162** of Round 48 feature the same protein complex: the RnIA-RnIB toxin-antitoxin complex. The difficulty in this system was a conformational change of the RnIA dimer upon binding of RnIB. The longer RnIA toxin consists of three domains: D1, D2, and D3. The unbound RnIA homodimer associates through D3. Upon binding of two RnIB antitoxin molecules, the bound RnIA homodimer shows a conformational rearrangement for D1 and D2, forming an intertwined dimer through D2 and D3. Each D3 domain binds one RnIB molecule. D1 could therefore be ignored for both targets; the information that was shared with the predictors was to focus on D3, know that D2 moves, and ignore D1. For **T161**, the bound RnIA dimer, none of the predictors foresaw the D2 intertwining. However, Bates managed—as the only predictor—to generate an acceptable model for the dimer through a correct association of the two D3 domains. For **T162**, which could be reduced to the binding of RnIB to RnIA/D3, all submitted models (1756 predictor and 165 scorer models; totaling 1921 structures) were made available to predictors. The poor

<figure><img src="image_4.png" /><figcaption>**FIGURE 2** | The timeline of the 8th CAPRI Evaluation period. CAPRI Rounds and Meetings are indicated by blue and cyan boxes, resp. CASP Rounds and Meetings are indicated by pink and brown boxes, resp. The release of AlphaFold v2 is indicated by the green box, separating the timeline in a pre- and post-AlphaFold period.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**FIGURE 3** | Organization of the B. anthracis surface layer assembly domain, T160. (A) Organization of domains D1—D6. The two nanobodies are not depicted, as they lie on top, perpendicular to the plane of the molecule; Nb694 interacts with D1 through interface 2, and Nb684 interacts with D1 and D2 through interfaces 1 and 5. The inter-domain surfaces are indicated with dashed lines and numbered in decreasing size, going from 610 Å² for interface 1 to 150 Å² for interface 9. Accurately predicted interfaces (by any group) are shown in red. (B) Model 2 of the Andreani/Guerois group overlapped onto the target. This was the only model accurately predicting three of the nine interfaces: Interfaces 3 and 8 to medium quality, and interface 6 to acceptable quality. The model is depicted in blue and purple to account for the fact that interface 7 was wrongly predicted.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**FIGURE 4** | Combined interface quality of targets T187 and T188. The figure shows for the top five predictor models submitted to T187 and T188 the interface quality as expressed by the *i*-rms for the protein–protein (y-axis) and protein-dsDNA (x-axis) interfaces. T187 submissions are indicated by diamonds and T188 by right-handed triangles. The color of the icons reflects the CAPRI quality of the protein dimer interface. Note that only the protein-DNA interface was part of the assessment and not the protein dimer interface. The only three acceptable models for T188 are encircled.</figcaption></figure>

performance on T161 may have impacted the results on T162, as only one team—Andreani/Guerois—managed to produce correct models, with one of them of medium quality. This model was not based on any of the submitted models for T161. No acceptable scorer submissions were made for these two targets.

T163 of Round 49 featured a 2:2 association of two helical protein components, SYCE2 and TEX12. An unbound dimeric structure of TEX12 could be identified, but its binding mode differs from its association in T163. The largest interface between the two SYCE2 molecules buries 1400 Å². The smallest interface, number 5, between the two TEX12 molecules buries a third of that. The other three interfaces are heteromeric and bury between 900 and 1200 Å². The largest heteromeric interface was the only interface correctly predicted by three predictor and three scorer groups. All other interfaces were correctly predicted by at most one predictor or scorer group; with the notable exception of Kihara—nobody managed to correctly predict two interfaces. As one of the interfaces was heteromeric (T163.3) and the other homomeric (T163.2), Kihara achieved 2/1** on this target, while Venclovas, Kihara, Gray, and the LZerD server all had acceptable models. Scorers Seok and Kihara both identified an interface of medium quality, and Fernandez-Recio, Oliva, and Perthold of acceptable quality.

T186 featured the oxidoreductase complex of the respiratory complex peripheral arm of *E. coli*. A template from *T. thermophilus* lacking the ~100 residue loop to be modeled was provided. The loop was part of the chain G sequence and buried 2675 Å² with the remainder of chain G (T186.2), and 1070 Å² with chain D (T186.3). This was yet again an extremely challenging target, with no acceptable solutions for T186.3, and only one predictor (Kihara) and 2 scorer groups (Kihara and Venclovas) obtaining acceptable quality models for T186.2. Significantly more groups (five predictors and 12 scorers

including three scoring servers) achieved acceptable quality models for T186.1, the interface between chains G and D, but the fact that a template was available, and no medium quality model was obtained illustrates the poor quality of the loop modeling.

## 5.2 | Results on the Post-AlphaFold Targets

The first targets of the post-AlphaFold era would not have been proposed as CAPRI targets had AlphaFold not existed. **T187** and **T188** feature the same complex, a transposase homodimer bound to two double-stranded DNA molecules. An unbound structure of the transposase homodimer existed, but it would not have been able to accommodate the dsDNA, making the modeling a virtually impossible task. However, upon running AlphaFold, we discovered that it was modeling the *bound* state of the monomer. At the time, it was not evident to run multimers with AlphaFold. As such, most predictors used classical docking to create the dimer, and then protein-DNA docking to create the full complex. Modeling a straight double-stranded helix would not allow capturing the majority of the protein-DNA contacts properly, explaining the fact that no acceptable solutions were obtained for T187. Providing a single dsDNA molecule in bound conformation for T188 removed the requirement of modeling the correct conformation of the DNA. Still, only two predictor groups managed to produce acceptable models: Fernandez-Recio and Pierce. It should be noted that Fernandez-Recio did not produce an acceptable protein dimer model based on CAPRI criteria for the protein–protein interface, but did reproduce the protein-DNA contacts quite accurately, resulting in a final model with $f_{nat}$ = 0.338 and *i*-rms = 4.0 Å. Pierce, however, did produce an acceptable dimer model, resulting in a better final model, with $f_{nat}$ = 0.405 and *i*-rms = 3.3 Å. Following our expectations, producing an acceptable transposase dimer was possible, although not a given, as illustrated in Figure 4. Many acceptable and medium-quality protein dimers were created, having *i*-rms
roughly between 2 and 4 Å, but the modeling of the protein-DNA interface ranged between 3 and more than 60 Å. Five models are worth noting: the three already mentioned acceptable models for T188, and two incorrect models for T187 (one with medium quality protein dimer interface), both by Kihara. These models are the five left-most points in Figure 4.

Round 55 represents the first true post-AlphaFold CAPRI Round, in the sense that by now all participants had incorporated AlphaFold in one way or another into their pipelines. For all targets in this Round, the default 25 models as calculated by AlphaFold v2.3 were made available to predictors. The only “conventional” target in that Round was **T232**, an enzyme/inhibitor complex featuring a serine/threonine phosphatase in complex with its regulator TIPRL. A 3.8 Å crystal structure of the complex existed (PDB ID 5W0W) [31], which was not compatible with the EM density map of T232. This information was shared with the participants, but the density map was not. The phosphatase consists of two chains, A and C, and in the crystal structure, TIPRL makes contact with both chains. In the EM structure of T232, TIPRL is only in contact with chain C, burying 1000 Å². It is displaced by 25.9 Å and rotated over 129°, resulting in a completely different interface, as indicated by the *i*-rms vs. the crystal structure of 18.6 Å. This was, however, not a difficult target: all 25 AlphaFold models were of medium quality. No less than 24 of the 25 predictor groups also submitted medium quality models for this target, as did all 14 scorer groups. These included 5 docking servers and 3 scoring servers. Interestingly, virtually no lower quality, acceptable models were submitted.

The other three targets of the Round were antibodies. **T231** featured an antibody binding to a short peptide fragment, requiring the application of the more stringent criteria developed for protein-peptide assessment [10, 11]. The pre-calculated AlphaFold models were mostly of acceptable quality, with two

<figure><img src="image_7.png" /><figcaption>**FIGURE 5** | Best model for the antibody-peptide structure T231. Superimposition of the best prediction for T231 onto the reference structure. The structures are colored as follows: Magenta for the antibody heavy chain, yellow for the light chain, and green for the bound peptide; the reference structure is colored blue. The model was submitted by the Kozakov/Vajda team as first ranked.</figcaption></figure>

medium-quality (ranked 14 and 15) and one incorrect model (ranked 22). Medium-quality models were submitted by 10 predictor groups, among which three servers, with an additional 8 groups submitting models of acceptable quality. Of the 14 scorer groups, 6 submitted medium-quality models, and another 7 submitted acceptable-quality models. The best submitted model, shown in Figure 5, came from the Kozakov/Vajda group: it was their first-ranked model and featured a native contact recall ($f_{nat}$) of 82.6%, with a *L*-rms of 1.28 Å and *i*-rms of 0.71 Å. This model was recognized by the HDOCK scoring server and ranked 6 in their submission.

Targets **T233** and **T234** constitute the final targets of this assessment period. Both targets feature binding of an antibody to the human major histocompatibility complex-I: HLA-A*11:01 for T233 and HLA-A*02:01 for T234. As for all these systems, a natural peptide was found in the HLA pocket, but its sequence could not be determined, and it was not relevant for the interactions. As such, its presence was not disclosed to the predictors, an information that could have been relevant for the modeling from the a posteriori feedback we received. This was a difficult target, as indicated by the 25 pre-calculated AlphaFold structures, which were all incorrect. Nonetheless, 6 predictor groups (Kihara, Zou, Venclovas, S.Chang, Huang, and server HDOCK) all managed to obtain medium-quality predictions for T233. No acceptable submissions were recorded for T234.

In such cases, when few acceptable models are produced, the CAPRI Scoring Experiment is particularly motivating. Whereas only the first five submitted models are included for the participant ranking, all submitted models (up to 100) are included in the Scoring Experiment, and all submitted models are assessed. The models are shuffled and anonymized and provided to the scorers. For T233, this resulted in a set of 1696 models, of which 7 were of acceptable quality, 18 of medium quality, and a single one was of high quality. Therefore, barely 1.5% of models in the scoring set were of acceptable quality or better. Six scorer groups (Fernandez-Recio, Zou, Venclovas, S.Chang, Huang, and server HDOCK) managed to recognize a medium-quality model. The high-quality model in the set, submitted by Pierce as model 34, with $f_{nat}$ = 0.776, *L*-rms = 3.26 Å, and *i*-rms = 0.94 Å, was however not recognized. While there were no predictor-submitted models of acceptable quality or better for T234, there was one acceptable model submitted by scorer Giulini. This was one out of only four “correct” models in the scorer set (three acceptable and one medium), out of a total of 1677 models, corresponding to a mere 0.24%. The medium-quality model in the set, submitted by Venclovas as ranked 48th ($f_{nat}$ = 0.818, *L*-rms = 18.26 Å, *i*-rms = 1.18 Å), was not identified by any of the scorers.

## 5.3 | AlphaFold Performance

On July 15th, 2021, AlphaFold v2 was published with open source code [13]. Less than 2 weeks later, the DeepMind and ColabFold notebooks ensured global accessibility [32]. Rounds 47–49 and 52 of this Evaluation period were held before the release of AlphaFold, and Rounds 53 and 55 were held after. While the first AlphaFold-Multimer version (v2.1) became available on Oct 5th, 2021, it was plagued by clashes and
<table><caption>**TABLE 3** | AlphaFold performance on the 8th Evaluation period CAPRI targets.</caption><thead><tr><th>Target</th><th>#models produced</th><th>AF versions</th><th>Interface</th><th>Best predictors</th><th>AlphaFold</th></tr></thead><tbody><tr><td colspan="6">Pre-AlphaFold targets</td></tr><tr><td rowspan="3">T160</td><td rowspan="3">25</td><td rowspan="3">v2.1</td><td>3 & 8</td><td>Medium</td><td>High</td></tr><tr><td>6</td><td>Acceptable</td><td>Medium</td></tr><tr><td>7</td><td>Acceptable</td><td>Incorrect</td></tr><tr><td></td><td></td><td>Rest</td><td>Incorrect</td><td>Incorrect</td><td></td></tr><tr><td>T161</td><td>50</td><td>v2.1, v2.2</td><td></td><td>Acceptable</td><td>Medium</td></tr><tr><td>T162</td><td>50</td><td>v2.1, v2.2</td><td></td><td>Medium</td><td>High</td></tr><tr><td rowspan="5">T163</td><td rowspan="5">50</td><td rowspan="5">v2.1, v2.2</td><td>1</td><td>Incorrect</td><td>Medium</td></tr><tr><td>2</td><td>Acceptable</td><td>Incorrect (+4 × medium with clashes)</td></tr><tr><td>3</td><td>Incorrect</td><td>High</td></tr><tr><td>4</td><td>Acceptable</td><td>Medium</td></tr><tr><td>5</td><td>Medium</td><td>Medium</td></tr><tr><td>T186</td><td>25</td><td>v2.1</td><td></td><td>Acceptable</td><td>High</td></tr><tr><td colspan="6">Post-AlphaFold targets</td></tr><tr><td>T187<sup>a</sup></td><td>50</td><td>v2.1, v2.2</td><td></td><td>Medium</td><td>Medium</td></tr><tr><td>T231</td><td>25</td><td>v2.3</td><td></td><td>Medium</td><td>Acceptable</td></tr><tr><td>T232</td><td>25</td><td>v2.3</td><td></td><td>Medium</td><td>Medium</td></tr><tr><td>T233</td><td>25</td><td>v2.3</td><td></td><td>Medium</td><td>Incorrect</td></tr><tr><td>T234</td><td>25</td><td>v2.3</td><td></td><td>Incorrect</td><td>Incorrect</td></tr></tbody></table>

Note: Targets are listed in chronological order. For each listed AF version, 25 models were created; the top-5 following confidence rankings were evaluated using the exact same CAPRI criteria as for predictor and scorer submissions.

unrealistic protein structures. An updated version, v2.2, was released 6 months later, and by the time Round 55 was organized, all predictors had fully integrated AlphaFold into their respective pipelines. It was for obvious reasons not used before R53, whereas for R53, AlphaFold was used “as is”, often only in monomer mode.

Both AlphaFold v2.1 and v2.2 were parameterized on the PDB data set up to April 2018. Since all targets of this evaluation period were released afterwards, they fall out of the training set and could therefore be predicted with AlphaFold, which we did. In order to avoid introducing any bias due to template availability, the template date limit for any target was set to the first of the month that the respective target was released, for example, for T160, released on Aug 29th, 2018, the template date limit was set to Aug 1st, 2018. All other settings were left to default, and no relaxation was performed. The AlphaFold prediction results are listed in Table 3. Overall, it can be seen that for the pre-AlphaFold targets, AlphaFold does remarkably well, globally improving the prediction quality over the best predictors by one model quality category, but only for those interfaces where at least an acceptable level was reached. The notable exception is T163, where—for all five interfaces collectively—AlphaFold produces a medium-quality model; the clashes for the second interface are only just above the clash threshold and would likely be solved with relaxation.

<figure><img src="image_8.png" /><figcaption>**FIGURE 6** | AlphaFold model for the loop in T186. The reference structure is colored as follows: Chain D, blue; chain G, brown; the loop, yellow. The AlphaFold model (ranked 1) is superimposed onto the reference structure and colored gray. This model is of high quality, that is, both L-rms and i-rms values are below 1 Å.</figcaption></figure>

A remarkable performance can be observed for target T186 in modeling the ~100-residue loop, as AlphaFold achieves a high-quality prediction, with rms values below 1 Å. A detailed

<footnote>ªProtein-protein interface only.</footnote>
view of the loop, including the chains of the AlphaFold model superimposed onto the target, is shown in Figure 6. For target T160, the organization of the surface layer assembly domain, near-native structures of the linkages between domains D1, D2, and D3, and between domain D4 and D5, are obtained. However, as for the conventional predictors, all nanobody binding models were incorrect (interface 2 for Nb694, and interfaces 1 and 5 for Nb684). The difficulty for AlphaFold in predicting nanobody and antibody binding has been observed before [21]. The poor performance on the three antibody targets of post-AlphaFold Round 55 is therefore not surprising, but it should be noted that the best predictor groups did produce medium quality models.

# 6 | Performance of Predictors, Servers, and Scorers

Groups (predictors, servers, and scorers) were ranked according to their prediction performance over the 10 targets evaluated here (21 (sub)-targets). This ranking represents the official performance ranking for the 8th Evaluation period, comprising Rounds 47–49, 52, 53, and 55. The ranking includes contributions from both the pre- and post-AlphaFold Rounds, which are listed separately. In addition, we also discuss the performance of vanilla AlphaFold. For a full account of the results obtained by each group, we refer the reader to the CAPRI web site (https://www.capri-docking.org/assessment/).

## 6.1 | Performance of Predictor Groups

The consolidated ranking of predictor groups (including servers) is shown in Table 4A. The ranking shows the four best performing groups to be those of Venclovas, Kihara, and the teams Andreani/Guerois and Kozakov/Vajda, with correct predictions for at least 5 targets. One other group—Fernandez-Recio—shows correct predictions for 5 targets, but only one of those is of medium quality. Several observations can be made. The four mentioned groups are the only groups that managed to obtain medium-quality models for the pre-AlphaFold targets, whereas most groups submitted medium-quality models for one or even several of the post-AlphaFold targets. Of these four, the performance of the Andreani/Guerois team is remarkable, since they have not participated in any of the post-AlphaFold targets. Had they done so and only submitted default AlphaFold models (see the following subsection), they would have shared the top position with the Venclovas team.

Focusing rather on the real-life post-AlphaFold targets, no less than eight groups managed to predict three targets to acceptable quality or better. Venclovas, Zou, and Huang and his server HDOCK predicted all of these to medium quality and would rank therefore *ex aequo* at first position. The other four groups—Kihara, S. Chang, Pierce, and Fernandez-Recio—had models of acceptable quality in the set. It should be noted that Pierce and Fernandez-Recio were the only predictors with acceptable models for the difficult target T187/T188, capitalizing on their expertise in protein-DNA modeling.

## 6.2 | Performance of AlphaFold

Translating the per-interface prediction performance of AlphaFold (as shown in Table 3) to targets results in a ranking performance of 14/5***/4**. This, compared to the best performance of 7/4** by the group of Venclovas, would seem to indicate that AlphaFold largely outperforms all predictor groups. However, this image is heavily skewed. Dividing the contribution into a pre- and post-AlphaFold contribution shows that their overall performance is largely due to the exceptional models that were produced for the pre-AlphaFold targets, resulting in a 12/5***/3** performance, whereas for the post-AlphaFold targets only 2/1** is obtained, bearing in mind that the protein dimer interface of T187 was not a target. Comparing the AlphaFold performance on the pre-AlphaFold targets to that of the best predictors, it is obvious that AlphaFold generated far better models.

Prior to the public release of AlphaFold, the CASP14/CAPRI prediction Round was held, with results presented at the online CASP14 meeting [22]. Targets presented there were also not part of the AlphaFold training data, so we can apply vanilla AlphaFold to that set as well, and compare to the results obtained here. The score of the best predictor then—Seok—was 9/4**, that is, 9 targets were correctly predicted, of which four were of medium quality. Seok was the only predictor with 9 correct targets, but without any of them being of high quality. Other predictors did have high-quality targets, but none of them had more than one. Applying AlphaFold v2.1 to the CASP14/CAPRI target set, we obtain a score of 9/4***/1**, that is, the same 9 correct targets that Seok obtained, but with superior quality as four of these targets were of high quality. In our report, we already noted the extraordinary capability of AlphaFold to generate the correct structure of protein monomers in their oligomeric assembly state [22, 33].

## 6.3 | Performance of Scorer Groups

The ranking of scorer groups is shown in Table 4C. Similar to the predictor performance, the list is topped by four predictor groups that managed medium-quality models for both pre- and post-AlphaFold targets. These are Venclovas, Huang, Fernandez-Recio, and Kihara. Venclovas is the top scorer group (they were also the top predictor group). Fernandez-Recio and Zou were the only scorer groups with models of acceptable quality or better for 4 of the 6 post-AlphaFold targets, one of them being T188, the protein-DNA target.

Most scorers seem to do significantly better at scoring than predicting; Zou has 5/3** as a scorer, but only 3** as a predictor, Fernandez-Recio lists 7/3** as a scorer, but only 5/1** as a predictor, and top scorer and predictor Venclovas scores 11/3** and 7/4**, resp. This apparent discrepancy is only due in part to the fact that scorer ranking considers the top-10 submissions, as opposed to top-5 for predictors. Scorer Venclovas, for instance, would still rank first—with 10/3**—if only the top-5 were taken into consideration, but would fall to third position for top-1 ranking, scoring only 2**. In contrast to this, predictor Venclovas ranks first in all three cases, with 7/4** for both
<table><caption>TABLE 4 | Predictor and scorer performance.</caption><thead><tr><th rowspan="2">Rank</th><th rowspan="2">Group name</th><th rowspan="2">#targets</th><th colspan="2">Top 5</th><th colspan="2">Pre-AlphaFold</th><th colspan="2">Post-AlphaFold</th></tr><tr><th>Performance</th><th>Rank</th><th>Performance</th><th>Rank</th><th>Performance</th></tr></thead><tbody><tr><td colspan="9">(A) Predictor group ranking (human and server)</td></tr><tr><td>1</td><td>Venclovas</td><td>20</td><td>7/4**</td><td>2</td><td>4/1**</td><td>1</td><td>3**</td><td>3**</td></tr><tr><td>2</td><td>Kihara</td><td>21</td><td>7/3**</td><td>2</td><td>4/1**</td><td>5</td><td>3/2**</td><td>3/2**</td></tr><tr><td>3</td><td>Andreani/Guerois</td><td>13</td><td>6/2**</td><td>1</td><td>6/2**</td><td>—</td><td>—</td><td>—</td></tr><tr><td>4</td><td>Kozakov/Vajda</td><td>20</td><td>5/3**</td><td>4</td><td>3/1**</td><td>8</td><td>2**</td><td>2**</td></tr><tr><td>5</td><td>Huang</td><td>21</td><td>4/3**</td><td>7</td><td>1</td><td>1</td><td>3**</td><td>3**</td></tr><tr><td>6</td><td>Fernandez-Recio</td><td>21</td><td>5/1**</td><td>5</td><td>2</td><td>8</td><td>3/1**</td><td>3/1**</td></tr><tr><td>7</td><td>Zou, HDOCK</td><td>21</td><td>3**</td><td>—</td><td>0</td><td>1</td><td>3**</td><td>3**</td></tr><tr><td>8</td><td>S.Chang</td><td>18</td><td>4/2**</td><td>7</td><td>1</td><td>5</td><td>3/2**</td><td>3/2**</td></tr><tr><td>9</td><td>LZERD</td><td>21</td><td>3/2**</td><td>7</td><td>1</td><td>8</td><td>2**</td><td>2**</td></tr><tr><td>10</td><td>Pierce</td><td>18</td><td>3/2**</td><td>—</td><td>0</td><td>5</td><td>3/2**</td><td>3/2**</td></tr><tr><td>11</td><td>Vakser</td><td>17</td><td>2**</td><td>—</td><td>0</td><td>8</td><td>2**</td><td>2**</td></tr><tr><td>12</td><td>Gray</td><td>17</td><td>3/1**</td><td>5</td><td>2</td><td>18</td><td>1**</td><td>1**</td></tr><tr><td>13</td><td>CLUSPRO</td><td>19</td><td>2**</td><td>—</td><td>0</td><td>8</td><td>2**</td><td>2**</td></tr><tr><td>14</td><td>Bates</td><td>13</td><td>3/1**</td><td>7</td><td>1</td><td>14</td><td>2/1**</td><td>2/1**</td></tr><tr><td>15</td><td>Furman</td><td>6</td><td>3/1**</td><td>7</td><td>1</td><td>14</td><td>2/1**</td><td>2/1**</td></tr><tr><td>16</td><td>Karaca</td><td>4</td><td>2**</td><td>—</td><td>—</td><td>8</td><td>2**</td><td>2**</td></tr><tr><td>17</td><td>Brysbaert, Giulini</td><td>4</td><td>2/1**</td><td>—</td><td>—</td><td>14</td><td>2/1**</td><td>2/1**</td></tr><tr><td colspan="9">(B) Server group ranking</td></tr><tr><td>1</td><td>HDOCK</td><td>21</td><td>3**</td><td>—</td><td>0</td><td>1</td><td>3**</td><td>3**</td></tr><tr><td>2</td><td>LZERD</td><td>21</td><td>3/2**</td><td>1</td><td>1</td><td>2</td><td>2**</td><td>2**</td></tr><tr><td>3</td><td>CLUSPRO</td><td>19</td><td>2**</td><td>—</td><td>0</td><td>2</td><td>2**</td><td>2**</td></tr><tr><td>4</td><td>HADDOCK</td><td>18</td><td>1**</td><td>—</td><td>0</td><td>4</td><td>1**</td><td>1**</td></tr><tr><td>5</td><td>MDOCKPP</td><td>21</td><td>1**</td><td>—</td><td>0</td><td>4</td><td>1**</td><td>1**</td></tr><tr><td>6</td><td>GALAXYPPDOCK</td><td>13</td><td>1</td><td>1</td><td>1</td><td>—</td><td>—</td><td>—</td></tr><tr><td>7</td><td>PYDOCKWEB</td><td>9</td><td>1</td><td>—</td><td>0</td><td>6</td><td>1</td><td>1</td></tr><tr><td colspan="9">(C) Scorer group ranking (human and servers)</td></tr><tr><td>1</td><td>Venclovas</td><td>21</td><td>11/3**</td><td>1</td><td>8/1**</td><td>6</td><td>3/2**</td><td>3/2**</td></tr><tr><td>2</td><td>Huang</td><td>21</td><td>8/4**</td><td>2</td><td>5/1**</td><td>2</td><td>3**</td><td>3**</td></tr><tr><td>3</td><td>Fernandez-Recio</td><td>21</td><td>7/3**</td><td>4</td><td>3/1**</td><td>2</td><td>4/2**</td><td>4/2**</td></tr><tr><td>4</td><td>Kihara</td><td>21</td><td>7/2**</td><td>3</td><td>4/1**</td><td>8</td><td>3/1**</td><td>3/1**</td></tr><tr><td>5</td><td>Zou</td><td>21</td><td>5/3**</td><td>9</td><td>1</td><td>1</td><td>4/3**</td><td>4/3**</td></tr><tr><td>7</td><td>HDOCK</td><td>21</td><td>5/3**</td><td>6</td><td>2</td><td>2</td><td>3**</td><td>3**</td></tr><tr><td>8</td><td>S.Chang</td><td>10</td><td>4/3**</td><td>9</td><td>1</td><td>2</td><td>3**</td><td>3**</td></tr><tr><td>9</td><td>Oliva</td><td>20</td><td>4/2**</td><td>6</td><td>2</td><td>8</td><td>2**</td><td>2**</td></tr><tr><td>10</td><td>LZERD</td><td>21</td><td>5/1**</td><td>5</td><td>3</td><td>11</td><td>2/1**</td><td>2/1**</td></tr><tr><td>11</td><td>MDOCKPP</td><td>21</td><td>4/1**</td><td>9</td><td>1</td><td>8</td><td>3/1**</td><td>3/1**</td></tr></tbody></table>

(Continues)
<table><caption>TABLE 4 | (Continued)</caption><thead><tr><th rowspan="2">Rank</th><th rowspan="2">Group name</th><th rowspan="2">#targets</th><th colspan="2">Top 5</th><th colspan="2">Pre-AlphaFold</th><th colspan="2">Post-AlphaFold</th></tr><tr><th>Performance</th><th>Rank</th><th>Performance</th><th>Rank</th><th>Performance</th></tr></thead><tbody><tr><td>12</td><td>Giulini</td><td>4</td><td>3/2**</td><td>—</td><td>0</td><td>6</td><td>3/2**</td></tr><tr><td>13</td><td>Bates</td><td>21</td><td>3/1**</td><td>9</td><td>1</td><td>11</td><td>2/1**</td></tr></tbody></table>

Note: (A) Predictor groups, human and servers combined; (B) automatic servers; (C) scorer groups, human and servers combined. The lists are restricted to those groups producing at least acceptable models for two—(A) and (C)—or one—(B)—target, resp. For all groups, the model of highest quality in their top-5 submission (top-10 for scorers) for every target is considered, and ranking is performed following the formula for Score_G as outlined in the text. Groups are listed by the name of the corresponding PI. Servers are listed by their acronym. The performance is listed by the number of targets for which a model of acceptable quality or better was produced, followed by the number of these models that were of medium (**) quality. A dash indicates that no submission was made.

<figure><img src="image_9.png" /><figcaption>FIGURE 7 | Rank consistency for top predictors and scorers. The x-axis shows the rank for predictors (brown) and scorers (orange), both based on top-5 submissions. The y-axis shows the average rank over the top-1, top-5 and top-10 submissions. Only the first nine predictors and scorers are included. The dashed lines are linear regression fits calculated through the points; their correlation coefficients are shown in the legend.</figcaption></figure>

top-10 and top-5, and 6/3** for top-1. The Scoring Experiment in CAPRI, originally suggested at the CAPRI meeting in Gaeta, Italy in 2004, and implemented for the third CAPRI meeting in Toronto, Canada, significantly enriches in correct structures the set of structures one can choose from [7, 8]. Paradoxically though, this makes identifying good models easier, but identifying the best models more difficult.

Figure 7 shows the rank consistency for the top nine predictors and scorers. The figure shows the scaled ranksum (the average rank based on top-1, top-5 and top-10 submissions) as a function of actual rank, which—for the purpose of comparison—is calculated on the top-5 submissions for both predictors and scorers. The figure shows a close to ideal linear behavior for predictors; for example, Venclovas, Kihara, and Andreani/Guerois are ranked first, second, and third, but are also ranked exactly first, second, and third on the average ranking. In contrast to this, *scorers* Venclovas

and Kihara do slightly worse on the average ranking: any deviation from the diagonal indicates discrepancy between top-1, top-5, and top-10 based ranking. Whereas for predictors the behavior is linear, for scorers the data points are much more scattered. Linear regression fits indeed show a strong correlation for predictors, with a correlation coefficient nearing unity, but this value is only 0.32 for the scorer data. These data might hint at participants being more at ease with scoring their own models than those coming from multiple sources, as is the case in the CAPRI Scoring Experiment, but in any case illustrate the necessity of further developing scoring functions that are robust enough and not sensitive to the details of how the models were produced.

## 6.4 | Performance of Prediction Servers

From the server ranking in Table 4B, we note a poor performance overall on the difficult pre-AlphaFold targets, with only LZerD and GalaxyPPDock managing an acceptable model for a single subtarget, in both cases thanks to the correct prediction of one of the interfaces of a larger assembly (T163 for LZerD and T160 for GalaxyPPDock). The ranking is therefore dominated by the four post-AlphaFold targets, three of which included antibodies. HDOCK stands out, having achieved medium quality for three of the four targets, followed by LZerD and ClusPRO, which both managed two medium-quality submissions. Only a selected few predictor groups achieved 3** on the post-AlphaFold targets: Venclovas, Zou, and Huang, the last being the author of HDOCK.

## 7 | Conclusion and Discussion

The CAPRI 8th Evaluation results were originally presented at the 8th CAPRI assessment meeting, which took place in Grenoble, France, in February 2024, kindly organized by *Sergei Grudinin* and local co-organizers. The assessment presented here has been carried out over 11 target complexes representing a total of 24 evaluated interfaces. The large difference between the number of targets and evaluated interfaces comes mainly from T160, which was basically 3 targets combined into one (the domain organization and two nanobodies binding to it), but also the domain organization itself was split into six subtargets. All but one target belonged to the difficult prediction category, which is clearly reflected in the absence of any high-quality model from both predictors and scorers in both the pre- and post-AlphaFold eras (in fact all targets would have been considered
difficult if AlphaFold would not have become available). This reflects the challenging nature of the CAPRI endeavor, which, over the years, has been and remains a constant catalyst of new developments in the prediction of biomolecular complexes.

Overall, the CAPRI community is very much alive with about 50 predictor human groups and 10 servers participating and only a slightly lower number for the scoring rounds. We do, however, observe a slight drop in the post-AlphaFold era. Despite the rise of the AI, the best predictors remain human teams, with *Venclovas* leading in both the predictions and scoring categories, closely followed by the predictor *Kihara* and scorer *Huang* groups. The latter two groups are also the ones developing the top 2 ranked servers HDOCK (Huang) and LZerD (Kihara), which are participating in both prediction and scoring rounds.

The most striking event in this assessment period has been the public and open-source release of AlphaFold and its model weights. All groups have embraced and integrated in one way or another the new AI tools in their prediction pipelines. While our post-prediction analysis using AlphaFold indicates a high performance of AlphaFold over the 21 evaluated targets with 14 acceptable or better predictions (about twice the number from best human predictors), this number is biased by a few pre-AlphaFold targets with multiple interfaces. Focusing on the post-AlphaFold targets, the situation is different, with the best performance still being obtained by human predictor groups. Furthermore, in this evaluation round, the few antibody and nanobody targets, which typically lack any conservation in their binding regions, clearly reveal the limitations of AlphaFold for these kinds of targets. Future CAPRI rounds will have to show if more recent AI models like, for example, AlphaFold3, do perform better for this type of complexes. Another development which might boost the prediction performance in the future is adaptations of the sampling strategies, as done, for example, in MassiveFold [20] and by various CAPRI predictor groups [34–38]. This, however, comes at extremely high computational costs and might be difficult to implement in CAPRI settings where targets can be released for prediction at any time during the year and on short notice. Furthermore, with such large datasets of models, scoring might remain the main limiting factor, as indicated in the latest CASP16 assessment meeting, the analysis of which will be published later this year. Also in this CAPRI Evaluation Round scorers were in most cases not able to select the best quality models from the scoring sets.

Next to the CAPRI community, which continues relentlessly to participate in this experiment despite the frustration of tackling such difficult targets, we wish to acknowledge here all the experimentalists providing structures to CAPRI prior to publication. Without those invaluable contributions over many years, the field would not be where it currently stands. We surely hope that the structural biology community at large will continue to provide targets, especially in the areas where the performance of current AI tools is still limited, like obviously the antibody and nanobody complexes, but also other types of complexes involving (flexible) peptides, glycans, and nucleic acids, in particular RNA complexes whose prediction still remains extremely challenging.

## Author Contributions

**Marc F. Lensink:** conceptualization, investigation, writing – original draft, methodology, validation, visualization, writing – review and editing, software, project administration, data curation, resources, supervision. **Nessim Raouraoua:** writing – review and editing, software, methodology. **Guillaume Brysbaert:** writing – review and editing, software, methodology. **Sameer Velankar:** writing – review and editing, resources. **Shoshana J. Wodak:** writing – review and editing, supervision, conceptualization, methodology, project administration, validation, investigation. **Alexandre M. J. J. Bonvin:** writing – review and editing, project administration, conceptualization, methodology, supervision, validation, investigation, writing – original draft.

## Acknowledgments

The French State under the France-2030 programme and the Initiative of Excellence of the University of Lille are acknowledged for the funding and support granted to the R-CDP-24-002-PIE project. We also acknowledge the University of Lille's Intensive Scientific Computing Mesocentre for providing access to computing resources. Alexandre M.J.J. Bonvin acknowledges financial support from Horizon Europe and from the European High Performance Computing Joint Undertaking, projects BioExcel (823830 and 101093290).

## Data Availability Statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

## Peer Review

The peer review history for this article is available at https://www.webofscience.com/api/gateway/wos/peer-review/10.1002/prot.70018.

## References

1. O. Carugo and K. Djinović-Carugo, "Structural Biology: A Golden Era," *PLoS Biology* 21 (2023): e3002187, https://doi.org/10.1371/journal.pbio.3002187.

2. G. Masrati, M. Landau, N. Ben-Tal, A. Lupas, M. Kosloff, and J. Kosinski, "Integrative Structural Biology in the Era of Accurate Structure Prediction," *Journal of Molecular Biology* 433 (2021): 167127, https://doi.org/10.1016/j.jmb.2021.167127.

3. S. J. Ziegler, S. J. B. Mallinson, P. C. S. John, and Y. J. Bomble, "Advances in Integrative Structural Biology: Towards Understanding Protein Complexes in Their Cellular Context," *Computational and Structural Biotechnology Journal* 19 (2021): 214–225, https://doi.org/10.1016/j.csbj.2020.11.052.

4. H. Schwalbe, P. Audergon, N. Haley, et al., "The Future of Integrated Structural Biology," *Struct. Lond. Engl* 2024, no. 32 (1993): 1563–1580, https://doi.org/10.1016/j.str.2024.08.014.

5. J. Janin, K. Henrick, J. Moult, et al., "CAPRI: A Critical Assessment of Predicted Interactions," *Proteins* 52 (2003): 2–9, https://doi.org/10.1002/prot.10381.

6. S. J. Wodak, S. Vajda, M. F. Lensink, D. Kozakov, and P. A. Bates, "Critical Assessment of Methods for Predicting the 3D Structure of Proteins and Protein Complexes," *Annual Review of Biophysics* 52 (2022): 183–206, https://doi.org/10.1146/annurev-biophys-102622-084607.

7. M. F. Lensink, R. Méndez, and S. J. Wodak, "Docking and Scoring Protein Complexes: CAPRI 3rd Edition," *Proteins* 69 (2007): 704–718, https://doi.org/10.1002/prot.21804.
8. M. F. Lensink and S. J. Wodak, "Docking and Scoring Protein Interactions: CAPRI 2009," *Proteins* 78 (2010): 3073–3084, https://doi.org/10.1002/prot.22818.

9. M. F. Lensink and S. J. Wodak, "Docking, Scoring, and Affinity Prediction in CAPRI," *Proteins* 81 (2013): 2082–2095, https://doi.org/10.1002/prot.24428.

10. M. F. Lensink, S. Velankar, and S. J. Wodak, "Modeling Protein-Protein and Protein-Peptide Complexes: CAPRI 6th Edition," *Proteins* 85 (2017): 359–377, https://doi.org/10.1002/prot.25215.

11. M. F. Lensink, N. Nadzirin, S. Velankar, and S. J. Wodak, "Modeling Protein-Protein, Protein-Peptide, and Protein-Oligosaccharide Complexes: CAPRI 7th Edition," *Proteins* 88 (2020): 916–938, https://doi.org/10.1002/prot.25870.

12. I. A. Vakser, "Protein-Protein Docking: From Interaction to Interactome," *Biophysical Journal* 107 (2014): 1785–1793, https://doi.org/10.1016/j.bpj.2014.08.033.

13. J. Jumper, R. Evans, A. Pritzel, et al., "Highly Accurate Protein Structure Prediction With AlphaFold," *Nature* 596 (2021): 583–589.

14. M. Baek, F. DiMaio, I. Anishchenko, et al., "Accurate Prediction of Protein Structures and Interactions Using a Three-Track Neural Network," *Science* 373 (2021): 871–876, https://doi.org/10.1126/science.abj8754.

15. R. Evans, M. O'Neill, A. Pritzel, et al., "Protein Complex Prediction With AlphaFold-Multimer."

16. G. Ahdritz, N. Bouatta, C. Floristean, et al., "OpenFold: Retraining AlphaFold2 Yields New Insights Into Its Learning Mechanisms and Capacity for Generalization," *Nature Methods* 21 (2024): 1514–1524, https://doi.org/10.1038/s41592-024-02272-z.

17. J. Abramson, J. Adler, J. Dunger, et al., "Accurate Structure Prediction of Biomolecular Interactions With AlphaFold 3," *Nature* 630 (2024): 493–500, https://doi.org/10.1038/s41586-024-07487-w.

18. J. Wohlwend, G. Corso, S. Passaro, et al., *Boltz-1: Democratizing Biomolecular Interaction Modeling* (bioRxiv 2024.11.19.624167bioRxiv, 2024).

19. C. Discovery, J. Boitreaud, J. Dent, et al., *Chai-1: Decoding the Molecular Interactions of Life* (bioRxiv 2024.10.10.615955, 2024).

20. N. Raouraoua, C. Mirabello, T. Véry, et al., "MassiveFold: Unveiling Alphafold's Hidden Potential With Optimized and Parallelized Massive Sampling," *Nature Computational Science* 4 (2024): 824–828, https://doi.org/10.1038/s43588-024-00714-4.

21. M. F. Lensink, G. Brysbaert, N. Raouraoua, et al., "Impact of AlphaFold on Structure Prediction of Protein Complexes: The CASP15-CAPRI Experiment," *Proteins* 91 (2023): 1658–1683, https://doi.org/10.1002/prot.26609.

22. M. F. Lensink, G. Brysbaert, T. Mauri, et al., "Prediction of Protein Assemblies, the Next Frontier: The CASP14-CAPRI Experiment," *Proteins* 89 (2021): 1800–1823, https://doi.org/10.1002/prot.26222.

23. A. Fioravanti, F. van Hauwermeiren, S. E. van der Verren, et al., "Structure of S-Layer Protein Sap Reveals a Mechanism for Therapeutic Intervention in Anthrax," *Nature Microbiology* 4 (2019): 1805–1814, https://doi.org/10.1038/s41564-019-0499-1.

24. G. Garcia-Rodriguez, D. Charlier, D. Wilmaerts, J. Michiels, and R. Loris, "Alternative Dimerization Is Required for Activity and Inhibition of the HEPN Ribonuclease RnI A," *Nucleic Acids Research* 49 (2021): 7164–7178, https://doi.org/10.1093/nar/gkab513.

25. J. M. Dunce, L. J. Salmon, and O. R. Davies, "Structural Basis of Meiotic Chromosome Synaptic Elongation Through Hierarchical Fibrous Assembly of SYCE2-TEX12," *Nature Structural & Molecular Biology* 28 (2021): 681–693, https://doi.org/10.1038/s41594-021-00636-z.

26. P. Kolata and R. G. Efremov, "Structure of Escherichia Coli Respiratory Complex I Reconstituted Into Lipid Nanodiscs Reveals an

Uncoupled Conformation," *eLife* 10 (2021): e68710, https://doi.org/10.7554/elife.68710.

27. A. V. Shkumatov, N. Aryanpour, C. A. Oger, G. Goossens, B. F. Hallet, and R. G. Efremov, "Structural Insight Into Tn3 Family Transposition Mechanism," *Nature Communications* 13 (2022): 6155, https://doi.org/10.1038/s41467-022-33871-z.

28. J. W. Beugelink, E. Sweep, B. J. C. Janssen, J. Snijder, and M. F. Pronker, "Structural Basis for Recognition of the FLAG-Tag by Anti-FLAG M2," *Journal of Molecular Biology* 436 (2024): 168649, https://doi.org/10.1016/j.jmb.2024.168649.

29. R. Méndez, R. Leplae, M. F. Lensink, and S. J. Wodak, "Assessment of CAPRI Predictions in Rounds 3-5 Shows Progress in Docking Procedures," *Proteins* 60 (2005): 150–169, https://doi.org/10.1002/prot.20551.

30. K. W. Collins, M. M. Copeland, G. Brysbaert, et al., "CAPRI-Q: The CAPRI Resource Evaluating the Quality of Predicted Structures of Protein Complexes," *Journal of Molecular Biology* 436 (2024): 168540, https://doi.org/10.1016/j.jmb.2024.168540.

31. C.-G. Wu, A. Zheng, L. Jiang, et al., "Methylation-Regulated Decommissioning of Multimeric PP2A Complexes," *Nature Communications* 8 (2017): 2272, https://doi.org/10.1038/s41467-017-02405-3.

32. M. Mirdita, K. Schütze, Y. Moriwaki, L. Heo, S. Ovchinnikov, and M. Steinegger, "ColabFold: Making Protein Folding Accessible to All," *Nature Methods* 19 (2022): 679–682, https://doi.org/10.1038/s41592-022-01488-1.

33. B. Ozden, A. Kryshtafovych, and E. Karaca, "Assessment of the CASP14 Assembly Predictions," *Proteins* 89 (2021): 1787–1799, https://doi.org/10.1002/prot.26199.

34. B. Savaş, İ. Yılmazbilek, A. Özsan, and E. Karaca, "Towards a Greener AlphaFold2 Protocol for Antibody-Antigen Modeling: Insights From CAPRI Round 55," *Proteins: Structure, Function, and Bioinformatics* (2025), https://doi.org/10.1002/prot.26820.

35. C. Christoffer, Y. Kagaya, J. Verburgt, et al., "Integrative Protein Assembly With LZerD and Deep Learning in CAPRI 47–55," *Proteins: Structure, Function, and Bioinformatics*, https://doi.org/10.1002/prot.26818.

36. R. Duan, X. Xu, L. Qiu, S. Zhang, and X. Zou, "Performance of Hybrid Strategies Combining MDockPP and AlphaFold2 in CAPRI Rounds 47–55," *Proteins: Structure, Function, and Bioinformatics*, https://doi.org/10.1002/prot.26805.

37. N. Raouraoua, M. F. Lensink, and G. Brysbaert, "Massive Sampling Strategy for Antibody-Antigen Targets in CAPRI Round 55 With MassiveFold," *Proteins: Structure, Function, and Bioinformatics*, https://doi.org/10.1002/prot.26802.

38. R. Gowthaman, M. Park, R. Yin, J. D. Guest, and B. G. Pierce, "AlphaFold and Docking Approaches for Antibody-Antigen and Other Targets: Insights From CAPRI Rounds 47–55," *Proteins: Structure, Function, and Bioinformatics*, https://doi.org/10.1002/prot.26801.