# CAPRI-Q: The CAPRI resource evaluating the quality of predicted structures of protein complexes

Keeley W. Collins, Matthew M. Copeland, Guillaume Brysbaert, Shoshana J. Wodak, Alexandre M.J.J. Bonvin, Petras J. Kundrotas, Ilya A. Vakser, Marc Lensink

<figure><img src="image_1.png" /></figure>

## To cite this version:

Keeley W. Collins, Matthew M. Copeland, Guillaume Brysbaert, Shoshana J. Wodak, Alexandre M.J.J. Bonvin, et al.. CAPRI-Q: The CAPRI resource evaluating the quality of predicted structures of protein complexes. Journal of Molecular Biology, 2024, Journal of Molecular Biology, 436 (17), pp.168540. 10.1016/j.jmb.2024.168540. hal-04981503

HAL Id: hal-04981503

https://lilloa.hal.science/hal-04981503v1

Submitted on 7 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

# CAPRI-Q: The CAPRI resource evaluating the quality of predicted structures of protein complexes

Keeley W. Collins¹, Matthew M. Copeland¹, Guillaume Brysbaert³, Shoshana J. Wodak⁴, Alexandre M. J. J. Bonvin⁵, Petras J. Kundrotas¹,*, Ilya A. Vakser¹,²,* and Marc F. Lensink³,*

1 - *Computational Biology Program, The University of Kansas, Lawrence, KS 66045, USA*

2 - *Department of Molecular Biology, The University of Kansas, Lawrence, KS 66045, USA*

3 - *Univ. Lille, CNRS, UMR 8576 - UGSF - Unité de Glycobiologie Structurale et Fonctionnelle, F-59000 Lille, France*

4 - *VIB-VUB Center for Structural Biology, Brussels, Belgium*

5 - *Bijvoet Centre for Biomolecular Research, Faculty of Science - Chemistry, Utrecht University, The Netherlands*

Correspondence to Petras J. Kundrotas, Ilya A. Vakser, Marc F. Lensink: pkundro@ku.edu (P.J. Kundrotas), vakser@ku.edu (I.A. Vakser), marc.lensink@univ-lille.fr (M.F. Lensink)
https://doi.org/10.1016/j.jmb.2024.168540

Edited by Rita Casadio

## Abstract

Protein interactions are essential for cellular processes. In recent years there has been significant progress in computational prediction of 3D structures of individual protein chains, with the best-performing algorithms reaching sub-Ångström accuracy. These techniques are now finding their way into the prediction of protein interactions, adding to the existing modeling approaches. The community-wide Critical Assessment of Predicted Interactions (CAPRI) has been a catalyst for the development of procedures for the structural modeling of protein assemblies by organizing blind prediction experiments. The predicted structures are assessed against unpublished experimentally determined structures using a set of metrics with proven robustness that have been established in the CAPRI community. In addition, several advanced benchmarking databases provide targets against which users can test docking and assembly modeling software. These include the Protein-Protein Docking Benchmark, the CAPRI Scoreset, and the DOCKGROUND database, all developed by members of the CAPRI community. Here we present CAPRI-Q, a stand-alone model quality assessment tool, which can be freely downloaded or used via a publicly available web server. This tool applies the CAPRI metrics to assess the quality of query structures against given target structures, along with other popular quality metrics such as DockQ, TM-score and /-DDT, and classifies the models according to the CAPRI model quality criteria. The tool can handle a variety of protein complex types including those involving peptides, nucleic acids, and oligosaccharides. The source code is freely available from https://gitlab.in2p3.fr/cmsb-public/CAPRI-Q and its web interface through the DOCKGROUND resource at https://dockground.compbio.ku.edu/assessment/.

© 2024 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

## Introduction

Protein interactions play a central role in most biological functions. Structural characterization of protein interactions is important for fundamental

biology, as well as for prevention and treatment of diseases. Experimental techniques for protein structure determination have provided a wealth of data over the past half century, as evidenced by the rapidly growing Protein Data Bank (PDB).¹
However, these are often time consuming and encounter difficulties on certain types of proteins.²,³ Experimentally determined protein structures are available for only a fraction of known proteins.⁴ Due to those limitations, computational approaches are essential and complementary to the experimental methods for structural studies of proteins and their complexes. Methods for predicting the structure of protein complexes from structures or sequences of the individual components (protein docking or modeling of protein assemblies) have improved over many years through integration of sophisticated energy functions, better sampling of the energy landscapes, and incorporation of existing knowledge on protein structures and interactions, including deep learning.⁵,⁶ Many protein docking and assembly modeling tools are publicly available, including a number of fully automatic servers. The number of these tools is growing as a result of the continuous methodology developments by the structure prediction community.

Adequate assessment of the quality of docking/modeling tools is a complicated task. Over the past decade there has been a sharp increase in the number of docking tools, with more than 60 available for commercial and academic use.⁷ This trend has accelerated with the introduction of powerful deep-learning approaches.⁸,⁹ Benchmarking new tools is often accompanied by comparison to a limited number of alternative tools, tested on a relatively small number of structures, typically randomly selected and lacking structural diversity.¹⁰ Furthermore, since many docking programs are regularly updated, studies comparing different docking techniques quickly become obsolete, creating demand for new evaluations.¹¹ The authors of new docking tools also tend to provide their own evaluations, often achieving better results than the average user, due to a deeper understanding of their own algorithm and of the associated tunable parameters.¹⁰,¹² They may also opt to focus on quality assessment approaches that are favorable to their tool. These reasons point to the need for a standard publicly available resource for the assessment of assembly modeling, which would offer objective assessment criteria agreed upon by the community.

The CAPRI experiment has been serving as the community-wide standard for docking and scoring evaluation since 2001. CAPRI satisfies the needs of the computational biology community by testing the performance of protein modeling procedures in blind prediction of three-dimensional structures of unpublished protein complexes. Prediction Rounds are launched on a rolling basis pending availability of suitable experimental protein complex structures. Participants register to predict the target structures of a Round. Each submitted model is assessed and ranked by the CAPRI assessors.¹³,¹⁴,⁹ The results of each Round are published on the CAPRI web site as soon as the

assessment is completed. However, several Rounds are typically grouped together into assessment periods, published in a CAPRI-dedicated journal issue. These journal issues describe current capabilities of the methods and define the state of the art in protein assembly modeling.

Here, to facilitate the development and comparison of protein docking/modeling techniques, we present CAPRI-Q, a tool implementing the prediction quality assessment used in the CAPRI experiment. CAPRI-Q is publicly available for download, as well as accessible through a web server. We show how the tool can be used to evaluate assembly models generated by different structure prediction servers including AlphaFold-multimer.¹⁵ The tool performs automatic sequence alignment, and we show that the quality assessment of the model structures is not significantly affected by missing residues and mutations in the protein sequence, evidencing robustness of the model classification.

# Methods

## Overview of the program

CAPRI-Q compares multiple models of a predicted protein-protein complex against a reference structure provided by the user (usually an experimentally determined structure of the complex). In addition to protein-protein complexes, CAPRI-Q handles complexes including peptides, nucleic acids, and oligosaccharides. Comparisons are performed on pairs of components, focusing on the binding interfaces between the interacting entities (two protein components or a protein component and another binding partner). For larger assemblies, distinct component pairs must be defined and evaluated independently, following the procedure described in Refs.⁶,¹⁶

The input files containing the coordinates of respectively, the models for the predicted structure and the reference structure (in PDB format), are filtered to remove hydrogen atoms, residues missing backbone atoms, and nonstandard residues. Equivalent larger and smaller components in the reference and predicted structures are designated as “receptor” and “ligand,” respectively. Each protein chain of the models of the predicted complex is matched to the chains of the reference structure using the EMBOSS Needleman-Wunsch sequence alignment.¹⁷ The residues contributing to the binding interface are defined, enabling the subsequent calculations of a set of parameters (metrics) evaluating the quality of the predicted models.

The main output file is a simple tab-delimited ASCII text file listing the ID of each model of the predicted complex and all the parameters for that model. Additional output files provide more
detailed information, for instance on the sequence alignments of the predicted and reference complexes, and the native residue-residue contacts at the binding interface.

## Superposition-dependent CAPRI metrics

For each model, CAPRI-Q evaluates the standard CAPRI quality metrics illustrated in Figure 1. A common metric is the root-mean-square deviation (RMSD) between the equivalent atoms in the optimally aligned model and the reference (native) structure. CAPRI-Q outputs several RMSD-based metrics, computed over specific subsets of atoms (Figure 1B): the L-RMSD metric, defined as the RMSD of the ligand atoms after optimally aligning the receptor moiety, and the *i*-RMSD parameter representing the RMSD computed on subsets of the native interface atoms (using a 10 Å interactomic distance cutoff), of both receptor and ligand (Figure 1B). Two distinct *i*-RMSD metrics are computed, one considering only the backbone atoms of the interface backbone atoms (*i*-RMSDbb) and the other considering only the

side-chain atoms of the interface residues (*i*-RMSDsc).

The L-RMSD may be misleading for large ligands which may align well at the native interface (and thus have small *i*-RMSD values) but display significant structural differences away from the interface. In CAPRI, the definition of the overall quality of the model accounts for this by relying on either L-RMSD or *i*-RMSD, depending on which score is better¹³ as will be presented below.

To help rationalize the match between the predicted and reference structures, two additional parameters are computed, describing the overall shift of the ligand moiety relative to the receptor in models versus the reference structure. These are the misorientation angle $\theta_L$ of the ligand moieties and the residual displacement $d_L$ of the ligand geometric center, computed after the receptors in the model and reference structures have been optimally superimposed¹⁸ (Figure 1B), but these parameters are not considered in defining model quality.

In addition to the RMSD-based metrics described above, CAPRI-Q computes the TM-score,¹⁹ representing a superposition-dependent metric normalized by the protein length. The TM-score is

<figure><img src="image_4.png" /><figcaption>**Figure 1.** Schematic illustration of the CAPRI model quality metrics. (A) Shows the quantities evaluated for the reference structure of a protein complex consisting of two protein components. These are: the residue-residue contacts between the receptor (R) and the ligand (L) (see text for definition), computed using a distance threshold of $d \le 5$ Å between atoms; and the residues of the ligand and receptor, respectively, contributing to the binding interface, using a more lenient threshold of $d \le 10$ Å for the interacting atoms between both subunits. (B) Illustrates the model quality metrics computed for each predicted model of the complex. These are, from left to right, the fractions of native ($f_{nat}$) and non-native ($f_{non-nat}$) residue-residue contacts in the predicted interface (see text for definition); the root mean square displacement (RMSD) of the backbone atoms of the ligand (L-RMSD), the mis-orientation angle $\theta_L$ and the residual displacement $d_L$ of the ligand center of mass, after the receptor in the model and experimental structures were optimally superimposed; the RMSD of the backbone atoms of all interface residues (*i*-RMSDbb) after only these residues have been optimally superimposed. Additionally, we also compute the RMSD of interface sidechain atoms (*i*-RMSDsc) (not shown).</figcaption></figure>
calculated using the distances *d*<sub>i</sub> between equiva-
lent residue pairs in the two optimally aligned struc-
tures.¹⁹ This score is computed using MM-align,²⁰
specifically designed for the alignment of multi-
chain structures (hereafter referred to as MM-
score). The MM-score provides the simultaneous
evaluation of the quality of the tertiary structure of
the protein moieties and their quaternary arrange-
ment. However, the absolute value of the MM-
score should be interpreted considering the ratio
between the numbers of residues in the two inter-
acting proteins. For example, the MM-score
between two complexes featuring closely similar
tertiary structures but different quaternary arrange-
ment can be artificially inflated (higher values indi-
cating a better match) if the receptor is
significantly larger than the ligand (even in the ideal
case in which all residues of the reference structure
are present in the model structure and vice versa).

Superposition-independent CAPRI metrics

Metrics requiring structural superposition of two
proteins are highly sensitive to how the
superposition or structural alignment is performed,
and to local structural distortions of the model or
reference structure.²¹ It is therefore useful to also
consider quality metrics not based on structural
superposition. CAPRI-Q computes three such met-
rics used by CAPRI to define model quality.⁹,²²
These are the fraction of correctly predicted ‘native’
ligand-receptor contacts fₘₐτ (with ‘native’ referring
to the reference structure), and the fraction of incor-
rectly predicted ligand-receptor contacts (contacts
not found in the reference structure) fᵣₐₙₐₙₐ

<figure><img src="image_5.png" /><figcaption>**Figure 2.** An illustration of different types of contacts contributing to the *f*<sub>nat</sub> and *f*<sub>non-nat</sub> parameters. The reference structure is shown on the top and the model is on the bottom. The receptor is in yellow and the ligand is in gray. **A** is an incorrect contact in the model that does not exist in the reference structure, which increases the *f*<sub>non-nat</sub> parameter. **B1** and **B2** are contacts in both the reference and the model structures, which increase *f*<sub>nat</sub>. **C** is a contact in the native structure not present in the model structure, which would decrease *f*<sub>nat</sub>.</figcaption></figure>

(Figure 2). These contacts are computed for inter-
face residues, defined as those having a heavy
atom within 5 Å away from any heavy atom of the
residues on the opposing side of the interface.

It should be noted that *f*<sub>nat</sub> can be deceptively
large for a model featuring many steric clashes
between the receptor and the ligand due do a
significant interpenetration of the two interacting
components. Such obviously incorrect models are
identified by computing the number of clashes
between the ligand and the receptor. When this
number exceeds a given limit, the clash threshold,
the model is flagged by adding the word “clashes”
to the model’s CAPRI classification. By default,
the clash threshold is set to the number of
interface residues.

In addition to the interface residues contact
fractions, CAPRI-Q also computes the local
distance difference test, *l-DDT*,<sup>23</sup> which evaluates
the conservation of the local distances in the model
compared to the reference structure. To calculate it,
all the distances between heavy atoms in the refer-
ence structure within a sphere of 15 Å (excluding
distances within the same residue) are computed
(Figure S1). Then, the distances between all the
matching atom pairs in the model structure are
determined irrespective of their spatial separation.
Finally, the corresponding distances in the refer-
ence and the model structures are compared and
the fraction of conserved distances is calculated.
A distance is considered conserved if it changes
less than a given threshold. Following the original
l-DDT paper,<sup>23</sup> we use four different thresholds,
4 Å, 2 Å, 1 Å, and 0.5 Å. The reported l-DDT score
is the average of the scores with these thresholds.

CAPRI overall model quality

In order to define the quality of the model overall,
CAPRI classifies predicted structures onto four
quality categories, based on the values of f<sub>nat</sub>, L-
RMSD, and i-RMSDbb<sup>24</sup>:

(i) high quality, ($f_{\text{nat}} \ge 0.5$) and either ($L\text{-RMSD} \le 1.0\AA$) or ($i\text{-RMSDbb} \le 1.0\AA$);

(ii) medium quality, ($f_{\text{nat}} \ge 0.3$) and either ($L\text{-RMSD} \le 5\AA$) or ($i\text{-RMSDbb} \le 2.0\AA$);

(iii) acceptable quality, ($f_{\text{nat}} \ge 0.1$) and either ($L\text{-RMSD} \le 10\AA$) or ($i\text{-RMSDbb} \le 4\AA$); and

(iv) incorrect, ($f_{\text{nat}} < 0.1$) or ($L\text{-RMSD} > 10.0\AA$) and ($i\text{-RMSDbb} > 4.0\AA$).

These criteria are applied in reverse order.

Model quality based on the DockQ score

CAPRI-Q also computes the DockQ score,²⁵ a
continuous quality score that integrates the main
quality metrics of the standard CAPRI evaluation
protocol.²⁴ DockQ is defined as the average of the
*f*<sub>nat</sub>, and scaled *L-RMSD* and *i-RMSDbb* metrics:
$$ \text{DockQ} = \frac{1}{3} \left( f_{\text{nat}} + \frac{1}{1 + \left( \frac{L-RMSD}{d_1} \right)^2} + \frac{1}{1 + \left( \frac{i-RMSD_{\text{bb}}}{d_2} \right)^2} \right), $$

where $d_1$ and $d_2$ are scaling parameters which determine how fast large L-RMSD and i-RMSD$_{bb}$ values, respectively, can be scaled to zero.²⁵ The values of these parameters ($d_1 = 8.5$ Å and $d_2 = 1.5$ Å) were obtained²⁵ by maximizing classification performance on the MOAL-set.²⁶ The range of the DockQ score is 0 to 1. Generally, models with DockQ < 0.23 are considered incorrect. High, medium and acceptable accuracy models tend to have DockQ ≥ 0.80, 0.49 < DockQ < 0.80, and 0.23 ≤ DockQ < 0.49, respectively.²⁵ It is important to keep in mind that DockQ considers both L-RMSD and i-RMSD$_{bb}$ simultaneously, whereas CAPRI classification assignments are based on either L-RMSD or i-RMSD$_{bb}$. Therefore, the model quality categories defined using DockQ may not exactly correspond to those obtained by the CAPRI protocol.

## Additional information and metrics

For the convenience of users, CAPRI-Q generates a “Summary” file containing supplementary information used in calculations of the above metrics. For the receptor and the ligand of the models and the reference structure, the Summary lists residues contributing to the binding interface of the reference structure, mapping of the chains between the model and the reference structures, solvent accessible surface area (SASA), and sequence identities (percentage of identical residues), seqID. The seqID is also used to flag models generated with sequences significantly deviating (in length or composition) from those in the reference structures (the default threshold used requires 70% sequence overlap with the target). These models marked as having “low_id”, and models flagged with “clashes”, are evaluated here, but not considered in the standard CAPRI evaluation.

Additional metrics reported in the Summary are the interface area (IA), fractions of ‘native’ (correctly predicted) ($F_{IR}$) and ‘non-native’ (incorrectly predicted, or overpredicted) ($F_{OP}$) interface residues, the RMSD of individual components (monomers) (m-RMSD), and the DockQ score.

IA is calculated as

$$ IA = \text{SASA}(R) + \text{SASA}(L) - \text{SASA}(RL), $$

where SASA(R), SASA(L) and SASA(RL) are solvent accessible surface areas of the receptor-only, ligand-only and the receptor-ligand complex respectively. SASA is calculated using the C library from the FreeSASA software.²⁷ The interface residues used to compute $F_{IR}$ and $F_{OP}$ are

defined as those losing SASA upon binding. $F_{IR}$ and $F_{OP}$ are computed separately for the receptor and the ligand of the model structure. m-RMSD is likewise evaluated separately for the ligand and receptor, based on C° RMSD between the model and the reference.

## Case studies

### Comparing different protein docking methodologies

CAPRI-Q functionality is illustrated on models produced by docking calculations performed for three dimers from the DOCKGROUND Docking Benchmark 1 (1B6C, 1GPW, 1K93)²⁸ and the trimeric CAPRI target T50 (CAPRI round 24 from February-March 2011, PDB code 3R2X).²⁹ The PDB codes refer to the experimentally determined structures of the complexes (i.e., bound structures), whereas in docking we used the experimentally determined structures of the unbound proteins: 1BKF (0.6 Å) and 1VJY (1.5 Å) for 1B6C; 1thf (3.6 Å) and 1K9V (0.7 Å) for 1GPW; 1K8T (10.0 Å) and 1CLL (9.0 Å) for 1K93; and 3GBN (0.6 Å) and 1U84 (0.6 Å) for 3R2X. The numbers in parentheses are the RMSD between the unbound and the bound structures; according to these values, 1B6C and 3R2X are easy (rigid-body) targets while 1GPW and 1K93 are medium difficulty and hard targets, respectively.³⁰

The calculations were performed using the HADDOCK³¹ and HDOCK³² docking servers and a local install of AlphaFold-Multimer¹⁵ version 2.3 with default parameters and databases. HADDOCK performs information-driven docking, clusters the models and reports the top four models for each cluster. The docking was performed using the “Easy” interface option of the server, with the interface residues specified. HDOCK performs hybrid free and template-based docking using a distance-dependent knowledge-based scoring function.³³ We also used its option to specify interface residues. All generated HADDOCK and AlphaFold-Multimer models, and the top 50 HDOCK models were evaluated by CAPRI-Q. The output of CAPRI-Q for 1GPW is summarized in Table 1. The best scoring model for each complex for each of the three modeling servers is shown in Supplementary Figure S2.

### Effect of missing residues

The CAPRI-Q model quality classification is robust, even when some residues are missing from the model structure. The effect of missing residues based on medium-quality models generated by AlphaFold-Multimer for the 1B6C target is shown in Supplementary Table S1. For
<table><caption>Table 1 CAPRI-Q sample output for case study 1GPW (see text). Three models with a range of quality classifications from each modeling method are shown in Table 1. The column for sequence identity was removed to reduce the size of the table (all were above 90%). From the Table it can be deduced that the CAPRI classification expectedly correlates with the metrics, as it improves with a higher $f_{nat}$, and lower L-RMSD or i-RMSDbb scores. Interestingly, the AlphaFold model with the highest $f_{nat}$ value is of medium rather than high quality, due to increased L-RMSD and i-RMSDbb values.</caption><thead><tr><th>Modeling tool</th><th>Number of clashes</th><th>f<sub>nat</sub></th><th>f<sub>non-nat</sub></th><th>TM-score</th><th>LDDT</th><th>L-RMSD</th><th>i-RMSDbb</th><th>i-RMSDsc</th><th>DockQ</th><th>Distance</th><th>Theta</th><th>Classification</th></tr></thead><tbody><tr><td>HADDOCK</td><td>8</td><td>0.12</td><td>0.89</td><td>0.72</td><td>0.80</td><td>16.86</td><td>7.06</td><td>7.64</td><td>0.12</td><td>10.76</td><td>57.44</td><td>incorrect</td></tr><tr><td>HADDOCK</td><td>21</td><td>0.75</td><td>0.61</td><td>0.90</td><td>0.81</td><td>5.55</td><td>1.93</td><td>2.46</td><td>0.61</td><td>4.30</td><td>16.12</td><td>medium</td></tr><tr><td>HADDOCK</td><td>26</td><td>0.71</td><td>0.60</td><td>0.90</td><td>0.82</td><td>5.35</td><td>1.95</td><td>2.44</td><td>0.60</td><td>4.01</td><td>16.33</td><td>medium</td></tr><tr><td>HDock</td><td>32</td><td>0.93</td><td>0.32</td><td>0.97</td><td>0.85</td><td>0.88</td><td>0.67</td><td>1.62</td><td>0.92</td><td>0.34</td><td>0.34</td><td>high</td></tr><tr><td>HDock</td><td>29</td><td>0.38</td><td>0.61</td><td>0.86</td><td>0.82</td><td>6.37</td><td>2.67</td><td>3.02</td><td>0.42</td><td>3.39</td><td>27.23</td><td>acceptable</td></tr><tr><td>HDock</td><td>28</td><td>0.07</td><td>0.94</td><td>0.67</td><td>0.80</td><td>13.60</td><td>7.32</td><td>7.90</td><td>0.13</td><td>2.55</td><td>65.35</td><td>incorrect</td></tr><tr><td>AlphaFold</td><td>17</td><td>0.90</td><td>0.31</td><td>0.98</td><td>0.89</td><td>2.68</td><td>1.04</td><td>1.66</td><td>0.83</td><td>2.34</td><td>6.23</td><td>medium</td></tr><tr><td>AlphaFold</td><td>18</td><td>0.87</td><td>0.30</td><td>0.98</td><td>0.89</td><td>2.43</td><td>0.99</td><td>1.63</td><td>0.83</td><td>2.04</td><td>5.46</td><td>high</td></tr><tr><td>AlphaFold</td><td>15</td><td>0.88</td><td>0.27</td><td>0.97</td><td>0.89</td><td>1.51</td><td>0.78</td><td>1.55</td><td>0.88</td><td>1.25</td><td>3.07</td><td>high</td></tr></tbody></table>

this case study, we removed from the initial model 5, 10, 15, or 30 randomly selected ligand residues from regions outside the interface in the model structure. We also removed from the initial model the same numbers of residues randomly selected from the interface. For the models with non-interface residues removed, none of the metrics other than the sequence identity were significantly affected. For the models with removed interface residues, the fraction of native contacts slightly decreased and the interface RMSDs slightly increased. However, these changes marginally affected the overall model quality classification. For the purpose of ranking models in CAPRI, a sequence alignment step ensures that all RMSD calculations are performed on the same number of residues. However, in our example, each model was evaluated separately and RMSD calculations were therefore not necessarily performed over the same amount of residues. Nonetheless, this did not affect the overall classification of the models.

## Effect of mutations on model assessment

For this case study, we produced three types of mutated structures by replacing either 5 or 20 randomly selected ligand residues in the native structure of the 3R2X target to alanine, aspartic acid, or proline. Mutated residues were all picked either outside the interface or within the interface region of the ligand. Structural models of the mutated proteins were generated using the NEST program from the JACKAL suite.³⁴ In this case, NEST mostly performed side chain re-packing, with the backbone almost unaffected (the backbone atoms may only be slightly shifted during structure optimization to remove steric clashes). This becomes apparent when residues that form helices mutate into helix-breaking prolines, since the helices in the 3R2X ligand remained intact after the residue replacements. Table S2 lists the CAPRI-Q metrics of the original and the mutated structures. The native sequence of the 3R2X PDB structure was also run through NEST, resulting in a slight displacement of the original atoms, leading to poorer values of the metrics for the original structure (Table S2).

For the ALA mutations, increasing the number of mutations outside the interface from 5 to 20 had little impact on the metric values. However, increasing the number of such mutations at the interface increased the value of *i-RMSDbb* and decreased that of *f*<sub>nat</sub>, albeit still by a relatively small amount. Nevertheless, none of these changes were sufficient to alter the classification of the model. For the mutations outside the interface, the type of mutation did not significantly affect any metric. For the interface residues, mutations to ALA had the least impact, whereas mutations to PRO had the most impact. Still, none of the metric values changed enough to affect the model classification. Indeed, as long as the
backbone of the model is maintained, mutations do not greatly affect the assessment of the predicted models.

## Availability of the tool

### Source code

The source code can be freely downloaded at
https://gitlab.in2p3.fr/cmsb-public/CAPRI-Q.

CAPRI-Q is written in C, except for the included source code for MM-score calculations which is written in C++. The FreeSASA library is included. However, the EMBOSS library needs to be downloaded and installed by the user. It is simple to compile and is called from within the CAPRI-Q program. Downloading the source code, as opposed to using the web server, enables users to run many models against a reference structure, and gives more control over the scoring parameters, such as the number of acceptable clashes. A Dockerfile and instructions to create a Docker or Podman image are also provided, as well as a ready-to-use Docker image.

### Web interface

CAPRI-Q is implemented as part of the DOCKGROUND web server at https://dockground.compbio.ku.edu/assessment/. Users can submit up to ten models per job to evaluate against a reference structure. The submission page is shown in Figure S3A. Users are emailed a link to the results once their job has completed. The example results page is shown in Figure S3B. The results can be downloaded as a text file in a tab-delimited format.

## Concluding remarks

CAPRI-Q is a stand-alone tool implementing the community-established standard for the quality assessment of protein assembly models. It is an enhanced version of the standard model evaluation procedure in CAPRI, a long-standing community-wide blind prediction experiment, as it also incorporates three complementary metrics: DockQ, a continuous score combining the CAPRI model quality metrics, and the MM-score and I-DDT, two commonly used model quality metrics for evaluating predicted protein structures.³⁵ CAPRI-Q is robust with respect to imperfections of the submitted model structures. The tool can be used either off-line as a command-line standalone program or on-line through the Web interface. The source code is freely available for download at https://gitlab.in2p3.fr/cmsb-public/CAPRI-Q, which also provides instructions to build or download a Docker image. The web interface is part of the DOCKGROUND resource at https://dockground.compbio.ku.edu/assessment/.

## CRediT authorship contribution statement

**Keeley W. Collins:** Writing – review & editing, Writing – original draft, Software, Methodology, Investigation, Conceptualization. **Matthew M. Copeland:** Writing – review & editing, Software, Resources, Methodology. **Guillaume Brysbaert:** Writing – review & editing, Software, Resources, Methodology. **Shoshana J. Wodak:** Writing – review & editing, Supervision, Methodology, Conceptualization. **Alexandre M.J.J. Bonvin:** Writing – review & editing. **Petras J. Kundrotas:** Writing – review & editing, Writing – original draft, Supervision, Methodology, Investigation. **Ilya A. Vakser:** Writing – review & editing, Writing – original draft, Supervision, Resources, Project administration, Funding acquisition. **Marc F. Lensink:** Writing – review & editing, Writing – original draft, Supervision, Software, Resources, Project administration, Methodology, Conceptualization.

## DECLARATION OF COMPETING INTEREST

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgements

This study was supported by NIH grant R01GM074255 and NSF grant DBI 2224122.

## Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.jmb.2024.168540.

Received 1 February 2024;

Accepted 11 March 2024;

Available online 16 March 2024

**Keywords:**

protein docking;

CAPRI;

quality assessment;

blind prediction;

protein modeling

## References

1. Bittrich, S., Bhikadiya, C., Bi, C., Chao, H., Duarte, J.M., Dutta, S., et al., (2023). RCSB protein data bank: efficient searching and simultaneous access to one million computed structure models alongside the PDB structures
enabled by architectural advances. *J. Mol. Biol.* **435**,
167994

2. Dokholyan, N.V., (2020). Experimentally-driven protein structure modeling. *J. Proteom.* **220**, 103777

3. Guzenko, D., Lafita, A., Monastyrskyy, B., Kryshtafovych, A., Duarte, J.M., (2019). Assessment of protein assembly prediction in CASP13. *Proteins* **87**, 1190–1199.

4. Aloy, P., Pichaud, M., Russell, R.B., (2005). Protein complexes: structure prediction challenges for the 21st century. *Curr. Opin. Struct. Biol.* **15**, 15–22.

5. Vakser, I.A., (2014). Protein-protein docking: from interaction to interactome. *Biophys. J.* **107**, 1785–1793.

6. Lensink, M.F., Brysbaert, G., Raouraoua, N., Bates, P.A., Giulini, M., Honorato, R.V., et al., (2023). Impact of AlphaFold on structure prediction of protein complexes: The CASP15-CAPRI experiment. *Proteins* **91**, 1658–1683.

7. Pagadala, N.S., Syed, K., Tuszynski, J., (2017). Software for molecular docking: a review. *Biophys. Rev.* **9**, 91–102.

8. Versini, R., Sritharan, S., Aykac Fas, B., Tubiana, T., Aimeur, S.Z., Henri, J., et al., (2023). A perspective on the prospective use of AI in protein structure prediction. *J. Chem. Inf. Model.*.

9. Wodak, S.J., Vajda, S., Lensink, M.F., Kozakov, D., Bates, P.A., (2023). Critical assessment of methods for predicting the 3D structure of proteins and protein complexes. *Annu. Rev. Biophys.* **52**, 183–206.

10. Plewczynski, D., Lazniewski, M., Augustyniak, R., Ginalski, K., (2011). Can we trust docking results? Evaluation of seven commonly used programs on PDBbind database. *J. Comput. Chem.* **32**, 742–755.

11. Wang, Z., Sun, H., Yao, X., Li, D., Xu, L., Li, Y., et al., (2016). Comprehensive evaluation of ten docking programs on a diverse set of protein-ligand complexes: the prediction accuracy of sampling power and scoring power. *PCCP* **18**, 12964–12975.

12. Chen, H., Lyne, P.D., Giordanetto, F., Lovell, T., Li, J., (2006). On evaluating molecular-docking methods for pose prediction and enrichment factors. *J. Chem. Inf. Model.* **46**, 401–415.

13. Lensink, M.F., Nadzirin, N., Velankar, S., Wodak, S.J., (2020). Modeling protein-protein, protein-peptide, and protein-oligosaccharide complexes: CAPRI 7th edition. *Proteins* **88**, 916–938.

14. Janin, J., Henrick, K., Moult, J., Eyck, L.T., Sternberg, M.J., Vajda, S., et al., (2003). CAPRI: a critical assessment of PRedicted interactions. *Proteins* **52**, 2–9.

15. Evans, R., O'Neill, M., Pritzel, A., Antropova, N., Senior, A., Green, T., et al., (2022). Protein complex prediction with AlphaFold-Multimer. *bioRxiv.* 2021.10.04.463034.

16. Lensink, M.F., Brysbaert, G., Mauri, T., Nadzirin, N., Velankar, S., Chaleil, R.A.G., et al., (2021). Prediction of protein assemblies, the next frontier: the CASP14-CAPRI experiment. *Proteins* **89**, 1800–1823.

17. Madeira, F., Pearce, M., Tivey, A.R.N., Basutkar, P., Lee, J., Edbali, O., et al., (2022). Search and sequence analysis tools services from EMBL-EBI in 2022. *Nucleic Acids Res.* **50**, W276–W279.

18. McLachlan, A., (1982). Rapid comparison of protein structures. *Acta Crystallograph. Section A* **38**, 871–873.

19. Zhang, Y., Skolnick, J., (2004). Scoring function for automated assessment of protein structure template quality. *Proteins* **57**, 702–710.

20. Mukherjee, S., Zhang, Y., (2009). MM-align: a quick algorithm for aligning multiple-chain protein complex structures using iterative dynamic programming. *Nucleic Acids Res.* **37**, e83.

21. Kufareva, I., Abagyan, R., (2012). Methods of protein structure comparison. *Methods Mol. Biol.* **857**, 231–257.

22. Mendez, R., Leplae, R., Lensink, M.F., Wodak, S.J., (2005). Assessment of CAPRI predictions in rounds 3–5 shows progress in docking procedures. *Proteins* **60**, 150–169.

23. Mariani, V., Biasini, M., Barbato, A., Schwede, T., (2013). IDDT: a local superposition-free score for comparing protein structures and models using distance difference tests. *Bioinformatics* **29**, 2722–2728.

24. Lensink, M.F., Mendez, R., Wodak, S.J., (2007). Docking and scoring protein complexes: CAPRI 3rd Edition. *Proteins* **69**, 704–718.

25. Basu, S., Wallner, B., (2016). DockQ: a quality measure for protein-protein docking models. *PLoS One* **11**, e0161879.

26. Moal, I.H., Torchala, M., Bates, P.A., Fernandez-Recio, J., (2013). The scoring of poses in protein-protein docking: current capabilities and future directions. *BMC Bioinf.* **14**, 286.

27. Mitternacht, S., (2016). FreeSASA: an open source C library for solvent accessible surface area calculations. *F1000Res* **5**, 189.

28. Gao, Y., Douguet, D., Tovchigrechko, A., Vakser, I.A., (2007). DOCKGROUND system of databases for protein recognition studies: unbound structures for docking. *Proteins* **69**, 845–851.

29. Lensink, M.F., Wodak, S.J., (2013). Docking, scoring, and affinity prediction in CAPRI. *Proteins* **81**, 2082–2095.

30. Mintseris, J., Wiehe, K., Pierce, B., Anderson, R., Chen, R., Janin, J., et al., (2005). Protein-protein docking benchmark 2.0: an update. *Proteins* **60**, 214–216.

31. van Zundert, G.C.P., Rodrigues, J., Trellet, M., Schmitz, C., Kastritis, P.L., Karaca, E., et al., (2016). The HADDOCK2.2 web server: user-friendly integrative modeling of biomolecular complexes. *J. Mol. Biol.* **428**, 720–725.

32. Yan, Y., Tao, H., He, J., Huang, S.Y., (2020). The HDOCK server for integrated protein-protein docking. *Nature Protoc.* **15**, 1829–1852.

33. Huang, S.Y., Zou, X., (2008). An iterative knowledge-based scoring function for protein-protein recognition. *Proteins* **72**, 557–579.

34. Xiang, Z., Honig, B., (2001). Extending the accuracy limits of prediction for side-chain conformations. *J. Mol. Biol.* **311**, 421–430.

35. Kryshtafovych, A., Schwede, T., Topf, M., Fidelis, K., Moult, J., (2023). Critical assessment of methods of protein structure prediction (CASP)-Round XV. *Proteins* **91**, 1539–1549.