Article Journal of Medicinal Chemistry, 2009, Vol. 52, No. 18 5717 Figure 6. Ligands from the Astex-85 benchmark that do not achieve good pose fidelity under any circumstances. Figure 7. Histogram and cumulative frequency of pose fidelity (A˚ , rmsd) using the best parameter set. Astex-85 benchmark. Figure 8. Plot of pose fidelity (A˚ , rmsd) versus % rank compared to about 100 property matched decoys (log scale). For Astex-85 benchmark. In each case, the parameter set that gave the best pose fidelity was used. code was also specified. Of the 94 calculations that started automatically, all but two yielded a docked pose and score for the ligand. Of the 20 targets that failed to produce a docked ligand, three failed during ligand identification and separation, five failed to prepare the ligand for docking, and 12 tried but failed to dock and score the ligand. We asked how well automatic docking could recapitulate the crystal structure when the docking protocol could not be modified. Of the 92 that yielded a docked and scored ligand, 58 posed within 2 A˚ rmsd of the crystal structure (Supporting Information Table S2 and http://data.docking.org/2009/ GOLDTables.doc). All parameter sets performed nearly equivalently, and no single one was as successful as combining the best result from all four. When the docked pose was within 2 A˚ rmsd, 27 of these also managed to rank in the top 5% of about 100 property matched decoys (Table 1 and http://data. docking.org/2009/GOLDTables.doc). DUD Benchmarking. Worrying about depending on a single positive control as the sole basis for judging docking performance, we turned to a benchmark for which more actives were available. We used DOCK Blaster to dock to targets in the DUD-38 benchmark27 for which the ligand pose was crystallographically observed. All targets could be handled by the parser and submitted to the DOCK Blaster queue. One ligand failed to be converted to SMILES auto- matically (1CKP), but the rest of the 37 targets completed normally yielding a redocked and scored ligand. We began by assessing performance as before using a single crystallographic ligand and its automatically gener- ated decoys, asking how well automatic docking could recapitulate the crystal structure. Of the 37 protein-ligand pairs that yielded a docked ligand score, 23 posed within 2 A˚ rmsd of the crystallographic ligand (Supporting Information Table S2 and http://data.docking.org/2009/DUDTables. doc). When the docked pose was within 2 A˚ rmsd, 15 of these also ranked the ligand in the top 5% of about 100 property matched decoys (Table 1 and http://data.docking. org/2009/DUDTables.doc). As part of DUD, these targets benefited from between 20 and 600 additional annotated actives and their correspond- ing property-matched decoys with which to test DOCK Blaster. We began by checking that the fully automatic docking results using DOCK Blaster were at least compar- able to the expert docking reported in the DUD paper (Figure 9 and Supporting Information Table S3). If auto- matic docking were consistently different from an expert, the conclusions of this study could be undermined. For instance, an expert can experiment with the placement of specific water molecules, modification of receptor parameters, or docking parameters and do so repeatedly until the results converge. Whereas fully automated docking often produced results somewhat inferior to an expert, we thought it was close in all but five cases. A critical question was how predictive the single molecule metric used thus far was of the multimolecule DUD metric (Table 3 and http://data.docking.org/2009/). For simplicity we rated each metric as “good” or “bad” to indicate whether the docking results looked compelling enough to use for a discovery project. For the single ligand metric, “good” meant both pose fidelity within 2 A˚ rmsd and rank strictly in the top 5% of the property matched decoys. For the multiligand (DUD) metric, “good” meant that either the adjusted logAUC53 was greater than 10, or that EF1, the