Package: disteval
DISTEVAL: Protein distance evaluationProject abstractBackground:Protein inter-residue contact and distance prediction are two key intermediate steps essential to accurate protein structure prediction. Distance prediction comes in two forms: real-valued distances and 'binned' distograms, which are a more finely grained variant of the binary contact prediction problem. The latter has been introduced as a new challenge in the 14thCritical Assessment of Techniques for Protein Structure Prediction (CASP14) 2020 experiment. Despite the recent proliferation of methods for predicting distances, few methods exist for evaluating these predictions.  Currently only numerical metrics, which evaluate the entire prediction at once, are used.  These give no insight into the structural details of a prediction. For this reason, new methods and tools are needed.Results:We have developed a web server for evaluating predicted inter-residue distances. Our server, DISTEVAL, accepts predicted contacts, distances, and a true structure as optional inputs to generate informative heatmaps, chord diagrams, and 3D models. All of these outputs facilitate visual and qualitative assessment. The server also evaluates predictions using other metrics such as mean absolute error, root mean squared error, and contact precision.Conclusions:The visualizations generated by DISTEVAL complement each other and collectively serve as a powerful tool for both quantitative and qualitative assessments of predicted contacts and distances, even in the absence of a true 3D structure.Webserverhttp://deep.cs.umsl.edu/disteval/Distance/contact evaluation usingdisteval.pyDownloadDownload fromhttps://github.com/ba-lab/disteval/releasesPrerequisitesPython3NumpyScikit-learnInstallation from PIPpipinstalldistevalTestExample 0. See helpdisteval-hDownload the test files fromhttps://github.com/ba-lab/disteval/blob/main/test/Example 1. Evaluate a predicted RR contacts filedisteval-n./test/1guuA.pdb-c./test/1guuA.contact.rrExpected output:Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 1.0, 'count': 9}
min-seq-sep: 12 xL: Top-L   {'precision': 1.0, 'count': 9}
min-seq-sep: 12 xL: Top-NC  {'precision': 1.0, 'count': 9}
min-seq-sep: 24 xL: Top-L/5 {'precision': 1.0, 'count': 1}
min-seq-sep: 24 xL: Top-L   {'precision': 1.0, 'count': 1}
min-seq-sep: 24 xL: Top-NC  {'precision': 1.0, 'count': 1}Example 2. Evaluate a predicted distance mapdisteval-n./test/1guuA.pdb-d./test/1guuA.predicted.npyExpected output:Evaluating distances..
min-seq-sep: 12 xL: Top-L/5 {'mae': 0.9403, 'mse': 1.5143, 'rmse': 1.2306, 'count': 10}
min-seq-sep: 12 xL: Top-L   {'mae': 1.7522, 'mse': 5.6841, 'rmse': 2.3841, 'count': 50}
min-seq-sep: 12 xL: Top-NC  {'mae': 1.9263, 'mse': 6.6872, 'rmse': 2.586, 'count': 603}
min-seq-sep: 24 xL: Top-L/5 {'mae': 1.8154, 'mse': 4.6469, 'rmse': 2.1557, 'count': 10}
min-seq-sep: 24 xL: Top-L   {'mae': 2.1541, 'mse': 8.1816, 'rmse': 2.8603, 'count': 50}
min-seq-sep: 24 xL: Top-NC  {'mae': 2.4536, 'mse': 9.6231, 'rmse': 3.1021, 'count': 295}
Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 0.9, 'count': 10}
min-seq-sep: 12 xL: Top-L   {'precision': 0.6, 'count': 30}
min-seq-sep: 12 xL: Top-NC  {'precision': 0.6, 'count': 30}
min-seq-sep: 24 xL: Top-L/5 {'precision': 0.5, 'count': 10}
min-seq-sep: 24 xL: Top-L   {'precision': 0.38462, 'count': 13}
min-seq-sep: 24 xL: Top-NC  {'precision': 0.38462, 'count': 13}Example 3. Evaluate trRosetta predictiondisteval-n./test/1guuA.pdb-r./test/1guuA.npzExpected output:Evaluating distances..
min-seq-sep: 12 xL: Top-L/5 {'mae': 0.5485, 'mse': 0.5375, 'rmse': 0.7331, 'count': 10}
min-seq-sep: 12 xL: Top-L   {'mae': 0.6789, 'mse': 0.7678, 'rmse': 0.8762, 'count': 50}
min-seq-sep: 12 xL: Top-NC  {'mae': 1.2951, 'mse': 3.8733, 'rmse': 1.9681, 'count': 741}
min-seq-sep: 24 xL: Top-L/5 {'mae': 0.537, 'mse': 0.4237, 'rmse': 0.6509, 'count': 10}
min-seq-sep: 24 xL: Top-L   {'mae': 0.6691, 'mse': 0.6725, 'rmse': 0.8201, 'count': 50}
min-seq-sep: 24 xL: Top-NC  {'mae': 1.2281, 'mse': 3.2863, 'rmse': 1.8128, 'count': 351}

Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 1.0, 'count': 10}
min-seq-sep: 12 xL: Top-L   {'precision': 0.8, 'count': 30}
min-seq-sep: 12 xL: Top-NC  {'precision': 0.8, 'count': 30}
min-seq-sep: 24 xL: Top-L/5 {'precision': 1.0, 'count': 10}
min-seq-sep: 24 xL: Top-L   {'precision': 0.84615, 'count': 13}
min-seq-sep: 24 xL: Top-NC  {'precision': 0.84615, 'count': 13}Example 4. Evaluate a CASP14 RR filewgethttp://deep.cs.umsl.edu/disteval/static/data/casp14/T1024/RaptorX_RR1
wgethttp://deep.cs.umsl.edu/disteval/static/data/casp14/casp14_pdbs/T1024.pdb

disteval-n./T1024.pdb-c./RaptorX_RR1Expected output:Evaluating distances..
min-seq-sep: 12 xL: Top-L/5 {'mae': 1.7837, 'mse': 4.9053, 'rmse': 2.2148, 'count': 78}
min-seq-sep: 12 xL: Top-L   {'mae': 2.4797, 'mse': 13.0069, 'rmse': 3.6065, 'count': 392}
min-seq-sep: 12 xL: Top-NC  {'mae': 3.6061, 'mse': 16.4059, 'rmse': 4.0504, 'count': 5459}
min-seq-sep: 24 xL: Top-L/5 {'mae': 1.7837, 'mse': 4.9053, 'rmse': 2.2148, 'count': 78}
min-seq-sep: 24 xL: Top-L   {'mae': 2.4398, 'mse': 12.8404, 'rmse': 3.5834, 'count': 392}
min-seq-sep: 24 xL: Top-NC  {'mae': 3.6114, 'mse': 16.4634, 'rmse': 4.0575, 'count': 4906}
Evaluating contacts..
min-seq-sep: 12 xL: Top-L/5 {'precision': 0.9359, 'count': 78}
min-seq-sep: 12 xL: Top-L   {'precision': 0.82143, 'count': 392}
min-seq-sep: 12 xL: Top-NC  {'precision': 0.68562, 'count': 633}
min-seq-sep: 24 xL: Top-L/5 {'precision': 0.9359, 'count': 78}
min-seq-sep: 24 xL: Top-L   {'precision': 0.80357, 'count': 392}
min-seq-sep: 24 xL: Top-NC  {'precision': 0.68631, 'count': 577}Evaluation through 3D modeling usingdisteval.pyPrerequisitesInstall cshsudoaptinstallcshDownload 'dssp-2.0.4-linux-amd64' fromhttps://osf.io/qydjv/chmod+xdssp-2.0.4-linux-amd64Download TM-score fromhttps://zhanglab.ccmb.med.umich.edu/TM-score/TMscore.gzwgethttps://zhanglab.ccmb.med.umich.edu/TM-score/TMscore.gz
gunzipTMscore.gz
chmod+xTMscoreDISTFOLDFollow instructionshereto download DISTFOLD, an updated version of CONFOLD.TestExample 1. Predicted contacts (RR file) & Secondary structuredisteval -f ./test/1guuA.fasta -n ./test/1guuA.pdb -c ./test/1guuA.contact.rr -s ./test/1guuA.ss -o ./build-1guuA  -bExpected output:TM-score RMSD    GDT-TS MODEL
0.297    10.100  0.385  1guuA_11.pdb
0.320     7.729  0.460  1guuA_8.pdb
...
0.465     3.935  0.630  1guuA_model1.pdb
0.483     5.776  0.600  1guuA_model2.pdb
0.550     4.534  0.665  1guuA_5.pdbExample 2. Predicted distance map (up to 12Å) without local distances & Secondary structuredisteval -f ./test/1guuA.fasta -n ./test/1guuA.pdb -d ./test/1guuA.predicted.npy -s ./test/1guuA.ss -o ./build-1guuA -b -m 6 -t 12Expected output:TM-score RMSD    GDT-TS MODEL
0.107    37.610  0.155  extended.pdb
0.630     3.016  0.745  1guuA_11.pdb
...
0.681     2.528  0.785  1guuA_6.pdb
0.681     2.489  0.790  1guuA_9.pdbExample 3. Predicted distance map (up to 12Å) including local distancesdisteval -f ./test/1guuA.fasta -n ./test/1guuA.pdb -d ./test/1guuA.predicted.npy -s ./test/1guuA.ss -o ./build-1guuA -b -m 2 -t 12Expected output:TM-score RMSD    GDT-TS MODEL
0.107    37.610  0.155  extended.pdb
0.253    10.230  0.340  1guuA_11.pdb
...
0.681     3.349  0.775  1guuA_13.pdb
0.684     2.330  0.795  1guuA_3.pdbExample 4. Reconstruction using a native (true) distance mapdisteval -f ./test/1guuA.fasta -n ./test/1guuA.pdb -o ./build-1guuA -p -b -m 2 -t 18Expected output:TM-score RMSD    GDT-TS MODEL
0.107    37.610  0.155  extended.pdb
...
0.987     0.265  1.000  1guuA_model2.pdb
0.991     0.214  1.000  1guuA_16.pdbExample 5. Distances predicted by trRosetta methoddisteval -f ./test/1guuA.fasta -n ./test/1guuA.pdb -r ./test/1guuA.npz -o ./build-1guuA -b -m 2 -t 12Expected output:TM-score RMSD    GDT-TS MODEL
0.107    37.610  0.155  extended.pdb
0.268     9.724  0.375  1guuA_14.pdb
...
0.876     0.979  0.940  1guuA_model1.pdb
0.880     1.151  0.950  1guuA_16.pdbUsing as a LibraryUsageExample 1. Convert PDB file to distance mapfromdistevalimportpdp2dmap

pdb2dmap('path_to_pdb_file')Example 2. Convert trRosetta prediction file (.npz) file to distance mapfromdistevalimporttrrosetta2maps

trrosetta2maps('path_to_trRosetta_npz_file')For other functionsPlease checkhttps://github.com/ba-lab/disteval/blob/main/disteval.pyContactBadri Adhikariadhikarib@umsl.eduUniversity of Missouri-St. LouisPublished ByBikash Shresthabsmmy@umsystem.eduUniversity of Missouri-St. Louis