Package: gene-ranking-shootout
Gene Ranking ShootoutA benchmark for methods that rank genes according to their relevance for a given phenotype (list of HPO terms).MethodsThe following methods are currently included in the benchmark:AMELIE (via web service)CADA (via custom Docker/Podman image)Phen2Gene (via official Docker/Podman image)Phenix algorithm (as implemented in VarFish)Exomiser Algorithms:PhenixPhiveHiPhive (variants: human only, human-mouse only, all human-mouse-fish-ppi)InstallationSimply install withpip(probably inside a conda environment or virtualenv):$gitclonehttps://github.com/bihealth/gene-ranking-shootout.git
$cdgene-ranking-shootout
$pipinstall-e.
$gene-ranking-shootout--helpUsageList datasets in the benchmark:$gene-ranking-shootoutdatasetlist
cada_clinvar_cases
cada_cases_test
cada_cases_validate
cada_all_cases
cada_collaborator_cases
cada_cases_trainShow the first entry in a dataset:$gene-ranking-shootoutdatasetheadcada_cases_test--count2{"name":"Patient:SCV000281758","disease_omim_id":"OMIM:617360","disease_gene_id":"Entrez:8621","hpo_terms":["HP:0001508"],"candidate_gene_ids":null}{"name":"Patient:SCV000864231","disease_omim_id":"OMIM:132900","disease_gene_id":"Entrez:4629","hpo_terms":["HP:0011499","HP:0000021"],"candidate_gene_ids":null}Simulate cases based on one or more datasets.
This will pick a number of cases from the datasets.
Further, it will pick another number of random genes based on the number of rare variants (freq below 0.1% in gnomAD genomes).
The results are written into a JSON file with the cases.
The simulation is randomized with a fixed seed that can be adjusted on the command line if necessary.$gene-ranking-shootoutdatasetsimulate\/tmp/cases.json\$(gene-ranking-shootoutdatasetlist)\--case-count4714\--candidate-genes-count1002023-05-0511:16:35|INFO|Loadingdata2023-05-0511:16:35|INFO|...4714casesoverall(9428duplicates)2023-05-0511:16:35|INFO|Simulatingcases2023-05-0511:16:37|INFO|Wrote4714casesYou can then run the benchmark on the cases with the different methods:$ gene-ranking-shootout benchmark
Usage: gene-ranking-shootout benchmark [OPTIONS] COMMAND [ARGS]...

  Group for benchmark sub commands.

Options:
  --help  Show this message and exit.

Commands:
  amelie          Benchmark the AMELIE web server.
  phen2gene       Benchmark the Phen2Gene container.
  summarize       Summarize the results.
  varfish-phenix  Benchmark the VarFish implementation of the Phenix...
$ gene-ranking-shootout benchmark amelie /tmp/cases.json /tmp/result-amelie.json
$ gene-ranking-shootout benchmark phen2gene /tmp/cases.json /tmp/result-phen2gene.json
$ gene-ranking-shootout benchmark varfish-phenix http://127.0.0.1:8081/hpo/sim/term-gene /tmp/cases.json /tmp/result-varfish-phenix.json
$ gene-ranking-shootout benchmark cada /tmp/cases.json /tmp/result-cada.json
$ gene-ranking-shootout benchmark exomiser http://localhost:8081/ phenix /tmp/cases.json /tmp/result-exomiser-phenix.json
$ gene-ranking-shootout benchmark exomiser http://localhost:8081/ phive /tmp/cases.json /tmp/result-exomiser-phive.json
$ gene-ranking-shootout benchmark exomiser http://localhost:8081/ hiphive /tmp/cases.json /tmp/result-exomiser-hiphive.json
$ gene-ranking-shootout benchmark exomiser http://localhost:8081/ hiphive-mouse /tmp/cases.json /tmp/result-exomiser-hiphive-mouse.json
$ gene-ranking-shootout benchmark exomiser http://localhost:8081/ hiphive-human /tmp/cases.json /tmp/result-exomiser-hiphive-human.jsonYou can also visualize the details of the benchmark results for each result file (below for 100 cases). This visualization displays the number of true disease genes (from case set definitions) at TOP10 and following positions in the ranked gene list of the respective method.$gene-ranking-shootoutbenchmarksummarize/tmp/result-amelie.json1:48################################2:17###########3:7####4:3##5:6####6:3##7:2#8:3##9:1.10:011-..:9######mssng:0Building CADA Podman ImageThere is no public REST API or docker image for CADA (yet).
Here is how to build the needed CADA Podman image:# cd docker/cada# bash build.sh...# podman run -it --rm localhost/cada-for-shootout:latest --helpRunning Phenix in VarFishSend an email to the author to get a copy of the necessary data.
Then, run the following in the background.$varfish-server-workerserverpheno--path-hpo-dirpath/to/varfish-server-worker-db/hpoRunning ExomiserThe following are more rough notes than a full manual.
This uses the current Exomiser RES API version 13.2.0 (current at: 2023-05-05).
You will need approximately 75GB of storage for download and extraction and afterwards 49GB.
Probably one could get rid of a lot of the variant-specific data but I did not go into detail here.$wgethttps://github.com/exomiser/Exomiser/releases/download/13.2.0/exomiser-rest-prioritiser-13.2.0.jar
$wgethttps://data.monarchinitiative.org/exomiser/latest/2302_phenotype.zip
$wgethttps://data.monarchinitiative.org/exomiser/latest/2302_hg19.zip
$unzip2302_phenotype.zip
$unzip2302_hg19.zip
$cat<<EOF > application.propertiesexomiser.data-directory=$PWDexomiser.hg19.data-version=1909exomiser.phenotype.data-version=2302exomiser.phenotype.random-walk-file-name=rw_string_10.mvEOF$java-Xmx6G-Xms2G-Dserver.address=0.0.0.0-Dserver.port=8081-jarexomiser-rest-prioritiser-13.2.0.jarDatasetsThe following datasets are included at the moment:cada_cases_test.json- converte from CADA'scases_test.tsvcada_cases_train.json- converte from CADA'scases_train.tsvcada_cases_validate.json- converte from CADA'scases_validate.tsvcada_clinvar_cases.json- converte from CADA'sclinvar_cases.tsvcada_collaborator_cases.json- converte from CADA'scollaborator_cases.tsvYou can conver TSV files with the following structure withgene-ranking-shootout dataset convert-tsv.Column 1: name for the case; must start withPatient:or is ignored.Column 2: disease_omim_id; asOMIM:123456orunknownColumn 3: disease_gene_id; asEntrez:123456Column 4: hpo_terms; as semicolon-separated list, e.g.,HP:0001234;HP:0005678If a row has less than 4 columns, we assume that column 2 is missing.
All further columns are ignored.
The file should not have a header.
You can find some files in the CADA repository here:https://github.com/Chengyao-Peng/CADA/tree/main/src/CADAThe call togene-ranking-shootout dataset convert-tsvshould be as follows.$gene-ranking-shootoutdatasetconvert-tsvinput.tsvoutput.jsonSome Preliminary ResultsThe following was generated on 2023/05/05 with all 4714 cases.$ for f in /tmp/result-*.json; do (set -x; gene-ranking-shootout benchmark summarize --bars-top-n 20 $f); echo; done
+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-cada.json
    1: 3462  ################################################
    2:  536  #######
    3:  205  ##
    4:  133  #
    5:   71  .
    6:   57  .
    7:   39  .
    8:   47  .
    9:   16  .
   10:   24  .
   11:   14  .
   12:   22  .
   13:   14  .
   14:   11  .
   15:    7  .
   16:    9  .
   17:    8  .
   18:    4  .
   19:    3  .
   20:    3  .

21-..:   29  .
mssng:    0  

+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-exomiser-hiphive-human.json
    1: 2593  ####################################
    2:  637  ########
    3:  375  #####
    4:  203  ##
    5:  132  #
    6:  104  #
    7:   99  #
    8:   80  #
    9:   61  .
   10:   46  .
   11:   44  .
   12:   30  .
   13:   30  .
   14:   30  .
   15:   15  .
   16:   22  .
   17:   12  .
   18:   11  .
   19:   13  .
   20:   10  .

21-..:  149  ##
mssng:    0  

+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-exomiser-hiphive.json
    1: 2418  #################################
    2:  686  #########
    3:  355  ####
    4:  226  ###
    5:  155  ##
    6:  122  #
    7:   86  #
    8:   95  #
    9:   54  .
   10:   54  .
   11:   52  .
   12:   28  .
   13:   29  .
   14:   27  .
   15:   16  .
   16:   11  .
   17:   23  .
   18:    9  .
   19:   10  .
   20:   14  .

21-..:  226  ###
mssng:    0  

+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-exomiser-hiphive-mouse.json
    1: 2418  #################################
    2:  685  #########
    3:  357  #####
    4:  227  ###
    5:  160  ##
    6:  121  #
    7:   90  #
    8:   96  #
    9:   55  .
   10:   56  .
   11:   58  .
   12:   37  .
   13:   31  .
   14:   26  .
   15:   22  .
   16:   18  .
   17:   26  .
   18:   19  .
   19:   14  .
   20:   15  .

21-..:  165  ##
mssng:    0  

+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-exomiser-phenix.json
    1: 2566  ####################################
    2:  619  ########
    3:  306  ####
    4:  208  ##
    5:  152  ##
    6:  119  #
    7:   90  #
    8:   80  #
    9:   73  #
   10:   64  .
   11:   49  .
   12:   41  .
   13:   40  .
   14:   26  .
   15:   29  .
   16:   27  .
   17:   15  .
   18:   17  .
   19:   10  .
   20:   16  .

21-..:  149  ##
mssng:    0  

+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-exomiser-phive.json
    1:  934  #############
    2:  298  ####
    3:  163  ##
    4:  101  #
    5:   50  .
    6:   44  .
    7:   30  .
    8:   33  .
    9:   16  .
   10:   16  .
   11:   10  .
   12:    9  .
   13:   17  .
   14:   13  .
   15:   12  .
   16:   16  .
   17:   14  .
   18:   23  .
   19:   28  .
   20:   33  .

21-..: 2836  #######################################
mssng:    0  

+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-phen2gene.json
    1: 2426  ##################################
    2:  470  ######
    3:  209  ##
    4:  125  #
    5:  101  #
    6:   67  .
    7:   51  .
    8:   62  .
    9:   53  .
   10:   41  .
   11:   33  .
   12:   37  .
   13:   42  .
   14:   33  .
   15:   34  .
   16:   28  .
   17:   18  .
   18:   29  .
   19:   17  .
   20:   19  .

21-..:  763  ##########
mssng:    0  

+ gene-ranking-shootout benchmark summarize --bars-top-n 20 result-varfish-phenix.json
    1: 1709  #######################
    2:  616  ########
    3:  357  ####
    4:  277  ###
    5:  184  ##
    6:  152  ##
    7:  131  #
    8:  118  #
    9:  105  #
   10:   78  #
   11:   71  .
   12:   57  .
   13:   67  .
   14:   64  .
   15:   67  .
   16:   71  .
   17:   56  .
   18:   48  .
   19:   34  .
   20:   48  .

21-..:  403  #####
mssng:    0