Package: pygor3
Pygor3Pygor3 is a python3 framework to analyze, vizualize, generate
and infer V(D)J recombination models using theIGoRsoftware.
With pygor3 new models can be easily created from given V(D)J genomic templates and
provides a python interface to execute and encapsulate
IGoR’s input/outputs by using a sqlite3 database that
contains input sequences, alignments, model parameters,
conditional probabilities of the model Bayes network,
best scenarios and generation probabilities in a single db file.
Pygor3 can be use to analyze and get observables from best scenarios and
also has command line utilities to import/export
IGoR generated files toAIRR standard format.A tutorial to use pygor3 can be foundhere.Pygor InstallationPygor3 uses IGoR's as a background program to infer, evaluate and infer models.
First install IGoR, if is not already installed in your system.IGoR InstallationIGoR can be installed using the github repositoryIGoR,
pre-compiled images or by using a docker image.Installation via IGoR's Github (recommended)Gnu-LinuxMake sure to have the compilers, for example in ubuntu you can install the following packages$sudoapt-getinstallbuild-essentialNow you can download, compile and install IGoR from source$gitclonehttps://github.com/statbiophys/IGoR.git$cdIGoR$./configure&&make&&sudomakeinstallAlternatively, if no root access then use the prefix to change the installation path$./configure--prefix=${HOME}/.local/&&make&&makeinstallTo test a correct installation use run_demo command.$igor-run_demoMacOSIf you don't have gnu gcc compiler, please install it withbrew.
First install brew with:/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"Install gcc$brewinstallgcc@6Now you can download, compile and install IGoR$gitclonehttps://github.com/statbiophys/IGoR.git$cdIGoR$./configureCC=gcc-6CXX=g++-6&&make&&sudomakeinstallAlternatively, if no root access then use the prefix to change the installation path$./configureCC=gcc-6CXX=g++-6--prefix=${HOME}/.local/&&make&&makeinstallTo test a correct installation use run_demo command.$igor-run_demoInstallation via pre-compiled versions.Additionally, there are some compiled versions of IGoR shipped without IGoR default
modelshere.GNU-Linux distributionsDownload the appimage, give it executable permissions$wgethttps://github.com/statbiophys/IGoR/releases/download/1.4.2/igor-x86_64.AppImage$chmod+xigor-x86_64.AppImageMacOSCurrently there is only one supported binary for macOS$wgethttps://github.com/statbiophys/IGoR/releases/download/1.4.2/igor-x86_64_macOS_Catalina.app.gz$gunzipigor-x86_64_macOS_Catalina.app.gz$chmod+xigor-x86_64_macOS_Catalina.app:warning: Autorization need it in order to execute igor-x86_64_macOS_Catalina.app.
Please follow the instructionshereto authorize it.and download the igor default models$wgethttps://github.com/statbiophys/IGoR/releases/download/1.4.2/igor_1.4.2_default_models.tar.gz$tar-xvzfigor_1.4.2_default_models.tar.gzand set up the paths in pygor's configuration files(Pygor configuration).Further details can be foundhereInstallation via IGoR's Docker HubAnother way to get IGoR is by using adockerimage, this option
doesn't require to get a c++ compiler or the source code, and can be installed
fromthisrepository.$dockerpullalfaceor/digorAnd to run it the use of the following script is recommended:$catdigor#!/bin/bashdocker run --rm -v "$(pwd)":/igor_data -u ${UID}   digor:dev -set_wd /igor_data $@Add executable permissions to the script and optionally add it to system PATH$chmod+xdigorTo get more information about docker followthis.Pygor installation via pipEnviroment setup(Optional, but recommended) Installcondaoranacondaand create (or use ) a virtual environment.$condacreate--namestatbiophyspython=3.7$condaactivatestatbiophys(Optional, but recommended) To use the demo notebooks we recommend to use jupyter-lab(statbiophys)$condainstalljupyterlabPygor Installation via pip from PyPi repositoryPygor can be installed fromPyPirepository
using the package managerpip(statbiophys)$pipinstallpygor3Pygor Installation via GithubFor the most version of pygor (from Github)(statbiophys)$gitclonehttps://github.com/statbiophys/pygor3.git(statbiophys)$cdpygor3(statbiophys)$pipinstall-e.(Optional) Pygor configurationIf installation from sources was successful pygor3 should recognize automatically
the binary executable ("paths.igor_exec") and the default data directory("paths.igor_data").
If not you could use this configuration to personalize the paths that pygor3 looks for IGoR.To get the path of the configuration file$pygor--configThis file is usually located in ${HOME}/.local/share/pygor3/config.json.(statbiophys-dev)$cat~/.local/share/pygor3/config.jsonthe variable "paths.igor_exec" is the executable path and "paths.igor_data" is the directory
where pygor3 will look for the default IGoR's models.{"paths.igor_prefix":null,"paths.igor_exec":"/home/<username>/digor","paths.igor_data":"/home/<username>/.local/share/igor","paths.igor_models":null,"paths.igor_src":null}:warning: Notice that paths.igor_exec value should the path of the igor executable FILE be between quotes "/path/to/executable_file"
and "paths.igor_data" is the path of the root directory to look for models directory.Command Line UsageQuickstartGet demo sample dataDownload a copy of demo sequences in current directory(statbiophys)$pygordemo-get-data--------------------------------Copy data from :  /home/olivares/GitHub/statbiophys/pygor3/pygor3/demoto:  /home/olivares/testing_pygor/demoThis command creates a directory demo with the following structure, with sequences to infer and evaluate
a new model.demo├── data│   └── IgL│       ├── IgL_seqs_memory_Functional.txt│       ├── IgL_seqs_memory_Nofunctional.txt│       ├── IgL_seqs_naive_Functional_small.txt│       ├── IgL_seqs_naive_Functional.txt│       └── IgL_seqs_naive_Nofunctional.txt└── notebooks├── d00_Configuration.ipynb├── d01_RunIGoR.ipynb├── d02_ExploringModels.ipynb├── d03_EditModel.ipynb├── d04_InferringNewModel.ipynbTo use the demo notebooks change directory to notebooks and run jupyter-lab(statbiophys)$cddemo/notebooks(statbiophys)$jupyter-labFor the command line tools change to directory data/IgL(statbiophys)$cddata/IgLNew ModelsNow to create a model from scratch,
download gene templates and anchors from IMGT websiteIMGTA list of available species to download from IMGT can be queried with imgt-get-genomes command and option --info.(statbiophys)$pygorimgt-get-genomes--info--------------------------------http://www.imgt.orgDownloading data from ...List of IMGT available species:Gallus+gallusCercocebus+atysMustela+putorius+furoMacaca+nemestrinaVicugna+pacosMus+cookiiBos+taurusCanis+lupus+familiarisOrnithorhynchus+anatinusMacaca+mulattaRattus+rattusMus+minutoidesDanio+rerioOncorhynchus+mykissTursiops+truncatusFelis+catusHomo+sapiensSalmo+salarMacaca+fascicularisMus+musculusMus+saxicolaCapra+hircusSus+scrofaMus+pahariOvis+ariesEquus+caballusCamelus+dromedariusOryctolagus+cuniculusPapio+anubis+anubisMus+spretusRattus+norvegicusFor more details access:http://www.imgt.org/download/GENE-DB/IMGTGENEDB-GeneListDownload genomic templates using VJ or VDJ corresponding to the type of chain.(statbiophys)$pygorimgt-get-genomes--imgt-speciesHomo+sapiens--imgt-chainIGL-tVJ--------------------------------http://www.imgt.orgget_ref_genomeHomo+sapiens IGLV http://www.imgt.org/genedb/GENElect?query=7.2+IGLV&species=Homo+sapienshttp://www.imgt.org/genedb/GENElect?query=7.2+IGLV&species=Homo+sapiensHomo+sapiens IGLJ http://www.imgt.org/genedb/GENElect?query=7.2+IGLJ&species=Homo+sapienshttp://www.imgt.org/genedb/GENElect?query=7.2+IGLJ&species=Homo+sapienshttp://www.imgt.org/genedb/GENElect?query=8.1+IGLV&species=Homo+sapiens&IMGTlabel=2nd-CYSNo anchor is found for : AC279423|IGLV(I)-11-1*01|Homo sapiens|P|V-REGION|22452..22620|169 nt|1| | | | |169+0=169|partial in 5'| |No anchor is found for : D87007|IGLV(I)-20*01|Homo sapiens|P|V-REGION|15573..15858|286 nt|1| | | | |286+0=286| | |No anchor is found for : AC279208|IGLV(I)-20*02|Homo sapiens|P|V-REGION|19943..20228|286 nt|1| | | | |286+0=286| | |...Number of features: 0Seq('TGCTGTGTTCGGAGGAGGCACCCAGCTGACCGTCCTCG')ID: D87017|IGLJ7*02|HomoName: D87017|IGLJ7*02|HomoDescription: D87017|IGLJ7*02|Homo sapiens|F|J-REGION|18513..18550|38 nt|2| | | | |38+0=38| | |Number of features: 0Seq('TGCTGTGTTCGGAGGAGGCACCCAGCTGACCGCCCTCG')----------------------Genomic VJ templates in files:models/Homo+sapiens/IGL/ref_genome/genomicVs__imgt.fasta models/Homo+sapiens/IGL/ref_genome/genomicJs__imgt.fastaThis command creates a directorymodelswith the following structuremodels/
└── Homo+sapiens
    └── TRB
        ├── models
        └── ref_genome
            ├── genomicDs.fasta
            ├── genomicDs__imgt.fasta
            ├── genomicDs__imgt.fasta_short
            ├── genomicJs.fasta
            ├── genomicJs__imgt.fasta
            ├── genomicJs__imgt.fasta_short
            ├── genomicJs__imgt.fasta_trim
            ├── genomicVs.fasta
            ├── genomicVs__imgt.fasta
            ├── genomicVs__imgt.fasta_short
            ├── genomicVs__imgt.fasta_trim
            ├── J_gene_CDR3_anchors.csv
            ├── J_gene_CDR3_anchors__imgt.csv
            ├── J_gene_CDR3_anchors__imgt.csv_short
            ├── V_gene_CDR3_anchors.csv
            ├── V_gene_CDR3_anchors__imgt.csv
            └── V_gene_CDR3_anchors__imgt.csv_shortImportant NoteIt is important to review carefully your downloaded genes templates. Pygor automatically rename from long
IMGT descriptions to a short ones. For instanceD86996|IGLV(I)-56*01|Homo sapiens|P|V-REGION|12276..12571|296 nt|1| | | | |296+0=296| | |D86996|IGLV(I)-56*01|Homo sapiens|P|V-REGION|12576..12876|301 nt|1| | | | |301+0=301| | |Are renamed as :IGLV(I)-56*01IGLV(I)-56*01For these cases, is important to rename it or remove it manually, before  create a new model.
For simplicity in this demo we remove the second IGLV(I)-56*01Create a new initial default model, with uniform distribution for the conditional probabilities
of Bayes network ("model_marginals.txt" file). Notice that in IGoR this file is called marginals,
but it is not the marginal probability of a recombination event.(statbiophys)$pygormodel-create-Mmodels/Homo+sapiens/IGL/-tVJ--------------------------------No D genes were found.[Errno 2] No such file or directory: 'models/Homo+sapiens/IGL//ref_genome//genomicDs.fasta'No D genes were found.[Errno 2] No such file or directory: 'models/Homo+sapiens/IGL//ref_genome//genomicDs.fasta'igortask.igor_model_dir_path:  models/Homo+sapiens/IGL/Writing model parms in file  models/Homo+sapiens/IGL//models/model_parms.txtWriting model marginals in file  models/Homo+sapiens/IGL//models/model_marginals.txtInitial models with uniform parameters model files will be created in filesmodel_parms.txtandmodel_marginals.txtat directory pathmodels/└── Homo+sapiens└── IGL├── models│   ├── model_marginals.txt│   └── model_parms.txt└── ref_genome├── genomicJs.fasta├── genomicJs__imgt.fasta├── genomicJs__imgt.fasta_short├── genomicJs__imgt.fasta_trim├── genomicVs.fasta├── genomicVs__imgt.fasta├── genomicVs__imgt.fasta_short├── genomicVs__imgt.fasta_trim├── J_gene_CDR3_anchors.csv├── J_gene_CDR3_anchors__imgt.csv├── J_gene_CDR3_anchors__imgt.csv_short├── V_gene_CDR3_anchors.csv├── V_gene_CDR3_anchors__imgt.csv└── V_gene_CDR3_anchors__imgt.csv_shortAt this point you can use a set of non-productive sequence to infer a model within IGoR directly
or by using pygor command (the simpler option).(statbiophys)$pygorigor-infer-Mmodels/Homo+sapiens/IGL/-idata/IgL/IgL_seqs_naive_Nofunctional.txt-onew_IgL_naive--------------------------------===== Running inference =====...WARNING: write_model_parms path  [Errno 2] No such file or directory: ''Writing model parms in file  new_IgL_naive_parms.txtWARNING: IgorModel_Marginals.write_model_marginals path  [Errno 2] No such file or directory: ''Writing model marginals in file  new_IgL_naive_marginals.txtDatabase file :  new_IgL_naiveThis will output the following filesnew_IgL_naive.dbnew_IgL_naive_BN.pdfnew_IgL_naive_PM.pdfnew_IgL_naive_marginals.txtnew_IgL_naive_parms.txtwhere new_hs_trb.db is a database with the encapsulated information about the new model and
the date used by IGoR to infer it, new_IgL_naive_BN.pdf is a plot of the Bayesian network(BN) of inferred
model, new_IgL_naive_PM.pdf are plots of the real marginals of events in BN, and finally the
new_IgL_naive_parms.txt and new_IgL_naive_marginals.txt the inferred model in IGoR's format.Model PlotsA model can be plotted from a database file, model directory or by passing the model_parms.txt and
model_marginals.txt(statbiophys)$pygormodel-plot-Mmodels/Homo+sapiens/IGL/-oIgL_plotor(statbiophys)$pygormodel-plot-Dnew_IgL_naive.db-oIgL_plotThis will output two pdf files with the Marginal Probabilities and Conditional probabilities of eventsDatabase filesThe .db files can contain all the information in IGoR's standard files in a single sqlite database file,
and can be examinated with any sqlite client, like sqlite3 or sqlitebrowser(statbiophys)$sqlite3new_IgL_naive.dbSQLite version 3.33.0 2020-08-14 13:23:32Enter ".help" for usage hints.sqlite> .tablesIgorDAlignments       IgorIndexedSeq        IgorMM_vj_insIgorDGeneTemplate     IgorJAlignments       IgorMP_EdgesIgorER_j_5_del        IgorJGeneCDR3Anchors  IgorMP_ErrorRateIgorER_j_choice       IgorJGeneTemplate     IgorMP_Event_listIgorER_v_3_del        IgorMM_j_5_del        IgorVAlignmentsIgorER_v_choice       IgorMM_j_choice       IgorVGeneCDR3AnchorsIgorER_vj_dinucl      IgorMM_v_3_del        IgorVGeneTemplateIgorER_vj_ins         IgorMM_v_choiceIgorIndexedCDR3       IgorMM_vj_dinuclPygor has its own methods to maniputate data a database file.
For instance, db-ls list the contents of the database and the number of records(statbiophys)$pygordb-ls-Dnew_IgL_naive.db--------------------------------=== Sequences tables igor-reads:IgorIndexedSeq  :  24985=== Genomes References tables igor-genomes:IgorVGeneTemplate  :  151IgorJGeneTemplate  :  10IgorDGeneTemplate  :  0IgorVGeneCDR3Anchors  :  111IgorJGeneCDR3Anchors  :  10=== Alignments tables igor-alignments:IgorIndexedCDR3  :  24985IgorVAlignments  :  846743IgorJAlignments  :  257400IgorDAlignments  :  0=== Model tables igor-model:IgorMP_Event_list  :  6IgorMP_Edges  :  3IgorMP_ErrorRate  :  1IgorER_v_choice  :  151IgorER_j_choice  :  10IgorER_v_3_del  :  21IgorER_j_5_del  :  21IgorER_vj_ins  :  41IgorER_vj_dinucl  :  4IgorMM_v_choice  :  151IgorMM_j_choice  :  1510IgorMM_v_3_del  :  3171IgorMM_j_5_del  :  210IgorMM_vj_ins  :  41IgorMM_vj_dinucl  :  16=== Output tables igor-pgen and igor-scenarios:In a similar way the commands db-rm, db-cp, db-import and db-export can be used to manipulate
database files.(statbiophys)$pygordb-cp-Dnew_IgL_naive.db-onew_IgL_naive_mdl.db--igor-genomes--igor-model--------------------------------**** Tables in source database :  new_IgL_naive.db=== Sequences tables igor-reads:IgorIndexedSeq  :  24985=== Genomes References tables igor-genomes:IgorVGeneTemplate  :  151IgorJGeneTemplate  :  10IgorDGeneTemplate  :  0IgorVGeneCDR3Anchors  :  111IgorJGeneCDR3Anchors  :  10=== Alignments tables igor-alignments:IgorIndexedCDR3  :  24985IgorVAlignments  :  846743IgorJAlignments  :  257400IgorDAlignments  :  0=== Model tables igor-model:IgorMP_Event_list  :  6IgorMP_Edges  :  3IgorMP_ErrorRate  :  1IgorER_v_choice  :  151IgorER_j_choice  :  10IgorER_v_3_del  :  21IgorER_j_5_del  :  21IgorER_vj_ins  :  41IgorER_vj_dinucl  :  4IgorMM_v_choice  :  151IgorMM_j_choice  :  1510IgorMM_v_3_del  :  3171IgorMM_j_5_del  :  210IgorMM_vj_ins  :  41IgorMM_vj_dinucl  :  16=== Output tables igor-pgen and igor-scenarios:**** Tables in destiny database:  new_IgL_naive_mdl.db=== Sequences tables igor-reads:=== Genomes References tables igor-genomes:IgorVGeneTemplate  :  151IgorJGeneTemplate  :  10IgorDGeneTemplate  :  0IgorVGeneCDR3Anchors  :  111IgorJGeneCDR3Anchors  :  10=== Alignments tables igor-alignments:=== Model tables igor-model:IgorMP_Event_list  :  6IgorMP_Edges  :  3IgorMP_ErrorRate  :  1IgorER_v_choice  :  151IgorER_j_choice  :  10IgorER_v_3_del  :  21IgorER_j_5_del  :  21IgorER_vj_ins  :  41IgorER_vj_dinucl  :  4IgorMM_v_choice  :  151IgorMM_j_choice  :  1510IgorMM_v_3_del  :  3171IgorMM_j_5_del  :  210IgorMM_vj_ins  :  41IgorMM_vj_dinucl  :  16=== Output tables igor-pgen and igor-scenarios:Model evaluationOnce we have an inferred model we can evaluate the probability of a particular sequence to be
generated (pgen) and get the most probable scenarios for the recombination of input sequences
or generate synthetic sequences. We can evaluate sequences using the following files: model_parms.txt, model_marginals.txt, genomicXs.fasta, and X_gene_CDR3_anchors.csv or we can just use a database file with all this information like above. For instance the "new_IgL_naive_mdl.db" file, in the example above, contains
only the model and genomes information, which is necessary for the alignment and evaluation for IGoR.(statbiophys)$pygorigor-evaluate-Dnew_IgL_naive_mdl.db-idata/IgL/IgL_seqs_naive_Functional_small.txt-oIgL_naive_evaluatedAn tsv airr standard format is created with the rearragement.sequence_id	sequence	rev_comp	productive	v_call	d_call	j_call	sequence_alignment	germline_alignment	junction	junction_aa	v_cigar	d_cigar	j_cigar	v_score	v_identity	v_support	v_sequence_start	v_sequence_end	v_germline_start	v_germline_end	v_alignment_start	v_alignment_end	d_score	d_identity	d_support	d_sequence_start	d_sequence_end	d_germline_start	d_germline_end	d_alignment_start	d_alignment_end	j_score	j_identity	j_support	j_sequence_start	j_sequence_end	j_germline_start	j_germline_end	j_alignment_start	j_alignment_end	sequence_aa	vj_in_frame	stop_codon	complete_vdj	locus	sequence_alignment_aa	n1_length	np1	np1_aa	np1_length	n2_length	np2	np2_aa	np2_length	p3v_length	p5d_length	p3d_length	p5j_length	scenario_rank	scenario_proba_cond_seq	pgen	quality	quality_alignment
0	CAGTCTCCCAGGTACAAAGTCACAAAGAGGGGACAGGATGTAACTCTCAGGTGTGATCCAATTTCGAGTCATGCAACCCTTTATTGGTATCAACAGGCCCTGGGGCAGGGCCCAGAGTTTCTGACTTACTTCAATTATGAAGCTCAACCAGACAAATCAGGGCTGCCCAGTGATCGGTTCTCTGCAGAGAGGCCTGAGGGATCCATCTCCACTCTGACGATTCAGCGCACAGAGCAGCGGGACTCAGCCATGTATCGCTGTGCTAGCAGCATTCCTCGGGCTGTCAGATACGCAGTATTTTGGCCCAGGCACCCGGCTGACAGTGCTCG	F		TRBV7-7*01	TRBD2*02	TRBJ2-3*01	GGTGCTGGAGTCTCCCAGTCTCCCAGGTACAAAGTCACAAAGAGGGGACAGGATGTAACTCTCAGGTGTGATCCAATTTCGAGTCATGCAACCCTTTATTGGTATCAACAGGCCCTGGGGCAGGGCCCAGAGTTTCTGACTTACTTCAATTATGAAGCTCAACCAGACAAATCAGGGCTGCCCAGTGATCGGTTCTCTGCAGAGAGGCCTGAGGGATCCATCTCCACTCTGACGATTCAGCGCACAGAGCAGCGGGACTCAGCCATGTATCGCTGTGCCAGCAGCATTCCTCGGGCTGTCAGATACGCAGTATTTTGGCCCAGGCACCCGGCTGACAGTGCTCG		TGTGCTAGCAGCATTCCTCGGGCTGTCAGATACGCAGTATTTT		285M	4M	45M	1425			2	285	16	283			20			290	292	10	13			225			7	50	6	50		6ATTCCT		6	4	CTGT		4	0	0	0	0	1	0.02729091.34834e-19		
0	CAGTCTCCCAGGTACAAAGTCACAAAGAGGGGACAGGATGTAACTCTCAGGTGTGATCCAATTTCGAGTCATGCAACCCTTTATTGGTATCAACAGGCCCTGGGGCAGGGCCCAGAGTTTCTGACTTACTTCAATTATGAAGCTCAACCAGACAAATCAGGGCTGCCCAGTGATCGGTTCTCTGCAGAGAGGCCTGAGGGATCCATCTCCACTCTGACGATTCAGCGCACAGAGCAGCGGGACTCAGCCATGTATCGCTGTGCTAGCAGCATTCCTCGGGCTGTCAGATACGCAGTATTTTGGCCCAGGCACCCGGCTGACAGTGCTCG	F		TRBV7-7*01	TRBD2*01	TRBJ2-3*01	GGTGCTGGAGTCTCCCAGTCTCCCAGGTACAAAGTCACAAAGAGGGGACAGGATGTAACTCTCAGGTGTGATCCAATTTCGAGTCATGCAACCCTTTATTGGTATCAACAGGCCCTGGGGCAGGGCCCAGAGTTTCTGACTTACTTCAATTATGAAGCTCAACCAGACAAATCAGGGCTGCCCAGTGATCGGTTCTCTGCAGAGAGGCCTGAGGGATCCATCTCCACTCTGACGATTCAGCGCACAGAGCAGCGGGACTCAGCCATGTATCGCTGTGCCAGCAGCATTCCTCGGGCTGTCAGATACGCAGTATTTTGGCCCAGGCACCCGGCTGACAGTGCTCG		TGTGCTAGCAGCATTCCTCGGGCTGTCAGATACGCAGTATTTT		285M	4M	45M	1425			2	285	16	283			20			290	292	10	13			225			7	50	6	50		6ATTCCT		6	4	CTGT		4	0	0	0	0	2	0.02729091.34834e-19		
...DocumentationAll the command line interface commands can be used in a python environment, like jupyter notebook, by
exporting the pygor3 packageimportpygor3asp3mdl=p3.IgorModel(model_parms_file="model_parms.txt",model_marginals_file="model_marginals.txt")For further details checkout thedocumentationand notebooks directory.ContactFor any question or issue please open anissueor emailus.LICENSEFree use of pygor3 is granted under the terms of theGNU General Public License version 3(GPLv3).