mattb112885/clusterDbAnalysis — pull request #25 by JamesRH (2012-12-10 20:31:19+00:00)
Title: JamesRH commits - small updates, genbank d/l and two treeing scripts

Here is the beginning of the long-awaited committing of my code. I have gone far too long between merging trees, and my branch had deviated radically from yours (and mostly not in a good direction). In this pull request, I am including only small changes in 2 of your src files, and 4 new scripts for: genbank download, genbank parsing, and multiple cluster run trees, and gene region tree generation scripts. These are the safest of my changes. At the bottom of this note there are some things I have weeded out that I don't think you want in iTEP, but which I would be happy to contribute if they are useful.

This is also a test of this development method. If this is works well, I'll continue to use this branch (JamesRHcommits) to pass changes I have made to you. Please let me know how this works for you. I am following these instructions (for the most part) https://help.github.com/articles/using-pull-requests. If you want to reject this pull request and have me do it differently, just let me know. For instance you may want to make a JamesRHcommits branch, and then I'll submit a pull request to that.

Thanks,
James H.

Your files I changed:

db_makeBlastScoreTable.py
I added options and code for 2 additional metrics. They can be non-symetric, so this is for the tblastn searching not the clustering. Should we throw a warning, or peel these off into another script?

db_getPresenceAbsenceTable.py
I added option and code to retrieve only a specific cluster from a specific runid.

Scripts contributed:

To download genbank files from a higher-level taxonID:
scripts/downloadGenbank.py
scripts/convertGenbank2table.py
Should these be put in clusterDbAnalysis/, since they are setup files?

Display an organismal tree, and display the genes contributing to particular clusters
scripts/organismTreeClusterTable.py

Display an protein sequence tree showing the gene region, also mark where this deviates from an organismal tree
scripts/proteinTreeGeneRegion.py

Your files I did NOT change, even though they were different in my branch:

In my branch I had modified getClusterFastas.py and db_makeClusterAlignment.py to allow multiple clusters in the input. We have agreed that it the least surprising behavior is to accept only one cluster for several scripts. Also, I think I will use your db_getClusterGeneInformation.py script instead. I think I have reverted any changes to your code that implemented this multiplexing, but let me know if you see something I missed. I am changing my files in /script/ to multiplex at the level of the scripts, where it makes more sence (they are for comparing multiple clusters across cluster runs). I still have to change a few of my scripts to not expect the multiplexing behavior.

Additional scripts I did NOT contribute (but will if you want them):
I have a few script that I didn't know if you wanted to incorperate. I haven't used them for this project, and I didn't want to clutter your code tree, but if you think they are usefull I'd be happy to contribute them.

rRNA_exstract.py
A simple script to exstract the rna records from a SEED multi-genbank file and save them as a fasta file with the fig|####.#.rna.### naming convention. It works with most of the odd non-formatting of multifastas, but crashes on these files currently in iTEP:
/data/Cluster_Files/genbank/188937.1.gbk
/data/Cluster_Files/genbank/192952.1.gbk
/data/Cluster_Files/genbank/2209.7.gbk
/data/Cluster_Files/genbank/2210.3.gbk
/data/Cluster_Files/genbank/269797.3.gbk

taxonID2binomial.py
Takes a piped taxonID and returns the Genus species strain information from NCBI's taxonomy database. Might this be usefull for generating your organisms file? I know we get that from the genbank now, but if I remember it might be a field that RAST or other inputs will change in the future.

file_picker_cmdline.py
A simple function that needs only the python base library that pops open a file selection box GUI. Would our use-case for the /scripts/ make something like this useful? I've used a --gui argument in the past that opens up a selection GUI when used. This might be a pain to incorporate into our argument parsing approach. What do you think?

Table2scores2fasta.py
I have a script that merges tblastn results into a fasta file based on some simple logic that I wrote for that virus project. I will be wrapping up some scripts to do this with iTEP output in the script that will automate finding tblast hits (the logic is based on if genes overlap or not, lengths, ect.) to replace the parts of my manual filtering.

Genbank2FASTAandTable.py and Generate_FASTA_and_table.py
My first pass at the genbank download and table generation. The only additional useful functions that we might want to pull out for a library take a RAST-formatted genbank and extract a FASTA protein, nucleotide, and genome file.

Table+MCL2table.py
This is a script I had to write for that virus project that merges MCL clusters and blast information. It was already implemented in iTEP. I may use something like this to merge calls to the iTEP database into a file that cytoscape can load (I am doing quite a bit of that manually, still. Other than that, I don't think you want this.

Table2scores.py and AddBitscores_blast.py
I re-implemented your code to calculate bitscores before I learned about that function in iTEP. I also later used modified code for the virus project. I appended the bitscores to the output, instead of simplifying and had a slightly different approch. I think that the 2 additional metrics (which were the only important parts were merged, but if you think there is a need to revisit how we do this this script may be useful. It could potentially remove the need to calculate self-bitscores during the SQL load (or in SQL at all), but as it was designed for tblastn output, there are some gotchas.

Plot_all-vs-all_histograms.py
Plots histograms of all pairwise sets of organisms's reciprocal blast scores. I used this when investigating the cutoffs for clustering, but it is probably not something that other's would want to do, since we have a better way to visualize the clustering now.