Package: pyRMSD
pyRMSD======pyRMSD goal is the fast (and easy!) calculation of rmsd collectiveoperations, specially matrices of large ensembles of proteinconformations. It also offers a symmetric distance matrix implementationwith improved access speed and memory efficiency.If you like it and you are using it for your scientific projects, pleasecite `the pyRMSDpaper <http://bioinformatics.oxfordjournals.org/content/29/18/2363>`_.pyRMSD distributed under MIT license, and it is currently on its version4.0 .Summary=======-  `1 - Features <#1---features>`_-  `Collective operations <#collective-operations>`_-  `Condensed matrix <#condensed-matrix>`_-  `2 - Usage <#2---usage>`_-  `Getting coordinates <#getting-coordinates>`_-  `Calculating the RMSD matrix <#calculating-the-rmsd-matrix>`_-  `Available calculators <#available-calculators>`_-  `Matrix handlers <#matrix-handlers>`_-  `Accessing the RMSD matrix <#accessing-the-rmsd-matrix>`_-  `Matrix statistics <#matrix-statistics>`_-  `3 - Building & Installation <#3---building--installation>`_-  `Before installation <#before-installation>`_-  `Linux and MacOs <#linux-and-macos>`_-  `Windows <#windows>`_-  `4 - The custom building script <#4---the-custom-building-script>`_-  `Unix-based systems <#unix-based-systems>`_-  `Windows systems <#windows-systems>`_-  `Modifying system variables <#modifying-system-variables>`_-  `5 - Testing (Developers) <#5---testing-developers>`_-  `6 - Benchmarks (Developers) <#6---benchmarks-developers>`_-  `Future improvements <#future-improvements>`_-  `Credits <#credits>`_1 - Features------------Collective operations~~~~~~~~~~~~~~~~~~~~~pyRMSD currently has 5 basic operations:1 - Pairwise RMSD calculation2 - One vs. following (of a sequence of conformers).3 - One vs. all the other conformations (of a sequence of conformers).4 - Pairwise RMSD matrix5 - Iterative superposition of a sequence.All methods can use the same coordinates for fitting and RMSDcalculation, or a different set of coordinates for fitting (superposing)and calculating RMSD (referred into the code as 'calculationcoordinates' ).Currently pyRMSD implements a total of 3 superposition algorithms(Kabsch's,QTRFIT and QCP) which can have serial or parallel versions(OpenMP and CUDA in one case).The available calculators so far are: \* KABSCH\_SERIAL\_CALCULATOR \*KABSCH\_OMP\_CALCULATOR \* QTRFIT\_SERIAL\_CALCULATOR \*QTRFIT\_OMP\_CALCULATOR \* QCP\_SERIAL\_CALCULATOR \*QCP\_OMP\_CALCULATOR \* QCP\_CUDA\_CALCULATOR (in CUDA capablemachines*)* QCP\_CUDA\_MEM\_CALCULATOR (in CUDA capable machines\*)In addition it offers 2 other calculators that do not performsuperposition (for cases in which the parts of interest of the systemare already superposed): \* NOSUP\_SERIAL\_CALCULATOR\* NOSUP\_OMP\_CALCULATORThis calculator will also center the coordinates, adding a littleunnecessary overhead. This overhead will be totally diluted whencalculating RMSD matrices though.Finally it also holds a hidden calculator,QCP\_SERIAL\_FLOAT\_CALCULATOR, maninly used to test againstQCP\_CUDA\_CALCULATOR in its float version.Methods 1, 2 and 3 can be used to modify the input coordinates (theinput coordinates will be superposed). The iterative superpositionmethod will always have this behaviour as it would be senselessotherwise. Conversely, RMSD matrix will never modify input coordinates.pyRMSD can also have fitting symmetries and rotational calculationsymmetries into account. Documentation about this is on its way.If you think you need new features to be added (or better examples)click `here <#contact_features>`_.\* Computing capability of the GPU must be equal or higher than 1.1(>1.2 if built with double precision support).###Condensed matrixpyRMSD contains also a C written data type called CondensedMatrix. Thisis a representation of a squared symmetric matrix and it will save youhalf of the, otherwise redundant, memory. Besides, its write and readaccess outperforms other implementations like pure python's list-basedand even Cython implementations (see the benchmarks folder). This meansthat it will speed up for free any application that heavily relies onaccessing a distance matrix, like clustering algorithms. See theexamples below to get more insight about how to use it. ##2 - UsageSome code snippets and explanations about them will be shown below. Notethat, as the code changes rapidly, this snippets can be outdated. I willput all my effort for this not to happen, but if you detect that thecode examples are being more problematic than helpful for you, please`contact me <#contact_features>`_. You will also find method andvariables documentation in the code. Do not hesitate to ask for moredocumentation if you find is missing.Getting coordinates~~~~~~~~~~~~~~~~~~~To use the module the first thing will be to extract all the coordinatesfrom a PDB file. Coordinates must be stored in numpy arrays, using thesame layout that Prody uses:::Coordset: [Conformation 1, Conformation 2, ..., Conformation N]Conformation: [Atom 1, Atom 2,..., Atom M]Atom: [x,y,z]In order to do this there's a convenience class function in*pyRMSD/utils/proteinReading.py* called **Reader**. This will read a pdbfile using the built in reader. Despite this, we encourage the use of`Prody <http://www.csb.pitt.edu/prody/>`_ if you need to do any kind ofselection/manipulation.::from pyRMSD.utils.proteinReading import Readerreader = Reader().readThisFile("my_trajectory.pdb").gettingOnlyCAs()coordinates = reader.read()num_of_atoms = reader.numberOfAtomsnum_of_frames = reader.numberOfFramesSee 'pyRMSD/pyRMSD/test/testPdbReader.py for a simple usage example.###Calculating the RMSD matrixTo calculate the RMSD matrix you can use a **MatrixHandler** or usedirectly one calculator object to feed a CondensedMatrix.Using **MatrixHandler** to get the RMSD pairwise matrix (given that wealready have read the coordinates) will look like this:::from pyRMSD.matrixHandler import MatrixHandlerrmsd_matrix = MatrixHandler()\.createMatrix(coordinates, 'QCP_OMP_CALCULATOR')Calculating the matrix using directly the RMSDCalculator is a little bitmore verbose:::import pyRMSD.RMSDCalculatorcalculator = pyRMSD.RMSDCalculator.\RMSDCalculator(coordsets,\"QCP_SERIAL_CALCULATOR")rmsd = calculator.pairwiseRMSDMatrix()rmsd_matrix = CondensedMatrix(rmsd)As the resulting matrix is symmetric and its diagonal is 0, thermsd\_matrix object will store only the upper diagonal triangle(condensed matrix), in the same way`scipy.spatial.distance.pdist <http://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.pdist.html>`_does.###Available calculators Programatically, available calculators can bequeried with:::from pyRMSD.availableCalculators import availableCalculatorsprint availableCalculators()Matrix handlers~~~~~~~~~~~~~~~A **MatrixHandler** object will help you to create the matrix and willalso help you saving and loading matrix data to disk.::from pyRMSD.matrixHandler import MatrixHandler# Create a matrix with the coordsets and using a calculatormHandler = MatrixHandler()matrix = mHandler.createMatrix( coordsets,\"QCP_CUDA_CALCULATOR")# Save the matrix to 'to_this_file.bin'm_handler.saveMatrix("to_this_file")# Load it from 'from_this_file.bin'm_handler.loadMatrix("from_this_file")# Get the inner CondensedMatrix instancermsd_matrix = m_handler.getMatrix()Accessing the RMSD matrix~~~~~~~~~~~~~~~~~~~~~~~~~You can access a matrix object contents like this:::rmsd_at_pos_2_3 = rmsd_matrix[2,3]The **row\_lenght** parameter will give you the... row length. Rememberthat the matrix is square and symmetric, so row\_length ==column\_length, rmsd\_matrix[i,j] == rmsd\_matrix[j,i] and as it is adistance matrix, rmsd\_matrix[i,i] == 0.Matrix statistics~~~~~~~~~~~~~~~~~The CondensedMatrix class also offers an efficient way to ask for themost common statistical moments. Use the methods **calculateMean**,**calculateVariance**, **calculateSkewness** and **calculateKurtosis**to get mean, variance, skewness and kurtosis ( easy, isn't it :) ). Youcan also use **calculateMax** and **calculateMin** to get the maximumand minimum value of the matrix.3 - Building & Installation---------------------------Before installation~~~~~~~~~~~~~~~~~~~**Users** only need to install Python version 2.6/2.7 (pyRMSD has onlybeen tested with those, however it may work with another versions of thePython 2.X family). Numpy is also required. Surely you already have itinto your machine, but, in the case you don't, it can be found`here <http://sourceforge.net/projects/numpy/files/>`_. There you willbe able to find installers for almost all the combinations of platformsand Python versions you can think about.**Developers** may remember that header files of Python and Numpy may beaccessible, and your Python installation must contain the python sharedlibrary. This usually means that you have to compile it using./configure --enable-shared before building Python (usually 2.7distributions already come with this library). Prody is not adependency, but I encourage its use to handle coordinates, as it iswell-tested and powerful tool.Linux and MacOs~~~~~~~~~~~~~~~Those users have the following choices:**1)** Using the 'setup.py' file inside the root folder by typing:::> python setup.py installwith superuser provileges. Use 'build' instead to only build it. This isthe usual way python packages are deployed. As 'distutils' do notsupport CUDA directly, your package will not be able to use CUDAcalculators.**2)** Using the custom build.py script in pyRMSD main folder. This willcompile a version of pyRMSD following your configuration details. Tofinish the installation you will need to change your PYTHONPATH in orderto point to the parent folder of the package (or copy it in a folderalready inside your PYTHONPATH). See`this <http://superuser.com/questions/247620/how-to-globally-modify-the-default-pythonpath-sys-path>`_if you have any problem modifying it.Windows~~~~~~~Windows Installation is discontinued. I keep some very basicinstructions `here <#windows-systems>`_ though.4 - The custom building script------------------------------pyRMSD includes a small build script that is indeed a recipe to compilethe C extensions of pyRMSD. The build.py script is the most versatileway to compile pyRMSD and will work in almost all situations. With thisscript one can build x86 and x64 distributions with enabled or disbledCUDA calculators. Invoke it from pyRMSD root folder with:::> python build.py \[OPTIONS\]By default this script won't do anything. OPTIONS can be one of these:--build -> to compile pyRMSD (OpenMP version).--cuda single/double -> to compile it with single or double precission(you must specify only one). Double precission will not work in oldcards even if they are CUDA capable.--clean -> Will remove any generated .o files. --build --clean is a goodcombination if you are not a developer.--clean-all -> Will remove all generated files. Combine this one withany other is not a good idea. It will remove any useful built file.--build-conf -> Will determine the file (inside *build\_conf* folder)storing the configuration info.--help/-h -> Will write some hints about the options.This script uses distutil's *sysconfig* package to get the search pathfor python headers and libs automatically.The building script will try to guess the location of the needed filesfor compilation, but it can be easily modified to be able to handle allkind of scenarios.Configuration files~~~~~~~~~~~~~~~~~~~As stated before, multiple configuration files can be used by thebuilding script to feed it with the correct variables. Thisconfiguration files are stored in the *build\_conf* folder and bydefault, the 'default.conf' file is loaded (equivalent to *--build-confdefault.conf*). These are the parameters that can be changed. If one keyis not present in the file, then the contents of the key inside the'default.conf' file are used.-  "CUDA\_BASE": Base folder of the CUDA distribution installed.-  "CUDA\_INCLUDE\_FOLDER": Folder inside CUDA\_BASE where CUDA headersare found.-  "CUDA\_LIBRARIES\_FOLDER": Folder inside CUDA\_BASE where CUDAlibraries are found.-  "CUDA\_ARCHITECHTURE": Arquitecture of the GPU (in the sm\_XXformat).-  "CUDA\_LIBRARY": Name of the cuda library (usually 'cudart').-  "CUDA\_OPTIONS": Some options for the CUDA compiler.-  "DEFINE\_USE\_CUDA": Allows to redefine the macro that tells thepreprocessor that CUDA is going to be used. Changing this means youalso changed parts of the code, so is not adviced.-  "PYTHON\_EXTENSION\_OPTIONS": Compiler options usually added tocompile Python extensions.-  "PYTHON\_INCLUDE\_FOLDER": If "AUTO" it will use'distutils.sysconfig' to obtain the location of Python's headerfiles, if not it must be the full location of python's header filesfolder.-  "PYTHON\_LIBRARY\_FOLDER": If "AUTO" it will use'distutils.sysconfig' to obtain the location of Python's libraries.If "AUTO\_ALT", it will use os.path.dirname (useful for thosesituations in which 'distutils.sysconfig' fails to get the propperfolder). The other option is the full location of python's librariesfolder.-  "PYTHON\_LIBRARY" : The python library to link against.-  "OPENMP\_OPTION" : The openmp flag for the compiler used (nowadaysthe only compiler is gcc, so this flag must be -fopenmp in almost allcases).-  "NUMPY\_INCLUDE" : If "AUTO" it will use ''numpy.get\_include' to getthe folder where numpy headers are. If not it must be the full pathof this folder.-  "PYTHON\_EXTENSION\_LINKING\_OPTIONS": Compiler options usually addedto link Python extensions.See the examples into the *build\_conf* if you want to create your ownconfiguration files.Unix-based systems~~~~~~~~~~~~~~~~~~The building script was used in a Ubuntu x86 and Ubuntu x64 Os, as wellas in MacOs (Snow Leopard) to perform a non CUDA build. It had the*python-dev* package installed, so python headers were available.PYTHON\_X constants were left unchanged.It was also used under Ubuntu x64 with CUDA 4.2 to build the CUDAenabled version.####Mac Users Roman Sloutsky warns that if you're not able to compileusing the build script with default configuration options, just try tochange "PYTHON\_LIBRARY\_FOLDER":"AUTO" to"PYTHON\_LIBRARY\_FOLDER":"AUTO\_ALT" in "default.conf". Creating a newconfiguration file with only this entry will also work.Windows systems~~~~~~~~~~~~~~~A preliminary version of the build script was also tested in Windows 732 and 64 systems using MinGW compiler tools. Here are the stepsfollowed to succesfully compile the extensions:- `Download <http://www.mingw.org/>`_ and install MinGW. Then add its/bin folder to Windows PATH- `Download <http://www.python.org/download/releases/2.7.3/>`_ andinstall Python 2.7.3- `Download <http://www.scipy.org/Download>`_ and install Numpy (testedwith v. 1.7.0 for python 2.7)- [Optional] `Download <http://www.csb.pitt.edu/prody/getprody.html>`_and install Prody (tested with v. 1.4.1 for python 2.7)*PYTHON\_INCLUDE\_FOLDER* and *PYTHON\_LIBRARY\_FOLDER* constants werechanged to match our Python installation paths.*PYTHON\_EXTENSION\_LINKING\_OPTIONS* and *PYTHON\_EXTENSION\_OPTIONS*were also changed to fit Windows extension creation options.Once everything is built, create (or modify) the PYTHONPATH systemvariable and make it point the pyRMSD folder.Modifying system variables^^^^^^^^^^^^^^^^^^^^^^^^^^In order to create or modify a system variable under Windows 7, you willhave to go to Control Panel -> System and Security -> System -> AdvancedSystem Settings.5 - Testing (Developers)------------------------Once installed you can run the tests in *pyRMSD/test* using:::> python -m unittest discoverCurrently only the *test\_create\_with\_reader* test will fail if allthe dependencies are fullfilled (it's unwritten yet). If you didn'tbuild pyRMSD with CUDA support, 5 tests will be skipped.If you compiled the package using the build script, an extra test suitewill be available in the *src/calculators/test* folder with pure Ctests. Run it inside this folder with ./test\_rmsdtools\_main6 - Benchmarks (Developers)---------------------------Also available to users, inside the */benchmark* folder, there are thebenchmarks used to assest the performance of pyRMSD.There one can find some small scripts to test OpenMP parametrizations,calculation time of every implementation or even a small floating pointerror check.Future improvements-------------------If you have used this package and you feel something ismissing/incorrect or whatever, you can change it and contribute. Someexamples of things that need to be improved are:\* Solving bug in the CondensedMatrix object (erroneous creation whenusing a numpy array)\* Adding number of threads option for any OpenMP calculator. **DONE**\* Adding number of blocks and threads per block option in CUDAcalculator. **DONE**\* Create a unique installer using Python distutils (difficult becauseof the use of CUDA).\* C code needs more comments and to encapsulate function arguments (anabsolutely needed major refactoring). **DONE** \* Matrix generation loadbalance can be improved. \* Names in C code must be moreself-explanatory. **IMPROVED** \* C code must be revised and furthersimplified. **DONE**\* Add more and better tests to increase coverage.\* Convert build.py in a Makefile generator. **PARTIALLY DONE** (it nowacts just like make would do)\* Add more comments...\* Symmetry features need to be explained. \* and of course improvingthis README!! If you want to add new features (for instance massweighting) do not hesitate to contact me at: victor.gil.sepulveda atgmail.com. Of course, you can fork the repository and add as manyfeatures and improvements as you want.Credits--------  Some Numpy helper functions were first seen inhttp://www.scipy.org/Cookbook/C\_Extensions/NumPy\_arrays, by LouPecora (if I'm not wrong).-  The initial Python implementation of superposition was extracted fromPrody source code (by `AhmetBakan <http://www.csb.pitt.edu/People/abakan/>`_) and modified, withthe only goal of providing a python example to compare performanceand stability. The iterative superposition algorithm is a directtranslation of his iterpose algorithm.-  QCP superposition method code was adapted from the code`here <http://theobald.brandeis.edu/qcp/>`_-  The statistics function code was adapted from the work ofjjhaag@dreamincode.net (available`here <http://www.dreamincode.net/code/snippet1447.htm>`_ ).-  Kabsch algorithm code was adapted from the work of `Dr. Bosco K.Ho <http://boscoh.com/>`_. I would like to give him special tanks forhis help.-  As far as I know the first CUDA implementation of QCP is from `YutongZhao <http://proteneer.com/blog/>`_. He went a step further trying toimprove memory coalescence by changing the coordinates overlay.Pitifully his code is not open source.