Scalable data analytics of the PDB with the MacroMolecular Transmission Format (MMTF) and big data technologies
Creators: name: Rose, Peter W.
name: Bradley, Anthony R.
name: Duarte, Jose M.
name: Pavelka, Antonin
name: Prlic, Andreas
name: Rose, Alexander S.
name: Valasatava, Yana
name: Yue, Yue
Publisher: name: F1000Research
Year: 2017

description: Advances in experimental techniques have led to an explosion in both the number and size of macromolecular structures in the Protein Data Bank (PDB). For this reason, the transfer and parsing of macromolecular data has become increasingly time-consuming. In this work we present the Macromolecular Transmission Format (MMTF), a new compact, extensible macromolecular file format. MMTF offers over 75% compression over mmCIF, and is over an order of magnitude faster to parse than the standard mmCIF format. We describe the new MMTF format, its Application Programming Interface, and demonstrate its use with Big Data Frameworks to enable nearly interactive data analytics on the PDB archive.<br> <br>Introduction<br>The MacroMolecular Transmission Format (MMTF) (http://mmtf.rcsb.org/) [1] was designed with three major aims. First, to minimize data size and network transfer times, the format applies custom encoding and compression techniques [2]. Both a lossless all atom and a lossy representation with reduced level of detail (C-alpha of polypeptides, phosphate backbone of polynucleotides, and all atoms of ligands) are available. Second, the data files should be fast to parse. Third, MMTF was designed to be self-contained, extensible, and interoperable. As a binary, machine-readable format, the preferred access to MMTF is through its APIs, provided in several programming languages. In this work we show that MMTF enables high-performance structural bioinformatics calculations.<br> <br>Methods<br>MMTF is a compact and extensible file format. Compression is enhanced through columnar encoding and custom compression strategies. Dictionary encoding is used to encode repeating residue, bond and atom information. Delta, run-length, and recursive index encoding strategies are used to compress data arrays, such as atomic coordinates and other atomic properties [2]. Decoder libraries are available for Java, JavaScript, Python, C, and C++. The PDB is available as individual MMTF files and as Hadoop Sequence files for parallel processing. The data files are updated weekly (http://mmtf.rcsb.org/download.html).<br><br>Results &amp; Conclusions<br>Using the Apache Spark distributed parallel computing framework, the PDB can be processed in parallel ranging from laptops, workstations, to compute clusters. We show how Big Data operations such as Filter, Map, Reduce, and Broadcasting can be used to implement structural bioinformatics workflows. Further, extracting structure and sequence information into Dataframes enables parallel SQL queries.<br> <br>Acknowledgements                                                             <br>This project was supported by the NCI of the NIH under award number U01 CA198942.<br> <br>References

	Bradley AR, Rose AS, Pavelka A, Valasatava Y, Duarte JM, Prlić A, Rose PW, PLOS Computational Biology 13(6): e1005575. doi: 10.1371/journal.pcbi.1005575 (2017).
	Valasatava Y, Bradley AR, Rose AS, Duarte JM, Prlić A, Rose PW.  PLoS One. 12: e0174846, doi:  10.1371/journal.pone.01748464 (2017).