BACKGROUND
single-molecule techniques have proliferated over the past decade  <cit> .
despite the power of these techniques and their widespread use, critical assessment of single-molecule data remains challenging.
while there are multiple reasons for this, principal among these are the inherent noise and stochasticity associated with single-molecule events, which contribute substantially to the analysis challenge.
to help manage similarly complex data sets generated from a number of techniques used in modern biological research, other fields have adopted standard data file formats, repositories, and analysis approaches.
examples include the pdb file format for structural data; the rcsb pdb repository of biomolecular structures; the nih genbank, ddbj, and embl ena repositories of gene and genome sequences; the ncbi blast and ensembl sequence alignment and analysis tools; and the cnssolve biomolecular structure determination tool  <cit> .
standardization has been a key part of the development and advancement of these resources and techniques, facilitating data sharing and dissemination.
in addition, the transparency of these formats, repositories, and tools encourages critical assessment of data.
individually the effect of these changes is difficult to assess, but cumulatively they contribute to increased reproducibility and reliability of measurements and, as a result, to the growth and widespread adoption of these techniques.
these examples represent important successes that have arisen naturally.
however, several institutions and scientific leaders have recently begun to insist on greater transparency in the dissemination and treatment of all types of scientific data  <cit> .
while there are many reasons for this desire and need, a number of well-documented instances within the drug discovery industry where the reproducibility of scientific results has been questioned  <cit>  has raised awareness that a lack of easy access to raw data  and a lack of tools for the primary analysis of the data can undermine clear communication of scientific results and can contribute to erroneous conclusions.
such high-profile problems cannot be attributed to any single failing, but a contributing cause is likely a current lack of standardization and control across the numerous measurement techniques that are combined to support these multidisciplinary development efforts  <cit> .
currently there is no standardization in place to unify the common aspects of most single-molecule data sets and to facilitate the use of the sophisticated analysis approaches that are continually being developed  <cit> .
we propose the single-molecule dataset  file structure as a general data format for storing and disseminating single-molecule data.
moreover, we take steps to facilitate this transition by making two previously established data-analysis packages created in independent labs compatible with this format.
implementation
there are many commonalities in how single-molecule data are collected, stored, and analyzed.
figure 1a outlines three unifying relationships that form the basis of the smd hierarchy.
most single-molecule datasets take the form of time series data  that are acquired simultaneously from one or more channels during an experiment.
while this is not always the rawest form of the data , the single-molecule trace unifies many different techniques.
at the highest level, a set of single-molecule traces  are unified by the particular experiment that was used to generate them .
finally, associated with each trace can be experimental information and quantities derived from the analysis of the raw single-molecule data .
the aim of smd is to encapsulate this hierarchy in a file structure that is independent of any particular programming language, data acquisition platform, or data analysis tool and that is widely compatible with distinct techniques and analysis strategies.figure  <dig>
structure of smd.
cartoon representation of the smd hierarchy.
each experiment, represented by the purple rectangle, encompasses the raw data of many single-molecule traces, each represented by a black rectangle.
representation of an individual single-molecule trace within the above experiment.
raw single-molecule data consist of time series data arising from one or more channels.
in this example, we depict two channels containing raw data as well as one channel containing an idealized trajectory determined in post-processing.
associated with the raw data of each trace are attributes that are unique to that trace , such as derived kinetic and thermodynamic parameters obtained from model fitting.
representation of the smd format in javascript object notation .
the color scheme is used from the cartoon representation in panel .
there are many file types that easily accommodate the hierarchy of smd .
indeed, in any high-level analysis package one of these formats is likely to be used.
however, to ensure the maximum interoperability between analysis tools, a standard text-based description is advantageous because it allows for straightforward determination of the data fields in a file without any prior knowledge of the specific experiment, data acquisition platform, or data analysis tools used.
for interoperability purposes, a smd object is represented in the widely used javascript object notation  format, whose nested structure naturally accommodates the smd hierarchy.
RESULTS
the smd format aims to strike a balance between defining enough structure to facilitate interoperability of software packages and exchange of data between groups and providing enough flexibility to accommodate data associated with different experimental techniques and analysis use cases.
the most important assumption we make is that the dataset holds traces with a fixed set of channels  that are annotated by some set of attributes .
the attributes may be quite specific to the type of experiment and analysis performed, but the channel values themselves should in general be suitable to visualization and analysis with different software packages.
figure 1b outlines how the three components of smd are structured in the json notation .
each trace contains four fields.
the values field stores the trace data where each data type is specified by a descriptive tag.
the index field contains a list of row labels for the trace .
any other trace-specific annotations  are placed in the attr field.
finally the id field is used to store a  <dig> digit hexadecimal number generated by running the md <dig> algorithm on the data for each trace.
the list of traces is itself stored in the data field of an outer top-level structure, which itself has a dataset-specific id  field as well as an attr field that holds top-level annotations or summary statistics that apply to the dataset as a whole  and a desc field that contains a string describing the data set.
additionally, the dataset-specific types specifies the data type for each instance of data being stored in each set of values.
a full description of the smd specification is provided in the additional file  <dig>
to facilitate the design and adoption of smd we made the ebfret  <cit>  and smart  <cit>  single-molecule data analysis packages and visualization tools compatible with the smd file format.
we note here that ebfret is a descendent of the previously released vbfret  <cit>  data analysis package.
we also provide a number of tools for the basic support and validation of smd files in both matlab™ and python packages.
full documentation of smd and these tools is available at https://smdata.github.io.
the collaboration that resulted in smd enabled many details that are important for ensuring generality to be implemented.
the ebfret and smart data analysis packages were developed independently from one another and as a result have significantly different functionalities and work flows.
the ability of smd to easily accommodate these packages with multiple graphical interfaces and distinct outputs provides a strong indication that smd will be able to accommodate the needs of many researchers.
CONCLUSIONS
adoption of smd or, as needed, a different format that encapsulates generalities not anticipated at this time, is an important step for the realization of the full potential of single-molecule measurements by and for a broad scientific community.
although it will require some discipline for researchers to abide by  a common set of standards, the potential long-term benefits are hard to overstate.
standardization will help facilitate the transfer of information among different labs by ensuring that a minimal structure and set of information are present.
in turn, this information sharing will facilitate further critical assessment  and reanalysis of single-molecule datasets, important steps in extracting the most from complex but information-rich single-molecule data.
moreover, adoption of a common data standard could help facilitate the creation of a repository for single-molecule data , which would enable a high degree of transparency and would ensure that data obtained now yields further insights in years to come.
we are hopeful that the flexibility of smd can easily accommodate the needs of current researchers and that it will enable researchers to reap the benefits that accompany widely adopted standardization.
availability and requirements
project name: single-molecule dataset
project home page:https://smdata.github.io
operating system: platform independent
programing languages: support provided for matlab™ and python, but smd is not tied to any particular programing language.
other requirements: none
licenses: creative commons
any restrictions to use by non-academics: none
additional file
additional file 1:
technical documentation for the smd format and supporting matlab™ and python packages.
max greenfeld and jan-willem van de meent contributed equally
competing interests
the authors declare that they have no competing interests.
authors’ contributions
mg, jmw, dsp, hm, chw, rlg and dh all contributed to the inception of the project.
mg, jwm and dsp carried out the design and implementation of the smd format.
mg, jmw, dsp, hm, chw, rlg and dh all contributed to the writing of the manuscript.
mg updated the smart package to be compatible with smd and jwm updated ebfret to be compatible with smd.
all authors read and approved the final manuscript.
the authors would like to thank any members of the single-molecule community who take the time to adopt the smd format.
in particular we would like to thank prof. frederick sacks for agreeing to make the widely used qub analysis package compatible with the smd format and for prof. taekjip ha for agreeing to make the widely used hammy analysis package compatible with the smd format.
additionally we would like to thank members of the herschlag and gonzalez labs as well as prof. aaron hoskins  for critical feedback.
this work was supported by a nih national institute of general medical science grant p <dig> gm <dig> to d.h.; a nsf career award  and a nih national institute of general medical sciences grant  to r.l.g.
; a nih national centers for biomedical computing grant  to c.h.w.
; a rubicon fellowship  from the netherlands organization for scientific research  to j.w.m.
; and a nih training grant in biotechnology  to m.g.

 single-molecule techniques have proliferated over the past decade. despite the power of these techniques and their widespread use, critical assessment of single-molecule data remains challenging. while there are multiple reasons for this, principal among these are the inherent noise and stochasticity associated with single-molecule events, which contribute substantially to the analysis challenge. to help manage similarly complex data sets generated from a number of techniques used in modern biological research, other fields have adopted standard data file formats, repositories, and analysis approaches. examples include the pdb file format for structural data; the rcsb pdb repository of biomolecular structures; the nih genbank, ddbj, and embl ena repositories of gene and genome sequences; the ncbi blast and ensembl sequence alignment and analysis tools; and the cnssolve biomolecular structure determination tool. standardization has been a key part of the development and advancement of these resources and techniques, facilitating data sharing and dissemination. in addition, the transparency of these formats, repositories, and tools encourages critical assessment of data. individually the effect of these changes is difficult to assess, but cumulatively they contribute to increased reproducibility and reliability of measurements and, as a result, to the growth and widespread adoption of these techniques. these examples represent important successes that have arisen naturally. however, several institutions and scientific leaders have recently begun to insist on greater transparency in the dissemination and treatment of all types of scientific data. while there are many reasons for this desire and need, a number of well-documented instances within the drug discovery industry where the reproducibility of scientific results has been questioned has raised awareness that a lack of easy access to raw data and a lack of tools for the primary analysis of the data can undermine clear communication of scientific results and can contribute to erroneous conclusions. such high-profile problems cannot be attributed to any single failing, but a contributing cause is likely a current lack of standardization and control across the numerous measurement techniques that are combined to support these multidisciplinary development efforts. currently there is no standardization in place to unify the common aspects of most single-molecule data sets and to facilitate the use of the sophisticated analysis approaches that are continually being developed. we propose the single-molecule dataset file structure as a general data format for storing and disseminating single-molecule data. moreover, we take steps to facilitate this transition by making two previously established data-analysis packages created in independent labs compatible with this format. implementation there are many commonalities in how single-molecule data are collected, stored, and analyzed. figure 1a outlines three unifying relationships that form the basis of the smd hierarchy. most single-molecule datasets take the form of time series data that are acquired simultaneously from one or more channels during an experiment. while this is not always the rawest form of the data, the single-molecule trace unifies many different techniques. at the highest level, a set of single-molecule traces are unified by the particular experiment that was used to generate them. finally, associated with each trace can be experimental information and quantities derived from the analysis of the raw single-molecule data. the aim of smd is to encapsulate this hierarchy in a file structure that is independent of any particular programming language, data acquisition platform, or data analysis tool and that is widely compatible with distinct techniques and analysis strategies.figure structure of smd. cartoon representation of the smd hierarchy. each experiment, represented by the purple rectangle, encompasses the raw data of many single-molecule traces, each represented by a black rectangle. representation of an individual single-molecule trace within the above experiment. raw single-molecule data consist of time series data arising from one or more channels. in this example, we depict two channels containing raw data as well as one channel containing an idealized trajectory determined in post-processing. associated with the raw data of each trace are attributes that are unique to that trace, such as derived kinetic and thermodynamic parameters obtained from model fitting. representation of the smd format in javascript object notation. the color scheme is used from the cartoon representation in panel. there are many file types that easily accommodate the hierarchy of smd. indeed, in any high-level analysis package one of these formats is likely to be used. however, to ensure the maximum interoperability between analysis tools, a standard text-based description is advantageous because it allows for straightforward determination of the data fields in a file without any prior knowledge of the specific experiment, data acquisition platform, or data analysis tools used. for interoperability purposes, a smd object is represented in the widely used javascript object notation format, whose nested structure naturally accommodates the smd hierarchy.  the smd format aims to strike a balance between defining enough structure to facilitate interoperability of software packages and exchange of data between groups and providing enough flexibility to accommodate data associated with different experimental techniques and analysis use cases. the most important assumption we make is that the dataset holds traces with a fixed set of channels that are annotated by some set of attributes. the attributes may be quite specific to the type of experiment and analysis performed, but the channel values themselves should in general be suitable to visualization and analysis with different software packages. figure 1b outlines how the three components of smd are structured in the json notation. each trace contains four fields. the values field stores the trace data where each data type is specified by a descriptive tag. the index field contains a list of row labels for the trace. any other trace-specific annotations are placed in the attr field. finally the id field is used to store a digit hexadecimal number generated by running the md algorithm on the data for each trace. the list of traces is itself stored in the data field of an outer top-level structure, which itself has a dataset-specific id field as well as an attr field that holds top-level annotations or summary statistics that apply to the dataset as a whole and a desc field that contains a string describing the data set. additionally, the dataset-specific types specifies the data type for each instance of data being stored in each set of values. a full description of the smd specification is provided in the additional file to facilitate the design and adoption of smd we made the ebfret and smart single-molecule data analysis packages and visualization tools compatible with the smd file format. we note here that ebfret is a descendent of the previously released vbfret data analysis package. we also provide a number of tools for the basic support and validation of smd files in both matlab™ and python packages. full documentation of smd and these tools is available at https://smdata.github.io. the collaboration that resulted in smd enabled many details that are important for ensuring generality to be implemented. the ebfret and smart data analysis packages were developed independently from one another and as a result have significantly different functionalities and work flows. the ability of smd to easily accommodate these packages with multiple graphical interfaces and distinct outputs provides a strong indication that smd will be able to accommodate the needs of many researchers.  adoption of smd or, as needed, a different format that encapsulates generalities not anticipated at this time, is an important step for the realization of the full potential of single-molecule measurements by and for a broad scientific community. although it will require some discipline for researchers to abide by a common set of standards, the potential long-term benefits are hard to overstate. standardization will help facilitate the transfer of information among different labs by ensuring that a minimal structure and set of information are present. in turn, this information sharing will facilitate further critical assessment and reanalysis of single-molecule datasets, important steps in extracting the most from complex but information-rich single-molecule data. moreover, adoption of a common data standard could help facilitate the creation of a repository for single-molecule data, which would enable a high degree of transparency and would ensure that data obtained now yields further insights in years to come. we are hopeful that the flexibility of smd can easily accommodate the needs of current researchers and that it will enable researchers to reap the benefits that accompany widely adopted standardization. availability and requirements project name: single-molecule dataset project home page:https://smdata.github.io operating system: platform independent programing languages: support provided for matlab™ and python, but smd is not tied to any particular programing language. other requirements: none licenses: creative commons any restrictions to use by non-academics: none additional file additional file 1: technical documentation for the smd format and supporting matlab™ and python packages. max greenfeld and jan-willem van de meent contributed equally competing interests the authors declare that they have no competing interests. authors’ contributions mg, jmw, dsp, hm, chw, rlg and dh all contributed to the inception of the project. mg, jwm and dsp carried out the design and implementation of the smd format. mg, jmw, dsp, hm, chw, rlg and dh all contributed to the writing of the manuscript. mg updated the smart package to be compatible with smd and jwm updated ebfret to be compatible with smd. all authors read and approved the final manuscript. the authors would like to thank any members of the single-molecule community who take the time to adopt the smd format. in particular we would like to thank prof. frederick sacks for agreeing to make the widely used qub analysis package compatible with the smd format and for prof. taekjip ha for agreeing to make the widely used hammy analysis package compatible with the smd format. additionally we would like to thank members of the herschlag and gonzalez labs as well as prof. aaron hoskins for critical feedback. this work was supported by a nih national institute of general medical science grant p gm to d.h.; a nsf career award and a nih national institute of general medical sciences grant to r.l.g.; a nih national centers for biomedical computing grant to c.h.w.; a rubicon fellowship from the netherlands organization for scientific research to j.w.m.; and a nih training grant in biotechnology to m.g.
the analysis and interpretation of raw single-molecule data benefits greatly from the ongoing development of sophisticated statistical analysis tools that enable accurate inference at the low signal-to-noise ratios frequently associated with these measurements. adoption of a common, standard data file format for sharing raw single-molecule data and analysis outcomes is a critical step for the emerging and powerful single-molecule field, which will benefit both sophisticated users and non-specialists by allowing standardized, transparent, and reproducible analysis practices. while a number of groups have released analysis toolkits as open source software, it remains difficult to compare analysis for experiments performed in different labs due to a lack of standardization. single-molecule techniques have emerged as incisive approaches for addressing a wide range of questions arising in contemporary biological research. smd is designed to accommodate a wide variety of computer programming languages, single-molecule techniques, and analysis strategies.
 single-molecule techniques have emerged as incisive approaches for addressing a wide range of questions arising in contemporary biological research. the analysis and interpretation of raw single-molecule data benefits greatly from the ongoing development of sophisticated statistical analysis tools that enable accurate inference at the low signal-to-noise ratios frequently associated with these measurements. while a number of groups have released analysis toolkits as open source software, it remains difficult to compare analysis for experiments performed in different labs due to a lack of standardization.  here we propose a standardized single-molecule dataset file format. smd is designed to accommodate a wide variety of computer programming languages, single-molecule techniques, and analysis strategies. to facilitate adoption of this format we have made two existing data analysis packages that are used for single-molecule analysis compatible with this format.  adoption of a common, standard data file format for sharing raw single-molecule data and analysis outcomes is a critical step for the emerging and powerful single-molecule field, which will benefit both sophisticated users and non-specialists by allowing standardized, transparent, and reproducible analysis practices. electronic supplementary material the online version of this article contains supplementary material, which is available to authorized users. keywords single moleculestandardizedfile formatsmartebfretsmdissue-copyright-statement© the author 2015 