A literature study and public survey on mass cytometry dataset release and reuse

To the Editor, Data sharing practices differ vastly across scientific disciplines [1]. For example, public deposition and release of sequencing datasets from bulk and single cell RNA and DNA sequencing (collectively termed “SeqData” here) have been required by many journals for several years [2, 3], and the US National Institutes of Health (NIH) are finalizing proposals for overall data management and sharing [4]. In contrast, fluorescence flow or mass cytometry datasets are rarely released at the time of publication. As these datasets increase in both dimensionality and number of samples, they invite analysis beyond that of the original publication, especially when the datasets are coupled with other orthogonal assay types in papers integrating multiple data types. As well as enabling opportunities for novel findings through further analyses, data deposition also increases the credibility of the scientific findings by allowing reproduction of the results. Here, we describe the extent of the data sharing gap between mass cytometry and SeqData, as well as why authors are less likely to deposit mass cytometry data than other data types. We collected and analyzed all mass cytometry articles published January 2018–June 2020, and designed a survey probing the reasons that authors decide whether to share data. The results of the literature study and data sharing survey indicate that sharing of mass cytometry data occurs at a much lower rate than SeqData data, and that a major reason that authors choose to not share cytometry data is due to the lack of hard mandates to do so. We argue that it is time for journals and funding agencies to reevaluate deposition mandates for cytometry datasets. In the period January 1, 2018 to June 17, 2020, 692 mass cytometry papers were published in 210 different journals, including preprint servers (Figures S1–S6; see Supplemental Methods for methodological details). Half of these papers were published in the top 19 journals (Figure S7; Table S1). In these papers, 89 previously published datasets were reused 167 times. Reuse was heavily skewed toward a few datasets, with 59 of the datasets reused only once. Levine et al. [5] and Samusik et al. [6], were reused 17 and 15 times, respectively; these were the only two datasets reused more than 6 times, and together they account for 19% of all dataset reuse (Table S2). Since dataset reuse is a function of overall dataset release, we examined how often datasets were released at the time of publication. Of the 514 peer-reviewed publications describing a new mass cytometry dataset, only 26% of them had accessible datasets (Figure S8C) For over 10 years, most journals have mandated public deposition of microarray datasets and various types of SeqData to sites such as GEO and ArrayExpress at the time of publication. Out of these 514 publications, 159 papers also described generating SeqData. Looking at publications describing both mass cytometry and SeqData, these data were shared in 21 and 68% of the cases respectively (Table 1). In other words, new SeqData datasets were released more than three times as often as new mass cytometry datasets from the same papers. To better understand why authors are less likely to release mass cytometry datasets than SeqData datasets, we designed a survey on dataset release for CyTOF/mass cytometry, high-dimensional (18+ color) flow cytometry, and RNAseq. There were 128 respondents to the Survey over a 6-week period (Figures S9–S11). The majority of respondents were from North America, followed by Europe. Approximately 75% were from academic institutions and almost 50% of respondents were principal investigators. The greatest number of respondents use mass cytometry in their research, followed by RNAseq and then high-dimensional flow (Figure S12–S15). When asked how many of their papers publicly released the new datasets at time of publication, the majority of respondents reported two or fewer for all three assay types (Figure S16–S18). For those papers where datasets were publicly released, the most common reason was that the respondent “believed in sharing data” and did it whenever they could (Figure S19–S21). The second most common response was that the journal required it, which gave the authors the push to release the data. When datasets were not made public, the most frequent response differed between assay types (Figure S22–S24). For mass cytometry, the most common reason was that the authors were not finished analyzing the data yet and did not want to get scooped before publishing another paper. The second most common reason was that the authors did not know where to release the data. For high-dimensional flow data, the top three reasons all relate to not knowing how or where to release the data. For RNAseq, the most common reason was that the respondent did not collect the data themselves, and considered deposition to be the duty of the owners of the data. The second most common reason was that the authors were not finished analyzing the data yet and did not want to get scooped before publishing another paper. When respondents were asked whether they had reused a public dataset in their research (Figure S25), about 75% of respondents indicated they had. When broken down by assay type, public RNA datasets had been reused most often, followed by mass cytometry and then high dimensional flow cytometry. Regardless of assay type, most respondents had only reused 1–5 public datasets (Figure S26). Since SeqData deposition at the time of publication has been required by many journals and funding agencies for several years, it was important to query how respondents felt about mandatory deposition (Figure 1). The overwhelming majority of respondents stated that they fully supported the requirement, and most would even prefer stronger enforcement by journals. When asked whether a similar requirement for release of mass or flow cytometry datasets would affect their decisions on where to publish their papers, the overwhelming majority said that it would make no difference in their publication plans, and a sizable number of respondents stated that they would likely increase submission to journals with that requirement. With the increase in multi-platform studies, reproducibility has become even more important; accordingly, timely data release is fundamental. Publicly released data can also assist researchers in their own studies by providing independent datasets for verification of new findings, data for meta-analyses, as well as training and benchmarking data for new analysis methods. Many survey respondents stated that they did not know how or where to release data, even if they did want people to reuse datasets. Unfortunately, there are no large, central, government-funded repositories for cytometry data akin to GEO or ArrayExpress for SeqData. However, public repositories for cytometry data do exist, ranging from more structured databases such as Immport (https://www.immport.org/shared/home) from the US NIH and FlowRepository (http://flowrepository.org/) from the International Society for Advancement of Cytometry (ISAC), to completely unstructured sites such as Mendeley, Figshare, and Zenodo. One important difference between ImmPort and Flow Repository versus unstructured databases is that once datasets are made public in the structured databases, they cannot be altered or deleted without involving the repository administrative staff. This helps avoid loss of datasets through “link rot,” or dataset changes without documentation. Once a dataset has been deposited, a specific accession number or digital object identifier (DOI) is generated. These identifiers can then be included in the relevant paper at the time of publication, which will help others find relevant datasets (Figure S25B), and allow easy citation to properly credit the original researchers. Some respondents indicated that they find the metadata/annotation of public datasets to be insufficient for reuse or reproducibility (Figure S25B). This could be due to the time commitment necessary for proper annotation cited by several respondents. We acknowledge that proper annotation takes time, but such issues are not something specific to cytometry data. Instead, it is something required for proper public data release of any data type [7], and the cost of the time and effort may even be fundable in grant applications [8]. Similar to MIAME [9] or MINSEQE (http://fged.org/projects/minseqe/) standards for documentation of microarray and RNAseq data, respectively, MiFlowCyt [10] and MiSet RFC [11] provide documentation standards for cytometry data. The information required by these standards could be fulfilled via specific forms during data upload (e.g., FlowRepository) or in less formal ways through spreadsheets and other files attached to the accession containing the FCS files if not directly supported by a particular repository. The easier the datasets are to find and the better the metadata, the easier it is for other researchers to reuse [12]. Several respondents stated that they did not want to release their dataset(s) because they were still analyzing them for additional publications. We recognize that the time, effort, and expense needed to generate and analyze data is considerable. However, these are also considerations for SeqData datasets, where deposition has been required for years. We would also argue that with the increasing dimensionality of flow cytometry, mass cytometry, and sequencing datasets coupled with the insights from reading new studies, there is no simple point in time when analysis of a dataset is truly exhausted. The list of published papers demonstrates that the mass cytometry field currently has a low rate of dataset release (Figure S8; Table 1). However, the results of this Survey suggest that there is substantial support in the flow and mass cytometry communities for a journal mandate to release datasets at the time of publication, similar to the mandate for SeqData datasets in the majority of journals for the last several years. (Figure 1). The authors would like to thank Ryan Brinkman, Joanne Lannigan, Anna Belkina, El-ad David Amir, Lisa Wagar, Elena Hsieh, Bill O'Gorman, and Cheryl Kim for feedback on initial Survey questions. The authors would also like to thank Lisa Wagar, Holden Maecker, and Helen McGuire for feedback on early drafts of the paper. The authors declare no conflict of interest. Michael D. Leipold designed the public survey, gathered results, and created figures. Michael D. Leipold gathered the literature survey and created figures. Michael D. Leipold and Lars Rønn Olsen discussed and wrote the paper. The peer review history for this article is available at https://publons.com/publon/10.1002/cyto.a.24512. All data necessary to reproduce the analyses in this paper are available in the Supplementary Tables. Appendix S1. Supplementary Methods Appendix S2. Supplemental PDF 1-Survey about making CyTOF, high-dimensional fluorescence flow cytometry, and RNA data publicly available Appendix S3. Supplemental Sheet 1–022321 pull-Copy of Literature-Cytoforum-end 061720 Appendix S4. Supplemental Sheet 2-Literature papers-Corresponding authors list-2018 and later-061520 Appendix S5. Supplemental Sheet 3-Raw-Public Survey about making CyTOF, high-D flow, and RNA data publicly available (Responses) Appendix S6. Supplemental Sheet 4-Processed-080420-Parsed and Counted-Copy of Working Copy of Public Survey about making CyTOF, high-D flow, and RNA data publicly available (Responses) Appendix S7. Supplemental Sheet-5-Corelabs emails-061920 Appendix S8. Survey paper-MIFlowCyt-Item-Checklist Figure S1 Figure S2 Figure S3 Figure S4 Figure S5 Figure S6 Figure S7 Figure S8 Figure S9 Figure S10 Figure S11 Figure S12 Figure S13 Figure S14 Figure S15 Figure S16 Figure S17 Figure S18 Figure S19 Figure S20 Figure S21 Figure S22 Figure S23 Figure S24 Figure S25 Figure S26 Figure S27 Table S1 Half of all CyTOF papers released January 1, 2018-June 17, 2020 were published in 19 journals. Table S2: List of publicly available CyTOF datasets representing top 50% of all dataset reuses. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.