Data from automatic data collection at SPring-8
September 28, 2020
Data collected at SPring-8 PX-BLs with ZOO are processed with XDS through automated data processing
pipeline KAMO. The processed data are transferred to users with merged data.

<Reference of automatic data collection and automatic data processing>If you publish your structures from data collected using ZOO and analyzed with KAMO, please cite the following
papers as references. The citations are exceedingly important for us to continue developing more advanced
techniques at synchrotron radiation technology.
Thank you for your cooperation.
All staff
1. K. Hirata, K. Yamashita, G. Ueno, Y. Kawano, K. Hasegawa, T. Kumasaka, and M. Yamamoto
ZOO: an automatic data-collection system for high-throughput structure analysis in protein
microcrystallography
Acta Cryst​. D75, 1–13, (2019).
[doi: 10.1107 / S2059798318017795]
2. K. Yamashita, K. Hirata, and M. Yamamoto
KAMO: towards automated data processing for microcrystals
Acta Cryst​. D74, 441-449, (2018).
[doi: 10.1107 / S2059798318004576]

1. Structure of diffraction dataset directories
Files and directories copied on the media.

~/YYMMDD_ blname_username/ <- Root directory┣ “PuckID”-”PinID”/ <- Data directory Containing data
┃ ┣ ##.ppm, raster.jpg <- On gonio loop image after loop centering┃ ┣ scan**/ <- Directory for raster scan (including SHIKA analysis results)┃ ┗ data**/ <- Directory for dataset (helical, single, multi)┃┣ _kamoproc/ <- KAMO working directory, processing results┃ ┣ “PuckID”-”PinID”/ <- Processing results of XDS┃ ┣ report.html, correct.html, contents/ <- Reports of XDS processing┃ ┗ merge_**_**_”SampleName”/ <- Merge results for small wedge data┃
(e.g. multi mode data collection)┃┣ _kamo_30deg/ <- 30 deg divided data processing for helical mode data
┃ ┣ “PuckID”-”PinID”/ <- Processing results of XDS┃ ┣ report.html, contents/ <- Reports of XDS processing┃ ┗ merge_**_**_SampleName/ <- Merge results for the divided data┃
┣ YYMMDD_BL45XU_username_00.html <- Report file of ZOO┣ YYMMDD_BL45XU_username.tgz <- Archived report file containing processed reflection┃files┃┗ *csv, *db, *xlsx etc. <- Configuration files for automatic data collection

2. Diffraction data
The directory name for diffraction data corresponds to​​“PuckID”-“PinID”​of the sample sheet.
~/YYMMDD_ blname_username/┣ “PuckID”-”PinID”/┣ ##.ppm, raster.jpg <- On gonio loop image after loop centering┣ scan**/ <- Directory for raster scan (including SHIKA analysis results)┗ data**/ <- Directory for dataset (helical, single, multi)
12.1. Loop centering images
Image files for loop centering stored in the directory are altered upon selected options (e.g. zoom capture, LN2flash)
1. before_warmup.ppm <- Before warming up, just after mounting
2. after_warmup.ppm <- After warming up before precise centering
3. raster.jpg<- After loop centering, defined 2D scan area is shown with green rectangle
4. before.ppm <- Same as raster.jpg without 2D scan area indication
5. loop_zoom.ppm <- zoom up image after loop centering
6. loop_zoom_ln2.ppm<- zoom up image after loop centering and defrosting with liquid N2
2.2. Directory structureThe directory structure (directory tree) for the data-collection scheme is described. There are directories named“PuckID”-“PinID”​, i.e., “sample directory”, directly under the root directory. The sample directories contain raw
diffraction data from raster scan and data collection and some images produced with loop centering.
EIGER detector produces hdf5 files. In HDF5 format, every 100 frames are archived such as ​***_000001.h5​.
(BL32XU, BL41XU). PILATUS detector produces a cbf file from a frame. (BL45XU)
2.2.1. multi mode~/YYMMDD_ blname_username/┣ MUL0001-01/ <- Sample directory┣ ##.ppm, raster.jpg <- Images produced with loop centering (ref. 2.1.)┣ scan**/ <- Containing raster scan data (including SHIKA analysis results)┃ ┗ 2d/ <- Scan type (2d means 2 dimension)┃┗ spotfinder/: Containing SHIKA analysis results┃┗ thumb_2d/ <- Thumbnails of SHIKA diffraction images┗ data00/ <- Containing diffraction images from data collection┃(If data collection has been done twice, the number is incremented)┗ MUL0001-01-multi_###_******.cbf（###: number of crystals that satisfy a criteria. It is limited by "# ofcrystals".)
2.2.2. helical mode~/YYMMDD_ blname_username/┣ HEL0002-002/ <- sample directory┣ ##.ppm, raster.jpg <- Images produced with loop centering (ref. 2.1.)┣ scan**/ <- Containing raster scan data (including SHIKA analysis results)┃ ┣ 2d/ <- Scan type (2d means 2 dimension)┃ ┃ ┗ spotfinder/ <- Containing SHIKA analysis results┃ ┃┗ thumb_2d/ <- Thumbnails of SHIKA diffraction images┃ ┣ lv00_00/ <- Containing data of vertical scan at left edge┃ ┃ ┗ spotfinder/ <- Containing SHIKA analysis results┃ ┃┗ thumb_2d/ <- Thumbnails of SHIKA diffraction images┃ ┣ rv00_00/ <- Containing data of vertical scan at right edge┃ ┃ ┗ spotfinder/ <- Containing SHIKA analysis results┃ ┃┗ thumb_2d/ <- Thumbnails of SHIKA diffraction images┃ ┣ rv00_01/ <- Containing data of second attempt of vertical scan at right┃┃edge (index 00->01)┃┗ spotfinder/ <- Containing SHIKA analysis results┃┗ thumb_2d/ <- Thumbnails of SHIKA diffraction images┗ data00/ <- Containing diffraction images from data collection┃(If data collection has been done twice, the number is incremented)┗ cry##_******.cbf/h5: Raw data(##: number of crystals that satisfy a criteria. It is limited by "# of crystals".)22.2.3. single mode~/YYMMDD_ blname_username/┣ SIN003-003/ <- sample directory┣ ##.ppm, raster.jpg <- Images produced with loop centering (ref. 2.1.)┣ scan**/ <- Containing raster scan data (including SHIKA analysis results)┃ ┣ 2d/ <- Scan type (2d means 2 dimension)┃ ┃ ┗ spotfinder/ <- Containing SHIKA analysis results┃ ┃┗ thumb_2d/ <- Thumbnails of SHIKA diffraction images┃ ┗ lv00_00/ <- Containing data of vertical scan at left edge┃
┗ spotfinder/ <- Containing SHIKA analysis results┃┗ thumb_2d/ <- Thumbnails of SHIKA diffraction images┗ data00/ <- Containing diffraction images from data collection┃(If data collection has been done twice, the number is incremented)┗ single_001_******.cbf/h5/ <- Raw data●●

lv/rv01_00​means a vertical scan of the second crystal in the case of multiple crystals in helical mode.
In some cases, ZOO repeats vertical scan for centering the crystal at the left and right edges due toweak diffraction from this direction or if the X-ray beam does not hit the crystal. If ZOO cannot detect theedge of the crystal with vertical scan, it shifts the exposure position 10 um inside the crystal andexecutes vertical scan again (repeat up to 5 times). If ZOO fails vertical scan 5 times, it gives up datacollection. Results of the attempts are stored in ​lv/rv00_01/​(last digit can be incremented)

3. Results of automated data processing pipeline KAMO
The data are processed with the automated data processing pipeline KAMO under the ​_kamoproc/ directory.
Refer to ​XDS.INP​and ​CORRECT.LP​to evaluate the processing results.
~/YYMMDD_BLNAME_USER/┗ _kamoproc/ <- KAMO data processing directory┣ MUL0001-01/ <- in multi mode┃ ┗ data00/ <- same structure as data directory┃┣ MUL0001-01-multi_001_1-100/ <- working directory for data processing program┃┣ MUL0001-01-multi_002_1-100/┃┣ MUL0001-01-multi_003_1-100/┃┗ MUL0001-01-multi_0##_1-100/┃┣ HEL0002-002/ <- in helical mode┃ ┗ data00/┃┗ cry00_1-3600/ <- working directory for data processing program┃┣ SIN003-003/ <- in single mode┃ ┗ data00/┃┗ single_1-3600/ <- working directory for data processing program┃┣ report.html <- summary of data processing┗ correct.html <- summary of data statistics of helical/single mode data collection

Processing helical data as divided small wedge data (KAMO “buttagiri” processing)
For the data collected in helical mode, considering the heterogeneity of the crystals, KAMO processes the data
divided into multiple data sets within a 30-deg. oscillation range (e.g. The data that have 720 deg. of total
oscillation range is divided into 24 wedges with 30 deg. of oscillation range). The divided data are processedindependently followed by merged based on “sample_name” designated by users. This process is run under_kamo_30deg/. The reduced reflection data are found in working directories.

3Conceptual diagram of KAMO “buttagiri” processing* Data processing may not be possible depending on the quality of diffraction data
* In some clusters, completeness or redundancy falls below the threshold, and the mtz file is not output in theend.
~/YYMMDD_BLNAME_USERNAME/┗ _kamo_30deg/ <- KAMO“buttagiri” data processing directory┣ MUL0001-01/ <- in multi mode┃ ┗ data00/ <- same structure as data directory┃┣ MUL0001-01-multi_001_1-100/ <- working directory for data processing program┃┣ MUL0001-01-multi_002_1-100/┃┣ MUL0001-01-multi_003_1-100/┃┗ MUL0001-01-multi_00#_1-100/┃┣ HEL0002-002/ <- in helical mode (total 360 deg., 0.1 deg./frame)┃ ┗ data00/ <- same structure as data directory┃┣ cry00_1-300/ <- working directory for 0 – 30 deg.
┃
┣ cry00_301-600/ <- working directory for 30 – 60 deg.
┃
┣ … continue…
┃┗ cry00_3301-3600/ <- working directory for 330 – 600 deg.
┃
┣ SIN003-003/ <- in helical mode (total 360 deg., 0.1 deg./frame)┃ ┗ data00/┃┣ single_1-300/ <- working directory for 0 – 30 deg.
┃┣ single_301-600/ <- working directory for 30 – 60 deg.
┃┣ …continue…
┃┗ single_3301-3600/ <- working directory for 330 – 600 deg.
┃┗ report.html <- summary of data processing with “buttagiri”

44. Report from SPring-8
The archive saved in the root directory as ​YYMMDD_BL45XU_USERNAME.tgz contains report files further described
below~/YYMMDD_BL45XU_USERNAME/┣ report_YYMMDD_BL45XU_USERNAME_00.html <- summary of data collection (ref. 4.1)┃┣_kamoproc/ <- outputs from automated data processing pipeline KAMO (ref. 3)┃ ┣ correct.html <- summary of data statistics of helical and single mode(ref. 4.2)
┃ ┣ contents/ <- contains files referenced by correct.html┃ ┣ merge_***/ <- results of auto_multi_merge (ref. 5)┃ ┃ ┗ blend(/cc)_##A_final/┃ ┃┣ report.html (ref. 4.3)
┃ ┃┗ cluster_##/ <- Only the top clusters are included in the report┃ ┃┗ run_03/┃ ┃┣ XSCALE.INP┃ ┃┣ xscale.hkl┃ ┃┣ XSCALE.LP┃ ┃┣ aniso.log┃ ┃┗ pointless.log┃ ┣ “PuckID”-”PinID”/ <- in helical mode┃ ┃ ┗ data00/┃ ┃┗ cry00_1-3600/ <- working directory for data processing program┃ ┃┣ CORRECT.LP┃ ┃┗ XDS_ASCII.HKL┃ ┣ “PuckID”-”PinID”/ <- in single mode
┃ ┃┗ data00/┃ ┃┗ single_1-###/ <- working directory for data processing program┃ ┃┣ CORRECT.LP┃ ┃┗ XDS_ASCII.HKL┃ ┣ … continue…
┃┗ _kamo_30deg/ <- outputs from KAMO buttagiri┗ merge_***/ <- results of auto_multi_merge (ref. 5)┗ blend(/cc)_##A_final┣ report.html (ref. 4.3)
┗ cluster_##: Only the top clusters are included in the report┗ run_03┣ XSCALE.INP┣ xscale.hkl┣ XSCALE.LP┣ aniso.log┗ pointless.log

54.1. report_YYMMDD_BL45XU_USERNAME _00.html​, Summary of data collection

SHIKA report, The diffraction images of any grid of the heat map can be checked.
* Pop-up images cannot be viewed with this html file alone. Please check after the HDD arrives.
“#DS”, number of data sets from the pin“log comment”, status of data collection
1. Normal termination
Data collection has been done normally.
2. No crystals found in a raster scan
As a result of raster scan, there was no diffraction exceeding the threshold value; thus data collection
has not been done (helical/multi mode).
3. Crystals were not found in a raster scan
As a result of raster scan, there was no diffraction exceeding the threshold value; thus data collection
has not been done (single mode).
4. X-ray centering failed
As a result of vertical scan (performed after raster scan), there was no diffraction exceeding thethreshold value; thus data collection has not been done.
5. Loop centering failed (INOCC)
Loop centering failed for some reasons; thus data collection has not been done.
“meas time [min]”, measurement time for each pin6

4.2. correct.html​, summary of data statistics of helical/single modes_kamoproc/correct.html… continue…
The table and graphs are extracted from ​CORRECT.LP​of XDS output.
This report is not applied to the data collected in multi mode; please refer to report.html directly under_kamoproc/​.

4.3. report.html​; summary of data statistics of multi mode or helical mode “buttagiri”_kamoproc/merge_blend(or cc)_”Start Resolution”S_”Sample name”/*/report.html

The cluster numbers (​□​,​□​,​□​) in the list correspond to the numbers in the dendrogram. Roll over the cluster
number in the dendrogram to display the data-storage directory. Click the ▼ mark at the left end to display a listof statistical values. The resolution is cut off so that CC(1/2) is about 50% or more.
The resulting bulletins include only the top-level clusters. For other clusters, please check after the HDD arrives.

75. Clustering and merging
The clustering and merging process is executed based on the "Sample Name" designated in the automatic
measurement sample sheet.
The data are clustered by BLEND (lattice parameter) and CC (diffraction intensity correlation). The directory inwhich the clustering and merging process was executed is created using the following rules. The process isexecuted from half corner resolution. The threshold for mergeable data is set as a minimum redundancy of 2.0
and minimum completeness of 10%○ Under ​_kamoproc/​ ​or ​_kamo_30deg/○ Directory name● merge_”clustering method based on”_“Starting resolution”S_”SampleName”○ e.g. ​merge_blend_2.5S_sample1● Starting resolution is defined as “half corner resolution” at a particular wavelength, camera
distance, and detector.
● BLEND based on the hierarchical clustering analysis by unit cell.
● CC based on the correlation coefficients of diffraction data.
● SampleName○ Designated by users as “SampleName” in a sample sheet○ Data that have the same sample name are merged.
When automerge is started, the data are merged with the reflection up to the starting resolution. After a round of
the automerge process, KAMO auto_multi_merge estimates the resolution limit and attempts the automerge
process until the CC(1/2) of the highest resolution bin reaches around 50%. You can find theblend/cc_’RESO’A_final​directory under the merge directory.
KAMO auto_multi_merge executes hierarchical clustering based on unit cell or pairwise CC.
All data sets are scaled and merged with XSCALE (run_01)
Bad frames are removed (run_02)
Bad data sets are removed (run_03)
These directories are contained under the ​blend/cc_’RESO’A​and ​blend/cc_’RESO’A_final​directories.
Please refer to the official KAMO documentation.
https://github.com/keitaroyam/yamtbx/blob/master/doc/kamo-en.md

Reviewing the results of auto merge● “report.html”​under the merge directory is a summary of auto merge.
○ reflection file can be found in ​run_*​directory as ​xscale.hkl​.
○ mtz file is also available in the ​run_*/ccp4/​directory as ​xscale.mtz​.
● All processes through indexing to space group determination, resolution determination, and hierarchicalclustering are automated. The results should be reviewed by users. We do not take responsibility for anyinconvenience● We recommend that users should review the content of ​“report.html” and assess the ​XSCALE.INP
and ​XSCALE.LP​ ​of particular clusters for further analysis.

8e.g.) Results of auto_multi_merge with BLEND_kamoproc/┣ merge_blend_2.2S_complex1 <- working director for “complex1”┣ blend_3.03A_final <- containing final data┃┣ blend <- results of BLEND clustering┃┃┣ BLEND0.RData┃┃┣ BLEND_KEYWORDS.dat┃┃┣ CLUSTERS.txt┃┃┣ FINAL_list_of_files.dat┃┃┣ blend0.log┃┃┣ blend_cluster_summary.dat┃┃┣ dendro.json┃┃┣ forR_macropar.dat.bak┃┃┣ hctojson.R┃┃┣ tree.png <- dendrogram of BLEND clustering┃┃┣ tree.ps┃┃┗ xds_lookup_table.txt┃┃┃┣ cluster_0079 <- scaling results of a cluster (0079 = clusterID)┃┃┣ args.pkl┃┃┣ files <- containing reflection files (with link)┃┃┃┣ CPS0836-07┃┃┃┃ ....
┃┃┃┃ ....
┃┃┃┗ ....
┃┃┃┃┃┣ merge.log <- log for merge┃┃┣ merge_cluster_0079.sh <- script for calculation┃┃┣ merge_cluster_0079.sh.o1468055 <- output from computer cluster┃┃┣ merge_cluster_0079.sh.po1468055 <- output from computer cluster┃┃┣ result.pkl <- pickled result┃┃┗ run_0* <- (refer to section 5 for run_01, run_02, run_0)┃┃┣ XSCALE.INP┃┃┣ XSCALE.INP.1┃┃┣ XSCALE.LP┃┃┣ XSCALE.LP.1┃┃┣ aniso.log <- anisotropic resolution analysis┃┃┣ cc_files.dat┃┃┣ cc_frames.dat┃┃┣ ccp4┃┃┃┣ ctruncate.log┃┃┃┣ logfile.log┃┃┃┣ xds2mtz.log┃┃┃┗ ​xscale.mtz <- reflection file for structural analysis┃┃┃┃┃┣ corfac_absorp.dat┃┃┣ corfac_decay.dat┃┃┣ corfac_modpix.dat┃┃┣ pointless.log┃┃┗ ​xscale.hkl <- reflection file from XSCALE
┃┃
┃
┣ js
┃
┃
┣ LICENSE┃┃┣ d3.js┃┃┣ d3.min.js┃┃┗ index.js┃┃┃┣ cluster_summary.dat┃┣ multi_merge.log merge log┃┗ ​report.html <- report of cluster_0079 merging process┃9┣ cells.dat <- list of cell parameters┣ excluded.lst <- list of datasets excluded by clustering┣ formerge.lst <- list of data used to merge┣ input_files <- reflection files used to merge (with link)┃┣ CPS0836-07┃┣ CPS0836-08
┃
┗ ...
┃
┣ kwargs.pkl┣ multi_merge.log <- log of merge┣ multimerge.sh <- shell script for merge (for staff)┣ multimerge.sh.o1468049┗ prep_merge.log <- log for preparation of merge

10