Package: fastq2bcl
fastq2bclfastq2bcl convert fastq files in a bcl2fastq-able run directory.A FASTQ file is a text file that contains the sequence data
from the clusters that pass filter on a flow cell.Illumina sequencing instruments generate per-cycle BCL basecall files as primary sequencing output,
but many downstream analysis applications use per-read FASTQ files as input.bcl2fastqcombines these per-cycle BCL files from a run and translates them into FASTQ files.At the same time as converting, bcl2fastq also separates multiplexed samples
(demultiplexing). Multiplexed sequencing allows you to run multiple individual samples
in one lane. The samples are identified by index sequences that were attached to the
template during sample prep. The multiplexed sample FASTQ files are assigned to
projects and samples based on a user-generated sample sheet, and stored in
corresponding project and sample directoriesFASTQ sample sequence:@M11111:222:000000000-K9H97:1:1101:21270:1316 1:N:0:1
CTTCCTAGAAGTACGTGCCAGCACGATCCAATCTCGCATCACCTTTTTTCTTTCTACTTCTACTCTCCTCTTATCTCTTCTTTTTCTTGTTTTTTTTCTTTATTCCATCT
+
CCCCCFA,,,,C9C6E-:C9,C,C+,:EC9,CFDE,@+6+;,,,C,CF7,@9E,,,C<,,,;<C,,6,,:C@,,,,:<<@,,,5=A,<,,,4,9=:@<?,,,,9C,,9,,Structure of an illumina run directory:YYMMDD_M11111_0222_000000000-K9H97
├── Data
│   └── Intensities
│       ├── BaseCalls
│       │   └── L001
│       │       ├── C1.1
│       │       │   ├── s_1_1101.bcl
│       │       │   └── s_1_1101.stats
│       │       ├── CNN.1
│       │       │   ├── s_1_1101.bcl
│       │       │   └── s_1_1101.stats
│       │       ├── s_1_1101.control
│       │       └── s_1_1101.filter
│       └── L001
│           └── s_1_1101.locs
└── RunInfo.xmlfastq2bcltake as input a set of reads (fastq.gz files) and generates a flow cell directory with:RunInfo.xmlbcl and stat for each cyclefilter filecontrol filelocation fileSee also:Illumina specsUsageHelp:usage: fastq2bcl [-h] [--version] [-v] [-vv] [-m MASK] -r1 R1 [-r2 R2] [-i1 I1] [-i2 I2] [-o OUTDIR] [--exclude-umi] [--exclude-index]

Convert fastq.gz reads and metadata in a bcl2fastq-able run directory

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -v, --verbose         set loglevel to INFO
  -vv, --very-verbose   set loglevel to DEBUG
  -m MASK, --mask MASK  define mask in format 110N10Y10Y110N
  -r1 R1, --read-1 R1   fastq.gz with R1 reads
  -r2 R2, --read-2 R2   fastq.gz with R2 reads (optional)
  -i1 I1, --index-1 I1  fastq.gz with I1 reads (optional)
  -i2 I2, --index-2 I2  fastq.gz with I2 reads (optional)
  -o OUTDIR, --outdir OUTDIR
                        Set the output directory for mocked run. default: cwd
  --exclude-umi         Do not write UMI from the R1 and R2 fastq reads to the cycles
  --exclude-index       Do not write Index from the R1 and R2 fastq reads to the cyclesUsage examples:fastq2bcl -r1 single.fastq.gz
fastq2bcl -r1 R1.fastq.gz -r2 R2.fastq.gz -i1 I1.fastq.gz -i2 I2.fastq.gz
fastq2bcl -o output_dir -r1 single.fastq.gz
fastq2bcl -o output_dir --exclude-index -r1 single.fastq.gz
fastq2bcl -o output_dir -m 100Y20N -r1 R1.fastq.gz -r2 R2.fastq.gz -i1 I1.fastq.gz -i2 I2.fastq.gzCustom maskBy default fastq2bcl will generate aRunInfo.xmlfile where Reads entries are generated using the sequence length of fastq.gz files.For exammple, if I give as input 2 pairs with length 300 bp and 2 indexes with length 8p the resulting RunInfo will be:<?xml version="1.0"?>
<RunInfo xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Version="2">
    <Run Id="YYMMDD_run_0001_ABCD" Number="1">
        <Flowcell>ABCD</Flowcell>
        <Instrument>run</Instrument>
        <Date>YYMMDD</Date>
        <Reads>
            <Read c="300" Number="1" IsIndexedRead="N" />
            <Read NumCycles="8" Number="2" IsIndexedRead="Y" />
            <Read NumCycles="8" Number="3" IsIndexedRead="Y" />
            <Read NumCycles="300" Number="4" IsIndexedRead="N" />
        </Reads>
        <FlowcellLayout LaneCount="1" SurfaceCount="1" SwathCount="1" TileCount="1" />
    </Run>
</RunInfo>You can provide a custom mask (string). For example for 1 pair 350 bp with 1 index of 8bp:350N8YInstalluse pip to install in edit mode:pip install -e .Install packages for dev in a mamba environment:mamba create -n fastq2bcl
mamba install -n fastq2bcl -c conda-forge tox pyscaffold biopython pytest-covScriptsIn the directoryscriptsthere are some useful tools:scripts/bcl2fastq_docker.shrun bcl2fastq with docker on the current directory. Run it inside a run directory.scripts/build_flowcells.shgenerate all the test flowcells using the datasets indata/testdirectoryTestuse tox or pytest to test:tox
pytestTo test with pytest you need also pytest-cov in your environment.Lintyou can lint code with:tox -e lintPre commit hook is already configured and can be installed with this command:pre-commit installFastq sequence descriptionFields in fastq description:KeyDescriptioninstrumentInstrument IDrun_numberRun number on instrument.flowcell_idsFlowcell Identifierflowcell_idsFlowcell IDSlaneLane numbertileTile numberx_posPosition X of clustery_posPosition Y of clusterUMIOptional, appears when UMI is specified in sample sheet. UMI sequences for Read 1 and Read 2, seperated by a plus [+]readRead number - 1 can be single read or Read 2 of paired-endis_filteredY if the read is filtered (did not pass), N otherwisecontrol_number0 when none of the control bits are on, otherwise it is an even number. On HiSeq X and NextSeq systems, control specification is not performed and this number is always 0.indexIndex of the readSee alsohttps://support.illumina.com/help/BaseSpace_OLH_009008/Content/Source/Informatics/BS/FileFormat_FASTQ-files_swBS.htmFilter fileThe filter files can be found in the BaseCalls directory.
The filter file specifies whether a cluster passed filters.
Filter files are generated at cycle 26 using 25 cycles of data. For each tile, one filter file is generated.
Location:Data/Intensities/BaseCalls/L001File format:s_[lane]_[tile].filterThe format is described belowBytesDescription0-3Zero value (for backwards compatibility)4-7Filter format version number8-11Number of clusters12-(N+11)Where N is the cluster number. unsigned 8-bits integer Bit 0 is pass or failed filterFilter bytes example:bytes([0, 0, 0, 0]) # prefix 0
bytes([3, 0, 0, 0]) # version 3
struct.pack("<I", cluster_count) # number of cluster in little endian unsigned int
bytes([1]*cluster_count) # For each cluster an unsigned 8-bits integer Where Bit 0 is pass or failed filter

1 == PASS FILTER
0 == NO PASS FILTERIn hexdump:BYTES 0-3      BYTES 4-7      BYTES 8-11     BYTES 12-14
00 00 00 00    03 00 00 00    03 00 00 00    01 01 01At bytes 8-11 I have 3 clusters and each cluster is represented by a an unsigned 8-bit integer.Control fileThe control files are binary files containing control results.BytesDescription0-3Zero value (for backwards compatibility)4-7Format version number12-(2xN+11)Where N is the cluster numberBit 0: always empty (0)Bit 1: was the read identified as a control?Bit 2: was the match ambiguous?Bit 3: did the read match the phiX tag?Bit 4: did the read align to match the phiX tag?Bit 5: did the read match the control index sequence?Bits 6,7: reserved for future useBits 8..15: the report key for the matched record in the controls.fasta file (specified by the REPORT_KEY metadata)Locations fileThe BCL to FASTQ converter can use different types of position files and will expect a type based on the version of RTA used
The locs files can be found in the Intensities/L<lane> directoriesBcl fileThe BCL files can be found in the BaseCalls directory inside the run directory:Data/Intensities/BaseCalls/L<lane>/C<cycle>.1They are named as follows:s_<lane>_<tile>.bclFormat:BytesDescription0-3Number of N clusters in unsigned 32bits little endian integer4-(N+3)Unsigned 8 bits integerBits 0-1 are bases encoded as: [A,C,G,T] -> [0,1,2,3] -> [00,01,10,11]Bits 2-7 are shifted by 2 bits and contain the quality score.All bits ‘0’ is reserved for no call (N)Stat fileThe stats files can be found in the BaseCalls directory inside the run directory:Data/Intensities/BaseCalls/L00<lane>/C<cycle>.1They are named as follows:s_<lane>_<tile>.statsThe Stats file is a binary file containing base calling statistics; the content is described
below.The data is for clusters passing filter only:StartDescriptionData typeByte 0Cycle numberintegerByte 4Rverage Cycle IntensitydoubleByte 12Average intensity for A over all clusters with intensity for AdoubleByte 20Average intensity for C over all clusters with intensity for CdoubleByte 28Average intensity for G over all clusters with intensity for GdoubleByte 44Average intensity for A over clusters with base call AdoubleByte 52Average intensity for C over clusters with base call CdoubleByte 60Average intensity for G over clusters with base call GdoubleByte 68Average intensity for T over clusters with base call TdoubleByte 76Number of clusters with base call AintegerByte 80Number of clusters with base call CintegerByte 84Number of clusters with base call GintegerByte 88Number of clusters with base call TintegerByte 92Number of clusters with base call XintegerByte 96Number of clusters with intensity for AintegerByte 100Number of clusters with intensity for CintegerByte 104Number of clusters with intensity for GintegerByte 108Number of clusters with intensity for TintegerReferencesbcl2fastq source code from illumina downloadshttps://support.illumina.com/sequencing/sequencing_software/bcl2fastq-conversion-software/downloads.htmlSpec file from illumina supporthttps://support.illumina.com/content/dam/illumina-support/documents/documentation/software_documentation/bcl2fastq/bcl2fastq_letterbooklet_15038058brpmi.pdfhttp://support-docs.illumina.com/IN/NovaSeq6000Dx_HTML/Content/IN/NovaSeq/SequencingOutputFiles_fNV.htmhttps://support.illumina.com/help/BaseSpace_OLH_009008/Content/Source/Informatics/BS/FileFormat_FASTQ-files_swBS.htmhttps://docs.python.org/3/library/struct.html#format-charactersSee alsomkdata.shfile in bcl2fastq source code for insights on bcl format.Acknowledgmentshttps://github.com/sottorivalabhttps://humantechnopole.itNotesThis project is inspired by the test scripthttps://github.com/ShawHahnLab/igseq/blob/dev/tools/fastq2bcl.pyfromhttps://github.com/ShawHahnLabThis project has been set up using PyScaffold 4.5. For details and usage
information on PyScaffold seehttps://pyscaffold.org/.