Question: We are a small group of undergrads, mostly sophomore, from a small HBCU, and learning bioinformatics and genomics
that is not at all part of our regular syllabus, by trying to teach one another :)

Right now, we are focused on how to use `BBmap` and other such tools to process Illumina raw reads - but we are new to all of this, and experiencing a steep learning curve.

So could someone please educate us on how to interpret the `**BBDUK.sh**` stats options that create the various histogram files that are generated by invoking the following flags: `qhist`, `qchist`, `aqhist`, `bqhist`, and `lhist` ?

From several different posts here on BioStars and a few on SeqAnswers, we decided on this syntax for BBDUK based adapter and quality trimming, to keep it gentle (not too stringent) but also impose min length to prevent very crazy multi-mapping during read alignment to the reference genome (we're not there yet):

bbduk.sh -Xmx34g in=TrialSRR.fq.gz \
out=TrialSRR_AdQualTrm_minlen20.fq.gz \
ktrim=r k=23 mink=11 minlen=20 ref=adapters ordered threads=1 \
ftr=99 qtrim=r trimq=15 \
qhist=TrialSRR_minlen20.qhist \
qchist=TrialSRR_minlen20.qchist \
aqhist=TrialSRR_minlen20.aqhist \
bqhist=TrialSRR_minlen20.bqhist \
lhist=TrialSRR_minlen20.lhist;

When we generated these files, we see their contents as shown below. From these contents, think we understand a few of these numbers and column headings, but not sure that we do. Some things we gathered from this forum and the internet in general, as well as our educated guesses, are listed below. We request forum experts to add, comment, correct these inferences, and answer our pending questions. Thank you!

1. In our data, the maximum Illumina read length is 100 + 1 where the
final position is poorly calibrated and ideally trimmed out. Base#
here varies from 1-101 (human coding) or 0-100 (machine coding
starting with 0 index)?
2. In our data, 40 is the highest PHRED score
we see in our input Illumina FASTQ file, which is also the empirical
maximum for the HiSeq 2500 platform used to create these data, or is
the maximum 50 ?
3. In **QHIST** file, column total under count1 is # of nt for each PHRED
score between 0 and 41 - totaling, 1614284414	and total fraction1 is
1.00003.
4. 15983014 reads * (100+1) nt / read = 1614284414 total nt across all
reads in this library, so that tallies. It is not clear where this 1
extra nt comes from during sequencing, even from reading from other
Biostars posts. Please help. We know this 1 nt must be removed,
since it almost always poor quality, right?
5. In the **BQHIST** file, before any trimming of adapter or by quality, in the input file,
where reads are uniformly 100nt long, these gives the stats for each
of those 100 positions, right? As expected, PHRED score improves
from 5'end downstream and slightly degrades at the 3' end, right?
6. In **QHIST** file, no idea what the headers mean - **Read1_linear** or **Read1_log** ?
7. In **AQHIST** file, we think the count1 is total number of input raw
reads = 15983014 here, and we are guessing the number of reads with
average PHRED score across it's entire length, and their
corresponding fraction of the total is the third column, but not
sure at all.

We guess the **BQHIST** headers mean the following: - but what are **LW_1** and **RW_1**?

BaseNum=nt position in read
count_1=total # reads analyzed
min_1=minimum PHRED score at this position
max_1=maximum PHRED score at this position
mean_1=average PHRED score at this position
Q1_1=1st quartile of PHRED score at this position
med_1=Median or 2nd quartile of PHRED score at this position
Q3_1=3rd quartile of PHRED score at this position
**LW_1 = ??? not sure!** Help!
**RW_1 = ??? not sure!** Help!

Once again, thanks in advance!

**PS**. Apologies for any weirdness in our post's formatting, it's our first time here, and trying to use this markdown system....

**HIST file listing**

-rw-rw-r-- 1 lotte lotte 27 Nov 16 21:00 TrialSRR_minlen20.lhist
-rw-rw-r-- 1 lotte lotte 820 Nov 16 21:00 TrialSRR_minlen20.qchist
-rw-rw-r-- 1 lotte lotte 3.8K Nov 16 21:00 TrialSRR_minlen20.bqhist
-rw-rw-r-- 1 lotte lotte 1.7K Nov 16 21:00 TrialSRR_minlen20.qhist
-rw-rw-r-- 1 lotte lotte 740 Nov 16 21:00 TrialSRR_minlen20.aqhis

**LHIST file contents**

lotte@Uni-HPC:~Learning$ wc -l TrialSRR_minlen20.lhist
2 TrialSRR_minlen20.lhist
lotte@Uni-HPC:~Learning$ cat TrialSRR_minlen20.lhist
#Length	Count
101	15983014

**QCHIST file contents**

lotte@Uni-HPC:~Learning$ wc -l TrialSRR_minlen20.qchist
43 TrialSRR_minlen20.qchist
lotte@Uni-HPC:~Learning$ cat TrialSRR_minlen20.qchist
#Quality	count1	fraction1
0	277949	0.00017
1	0	0.00000
2	57657078	0.03572
3	0	0.00000
4	0	0.00000
5	369144	0.00023
6	1014149	0.00063
7	5704849	0.00353
8	5889111	0.00365
9	3397890	0.00210
10	4329169	0.00268
11	1607492	0.00100
12	1061212	0.00066
13	2845130	0.00176
14	1151754	0.00071
15	2854771	0.00177
16	4366874	0.00271
17	3243345	0.00201
18	8082441	0.00501
19	5925749	0.00367
20	6627712	0.00411
21	3527369	0.00219
22	6282643	0.00389
23	8649813	0.00536
24	11327269	0.00702
25	17252728	0.01069
26	23676889	0.01467
27	19632016	0.01216
28	17542010	0.01087
29	31795702	0.01970
30	44642598	0.02765
31	67690229	0.04193
32	53267289	0.03300
33	66360275	0.04111
34	129894448	0.08047
35	164453500	0.10187
36	90828097	0.05627
37	129516469	0.08023
38	114018646	0.07063
39	140521421	0.08705
40	173852845	0.10770
41	183146339	0.11345

**BQHIST file contents**

lotte@Uni-HPC:~Learning$ wc -l TrialSRR_minlen20.bqhist
102 TrialSRR_minlen20.bqhist
lotte@Uni-HPC:~Learning$ cat TrialSRR_minlen20.bqhist
#BaseNum	count_1	min_1	max_1	mean_1	Q1_1	med_1	Q3_1	LW_1	RW_1
0	15983014	0	34	31.06	31	31	33	23	34
1	15983014	2	34	30.70	31	31	34	16	34
2	15983014	2	34	28.71	27	31	34	10	34
3	15983014	2	37	33.42	33	35	37	10	37
4	15983014	2	37	34.49	35	35	37	19	37
5	15983014	2	37	34.86	35	35	37	23	37
6	15983014	2	37	35.08	35	36	37	25	37
7	15983014	2	37	35.12	35	37	37	25	37
8	15983014	2	39	36.66	35	39	39	25	39
9	15983014	2	39	36.65	35	39	39	25	39
10	15983014	2	39	36.66	35	39	39	25	39
11	15983014	2	39	36.39	35	38	39	19	39
12	15983014	2	39	36.47	35	38	39	23	39
13	15983014	2	41	37.58	36	39	41	24	41
14	15983014	2	41	37.66	36	39	41	25	41
15	15983014	2	41	37.57	36	39	41	24	41
16	15983014	2	41	37.28	36	39	41	17	41
17	15983014	2	41	37.45	36	39	41	23	41
18	15983014	2	41	37.60	36	39	41	24	41
19	15983014	0	41	37.58	36	39	41	24	41
20	15983014	0	41	37.58	36	39	41	24	41
21	15983014	0	41	37.52	36	39	41	24	41
22	15983014	0	41	37.46	36	39	41	23	41
23	15983014	0	41	37.39	36	39	41	22	41
24	15983014	0	41	37.34	36	39	41	21	41
25	15983014	0	41	37.09	36	39	40	18	41
26	15983014	0	41	37.05	36	39	40	18	41
27	15983014	0	41	36.45	36	39	40	10	41
28	15983014	0	41	36.70	36	39	40	16	41
29	15983014	0	41	36.72	36	39	40	16	41
30	15983014	0	41	36.55	36	39	40	15	41
31	15983014	0	41	36.42	35	38	40	15	41
32	15983014	0	41	36.46	35	38	40	15	41
33	15983014	0	41	36.42	35	38	40	15	41
34	15983014	0	41	36.20	35	38	40	9	41
35	15983014	0	41	35.95	35	38	40	9	41
36	15983014	0	41	36.29	35	38	40	15	41
37	15983014	0	41	36.24	35	38	40	9	41
38	15983014	0	41	36.14	35	38	40	9	41
39	15983014	0	41	35.97	35	38	40	9	41
40	15983014	0	41	36.12	35	38	40	9	41
41	15983014	0	41	36.09	35	38	40	9	41
42	15983014	0	41	36.03	35	38	40	9	41
43	15983014	0	41	35.82	35	38	40	9	41
44	15983014	2	41	35.93	35	38	40	9	41
45	15983014	2	41	36.10	35	38	40	9	41
46	15983014	0	41	35.82	35	38	40	8	41
47	15983014	0	41	35.87	35	38	40	9	41
48	15983014	0	41	35.97	35	38	40	9	41
49	15983014	0	41	35.91	35	38	40	8	41
50	15983014	0	41	35.47	34	38	40	7	41
51	15983014	2	41	35.55	34	38	40	7	41
52	15983014	2	41	35.56	34	38	40	7	41
53	15983014	2	41	35.51	34	38	40	7	41
54	15983014	2	41	35.18	34	38	40	2	41
55	15983014	2	41	35.16	34	38	40	2	41
56	15983014	2	41	34.94	33	38	40	2	41
57	15983014	2	41	34.87	33	38	40	2	41
58	15983014	2	41	34.83	33	38	40	2	41
59	15983014	2	41	34.71	33	38	40	2	41
60	15983014	2	41	34.53	33	37	40	2	41
61	15983014	2	41	34.36	33	37	40	2	41
62	15983014	2	41	34.15	32	37	40	2	41
63	15983014	2	41	34.03	32	37	40	2	41
64	15983014	2	41	33.92	32	37	39	2	41
65	15983014	2	41	33.65	32	36	39	2	41
66	15983014	2	41	33.18	31	36	39	2	41
67	15983014	2	41	33.09	31	36	39	2	41
68	15983014	2	41	32.85	31	36	39	2	41
69	15983014	2	41	32.72	31	35	39	2	41
70	15983014	2	41	32.42	31	35	38	2	41
71	15983014	0	41	32.24	31	35	38	2	41
72	15983014	0	41	32.11	31	35	38	2	41
73	15983014	0	41	31.94	31	35	37	2	41
74	15983014	0	41	31.71	30	35	37	2	41
75	15983014	0	41	30.53	29	34	36	2	40
76	15983014	0	41	30.96	30	34	36	2	40
77	15983014	0	41	31.00	30	34	36	2	40
78	15983014	0	41	31.11	30	34	36	2	40
79	15983014	0	41	30.98	30	34	36	2	40
80	15983014	0	41	30.80	30	34	36	2	39
81	15983014	0	41	30.67	30	34	36	2	39
82	15983014	0	41	30.53	30	34	35	2	39
83	15983014	0	41	30.31	30	34	35	2	39
84	15983014	0	41	29.73	29	34	35	2	39
85	15983014	0	41	29.65	29	34	35	2	38
86	15983014	0	41	29.50	29	34	35	2	37
87	15983014	0	41	29.18	29	34	35	2	37
88	15983014	2	41	29.06	29	34	35	2	37
89	15983014	0	41	28.79	29	34	35	2	37
90	15983014	2	41	28.79	29	34	35	2	37
91	15983014	0	41	28.68	29	33	35	2	36
92	15983014	0	41	28.49	29	33	35	2	36
93	15983014	0	41	28.34	29	33	35	2	36
94	15983014	2	41	28.11	29	33	35	2	36
95	15983014	0	41	27.54	27	33	35	2	36
96	15983014	2	40	27.41	27	33	35	2	36
97	15983014	2	40	27.13	26	33	35	2	36
98	15983014	2	40	26.88	26	33	35	2	36
99	15983014	2	39	26.50	25	33	35	2	36
100	15983014	0	39	24.30	19	30	34	2	35

**QHIST file contents**

lotte@Uni-HPC:~Learning$ wc -l TrialSRR_minlen20.qhist
102 TrialSRR_minlen20.qhist
lotte@Uni-HPC:~Learning$ cat TrialSRR_minlen20.qhist
#BaseNum	Read1_linear	Read1_log
1	31.063	26.885
2	30.698	24.777
3	28.714	20.556
4	33.421	24.922
5	34.491	29.127
6	34.864	29.350
7	35.080	29.805
8	35.118	30.052
9	36.658	30.166
10	36.652	30.040
11	36.662	30.395
12	36.389	28.285
13	36.465	29.323
14	37.579	29.032
15	37.663	30.035
16	37.574	29.065
17	37.282	27.307
18	37.446	29.248
19	37.598	29.421
20	37.576	29.258
21	37.578	29.198
22	37.519	28.767
23	37.462	28.431
24	37.387	27.753
25	37.338	27.351
26	37.086	25.540
27	37.050	25.438
28	36.447	23.315
29	36.702	24.383
30	36.717	24.062
31	36.555	23.301
32	36.422	22.903
33	36.459	22.835
34	36.423	22.594
35	36.197	21.844
36	35.950	21.385
37	36.290	21.849
38	36.243	21.372
39	36.145	20.975
40	35.969	20.686
41	36.124	20.852
42	36.087	20.575
43	36.031	20.427
44	35.823	19.860
45	35.934	20.178
46	36.099	20.175
47	35.817	18.978
48	35.872	19.512
49	35.970	19.451
50	35.906	19.198
51	35.474	18.051
52	35.548	18.662
53	35.564	18.477
54	35.506	18.265
55	35.178	17.768
56	35.164	17.730
57	34.940	17.352
58	34.868	17.281
59	34.826	17.139
60	34.712	16.925
61	34.534	16.697
62	34.362	16.509
63	34.147	16.268
64	34.035	16.146
65	33.921	15.996
66	33.645	15.748
67	33.178	15.393
68	33.091	15.371
69	32.846	15.159
70	32.724	15.043
71	32.425	14.815
72	32.241	14.697
73	32.106	14.588
74	31.937	14.460
75	31.713	14.296
76	30.530	14.150
77	30.962	14.091
78	30.998	13.886
79	31.111	13.882
80	30.976	13.727
81	30.800	13.572
82	30.667	13.455
83	30.529	13.311
84	30.307	13.133
85	29.732	12.769
86	29.653	12.709
87	29.497	12.535
88	29.178	12.281
89	29.063	12.139
90	28.794	11.890
91	28.790	11.777
92	28.681	11.583
93	28.490	11.350
94	28.342	11.122
95	28.105	10.832
96	27.542	10.419
97	27.407	10.170
98	27.129	9.831
99	26.876	9.495
100	26.500	9.128
101	24.302	8.607

**AQHIST file contents**

lotte@Uni-HPC:~Learning$ wc -l TrialSRR_minlen20.aqhist
42 TrialSRR_minlen20.aqhist
lotte@Uni-HPC:~Learning$ cat TrialSRR_minlen20.aqhist
#Quality	count1	fraction1
0	0	0.00000
1	0	0.00000
2	1003	0.00006
3	40192	0.00251
4	128895	0.00806
5	199688	0.01249
6	226919	0.01420
7	213066	0.01333
8	174075	0.01089
9	206481	0.01292
10	216097	0.01352
11	226341	0.01416
12	228745	0.01431
13	249807	0.01563
14	258218	0.01616
15	227851	0.01426
16	229582	0.01436
17	253982	0.01589
18	238962	0.01495
19	307747	0.01925
20	300699	0.01881
21	387786	0.02426
22	478053	0.02991
23	416609	0.02607
24	464175	0.02904
25	537010	0.03360
26	619228	0.03874
27	531987	0.03328
28	577129	0.03611
29	637311	0.03987
30	330176	0.02066
31	416774	0.02608
32	542824	0.03396
33	699675	0.04378
34	771215	0.04825
35	881653	0.05516
36	1175349	0.07354
37	1504781	0.09415
38	974260	0.06096
39	108547	0.00679
40	122	0.00001

Answer: Answers:

1. It seems bbduk is inconsistent in its indexing scheme, **BQHIST** uses zero-based index, and **QHIST** uses one-based index. I wouldn't call them "machine coding" vs "human coding", although it is true one-based systems are easier for humans to interpret, and zero-based systems are handier for computers to calculate.

2. Current version of Illumina FASTQ assigns scores between 0-40 to raw sequencing reads - see [FASTQ format][1]. Base qualities can be go above 40 in certain situations, e.g., when overlapping forward and reverse reads.

3. Probably rounding error.

4. I will let someone more knowledgeable answer this one.

5. Yes, but the drop in positions 2-3 is not very common.

6. https://www.biostars.org/p/398186/

7. Your interpretation seems correct, but here again there is likely rounding error, as the third column tallies to 0.999980.

[1]: https://en.wikipedia.org/wiki/FASTQ_format