Unix & Linux: Is there a way to determine the optimal value for the bs parameter to dd?
Channel: Roel Van de Paar
Date: 2020-08-17

Unix & Linux: Is there a way to determine the optimal value for the bs parameter to dd?


The Question: On occasion I've seen comments online along the lines of "make sure you set
'bs=' because the default value will take too long," and my own extremely-
unscientific experiences of, "well that seemed to take longer than that other
time last week" seem to bear that out. So whenever I use 'dd' (typically in the
1-2GB range) I make sure to specify the bytes parameter. About half the time I
use the value specified in whatever online guide I'm copying from; the rest of
the time I'll pick some number that makes sense from the 'fdisk -l' listing for
what I assume is the slower media (e.g. the SD card I'm writing to).
For a given situation (media type, bus sizes, or whatever else matters), is
there a way to determine a "best" value? Is it easy to determine? If not, is
there an easy way to get 90-95% of the way there? Or is "just pick something
bigger than 512" even the correct answer?
I've thought of trying the experiment myself, but (in addition to being a lot
of work) I'm not sure what factors impact the answer, so I don't know how to
design a good experiment.

Solutions: Please watch the whole video to see all solutions, in order of how many people found them helpful

== This solution helped 29 people ==
dd dates from back when it was needed to translate old IBM mainframe tapes, and
the block size had to match the one used to write the tape or data blocks would
be skipped or truncated. (9-track tapes were finicky. Be glad they're long
dead.) These days, the block size should be a multiple of the device sector
size (usually 4KB, but on very recent disks may be much larger and on very
small thumb drives may be smaller, but 4KB is a reasonable middle ground
regardless) and the larger the better for performance. I often use 1MB block
sizes with hard drives. (We have a lot more memory to throw around these days
too.)

== This solution helped 61 people ==
There's but one way to determine the optimal block size, and that's a
benchmark. I've just made a quick benchmark. The test machine is a PC running
Debian GNU/Linux, with kernel 2.6.32 and coreutils 8.5. Both filesystems
involved are ext3 on LVM volumes on a hard disk partition. The source file is
2GB (2040000kB to be precise). Caching and buffering are enabled. Before each
run, I emptied the cache with sync; echo 1 ＞|/proc/sys/vm/drop_caches. The run
times do not include a final sync to flush the buffers; the final sync takes on
the order of 1 second.
The same runs were copies on the same filesystem; the diff runs were copies to
a filesystem on a different hard disk. For consistency, the times reported are
the wall clock times obtained with the time utility, in seconds. I only ran
each command once, so I don't know how much variance there is in the timing.
 same diff
t (s) t (s)
dd bs=64M 71.1 51.3
dd bs=1M 73.9 41.8
dd bs=4k 79.6 48.5
dd bs=512 85.3 48.9
cat 76.2 41.7
cp 77.8 45.3
Conclusion: A large block size (several megabytes) helps, but not dramatically
(a lot less than I expected for same-drive copies). And cat and cp don't
perform so badly. With these numbers, I don't find dd worth bothering with. Go
with cat!

With thanks & praise to God, and with thanks to the many people who have made this project possible! | Content (except music & images) licensed under cc by-sa 3.0 | Music: https://www.bensound.com/royalty-free-music | Images: https://stocksnap.io/license & others | With thanks to the Stack Exchange Network (http://unix.stackexchange.com/questions/9432). Trademarks are property of their respective owners. Disclaimer: All information is provided "AS IS" without warranty of any kind. You are responsible for your own actions. Please contact me if anything is amiss at Roel D.OT VandePaar A.T gmail.com.