tatsuhiro-t/aria2 — pull request #115 by nmaier (2013-08-12 18:54:12+00:00)
Title: Unbuffered disk writes can cause massive fragmentation

I loaded a couple of thousand smallish (<1MB) files with 6 concurrent downloads to an NTFS partion on Win7 and had plenty of consecutive free space. The downloaded files had a little over 8 fragments per file after the queue completed as analyzed by contig.exe.

The main reason seems to be that aria2 does not buffer writes at all (not even stdlib buffers) and leaves this entirely to the OS. ProcessMonitor saw direct WriteFile calls with usually 1400 length, i.e one per received TCP segment, which is bad for a couple of reasons:
- Many small writes cause many syscalls, which isn't exactly great performance-wise.
- Many small writes may cause excessive fragmentation, as demonstrated here.
- The non-power-of-two size of the writes doesn't play nicely with cluster sizes and file system implementations.

I actually implemented a PoC write buffer in AbstractDiskWriter, which indeed reduces the fragmentation significantly. That PoC still has a couple of problems, such as AbstractDiskWriter being a suboptimal place for buffers and thus not working well when using --split.

There should be some buffering, ideally a per-connection receive buffer (instead of per-file write buffer).