Use only one background thread to scan directories.

Since IO is the bottleneck, many threads won't help. Just keep all the
scanning IO in the background so it doesn't affect startup time. Since
there is less IO contention, this should be faster and simplifies the
code.