Every driver in the kernel increases the patch-processing load on Linus. Linus is not an infinite resource. Having to go out on the web and track down a driver, perhaps choose between multiple variants, then integrate it into a kernel is beyond what most users will want to do. Kernel updates are much easier if the updated drivers come with the new kernel. Drivers that live with the kernel will better track changes in the kernel.
People notice right away when a kernel change breaks something.
There is also the view that drivers that are part of the kernel are less likely to be broken by kernel changes in the first place - developers will tend to see where particular features are used and try to avoid causing problems. There is no such feedback when the driver source is somewhere else. Device drivers in the kernel provide an important set of examples for kernel developers. Code that is in the kernel tree will encounter more eyeballs in general, and is more likely to be improved by people other than the maintainer. Drivers provided on vendor-supplied diskettes are all to likely to support only one distribution or packaging format.