We'll need to discuss about that as a general discussion for ZF3, but I'd be in favor, to avoid the duplication of folders, to do what most of us do in modules: having a top "Factory" class that holds all the factory (usually, I don't expect more than 4-10 factories per component so splitting even more is a bit overkill).

Basically it will become "Zend\Paginator\Factory\DbSelectAdapterFactory".