Reliable high-performance file-based data collection and distribution system