rewritten for large-scale data collection