more optimizations and processing for mining