Framework to intelligently manage many parallel tasks