A computer-implemented method, comprising: transmitting, from a primary control node (902, 1002) connected to one or more worker nodes (910, 912, 914, 916, 1010, 1012, 1014, 1016) on a communications grid (900, 1000), worker instructions related to a project being executed by the one or more worker nodes; generating, by the primary control node, a snapshot of the communications grid, wherein the snapshot of the communications grid includes a project status of each of the one or more worker nodes, wherein a project status of a worker node includes a status of a portion of the project being executed by the worker node in the communications grid, and wherein the snapshot includes a project checkpoint that indicates a point during execution of a portion of the project; determining, by the primary control node, that a failed worker node of the one or more worker nodes has failed, wherein the failed worker node failed at a failure time; determining that the failed worker node failed while the project was pending; identifying a project status of the failed worker node using the snapshot of the communications grid, wherein the project status of the failed worker node includes a status of a portion of the project being executed by the failed worker node at the failure time, and identifying a project checkpoint of the failed worker node from the snapshot that indicates a point during execution of a portion of the project prior to failure of the failed worker node; and transmitting updated worker instructions, wherein the set of instructions includes the project checkpoint that indicates the point during the execution of a portion of the project prior to the failure of the failed worker node, and wherein the updated worker instructions include the project status of the failed worker node and updated instructions related to the project being executed by the one or more worker nodes, wherein a functioning worker node completes, using the updated worker instructions, execution of the portion of the project being executed by the failed worker node from the project checkpoint. transmitting, from a primary control node (902, 1002) connected to one or more worker nodes (910, 912, 914, 916, 1010, 1012, 1014, 1016) on a communications grid (900, 1000), worker instructions related to a project being executed by the one or more worker nodes; generating, by the primary control node, a snapshot of the communications grid, wherein the snapshot of the communications grid includes a project status of each of the one or more worker nodes, wherein a project status of a worker node includes a status of a portion of the project being executed by the worker node in the communications grid, and wherein the snapshot includes a project checkpoint that indicates a point during execution of a portion of the project; determining, by the primary control node, that a failed worker node of the one or more worker nodes has failed, wherein the failed worker node failed at a failure time; determining that the failed worker node failed while the project was pending; identifying a project status of the failed worker node using the snapshot of the communications grid, wherein the project status of the failed worker node includes a status of a portion of the project being executed by the failed worker node at the failure time, and identifying a project checkpoint of the failed worker node from the snapshot that indicates a point during execution of a portion of the project prior to failure of the failed worker node; and transmitting updated worker instructions, wherein the set of instructions includes the project checkpoint that indicates the point during the execution of a portion of the project prior to the failure of the failed worker node, and wherein the updated worker instructions include the project status of the failed worker node and updated instructions related to the project being executed by the one or more worker nodes, wherein a functioning worker node completes, using the updated worker instructions, execution of the portion of the project being executed by the failed worker node from the project checkpoint.