Patent ID: 9729610
Date: 2017-08-08
CPC Classifications: G06F,H04L

Claim:
1. A computer-implemented method performed by one or more processors when running executable code, the computer-implemented method comprising: assigning a plurality of compute nodes of a distributed computing system to execute one or more tasks corresponding to a job assigned to the plurality of compute nodes; provisioning each compute node with a middleware layer that permits each compute node to interface with a given task processor regardless of the compute node's underlying configuration, and the middleware layer comprising a file system interception layer; wherein the file system interception layer automatically downloads any missing data files to local storage of the compute node as and when any missing data files are needed to process task dependent files so that the automatic download does not download file dependencies that are not required in order to process the one or more tasks assigned to a compute node, and wherein automatically downloading any missing data files at the file system interception layer comprises: