Patent ID: 8719271

Claim:
A computer-implemented method of handling a data profiling request by using a distributed file system, wherein the distributed file system is a file system that is distributed among several physical nodes managed by a master node, wherein the master node distributes files using information from a metadata repository, wherein the metadata repository comprises metadata for the master node and the several physical nodes, and wherein the computer-implemented method comprises: extracting, by a computer, tabular data from a data source of the distributed file system, wherein the tabular data is stored in a matrix structure of columns and rows containing data values; splitting said tabular data by columns from said matrix structure, wherein each column is stored in a separate file, and wherein the columns are stored in multiple files; storing the multiple files on a set of physical nodes of the distributed file system, wherein each file from the multiple files is stored at least once on two different physical nodes; receiving a data profiling request from an application client, wherein the data profiling request indicates a profiling analysis type and data files to be profiled; determining from the data profiling request which groups of files are requisite files that are required to be stored on a same physical node in order to perform the profiling analysis, wherein said determining is performed by a master node; creating jobs by the master node, wherein each job created uses one of the requisite files that has been determined by the master node; searching for physical nodes holding the requisite files for each job using information contained in the metadata repository; determining a selected physical node to execute each job according to one of the two following cases: in a first case in which all files required by a particular job are found in a first physical node and other physical nodes, wherein the first physical node is currently being utilized less than said other physical nodes, selecting the first physical node to execute the particular job; in a second case in which all requisite files required by the particular job are not found on a single physical node, assimilating the requisite files into a second physical node, wherein the second physical node has been predetermined to have been utilized less than any other physical node; and executing the job in a selected physical node, wherein the selected physical node is from the first physical node and the second physical node.