Patent ID: 8719271
Filing Date: 2014-05-06
Classification: G06F

Abstract:
1. A computer-implemented method of handling a data profiling request by using a distributed file system, wherein the distributed file system is a file system that is distributed among several physical nodes managed by a master node, wherein the master node distributes files using information from a metadata repository, wherein the metadata repository comprises metadata for the master node and the several physical nodes, and wherein the computer-implemented method comprises: extracting, by a computer, tabular data from a data source of the distributed file system, wherein the tabular data is stored in a matrix structure of columns and rows containing data values; splitting said tabular data by columns from said matrix structure, wherein each column is stored in a separate file, and wherein the columns are stored in multiple files; storing the multiple files on a set of physical nodes of the distributed file system, wherein each file from the multiple files is stored at least once on two different physical nodes; receiving a data profiling request from an application client, wherein the data profiling request indicates a profiling analysis type and data files to be profiled; determining from the data profiling request which groups of files are requisite files that are required to be stored on a same physical node in order to perform the profiling analysis, wherein said determining is performed by a master node; creating jobs by the master node, wherein each job created uses one of the requisite files that has been determined by the master node; searching for physical nodes holding the requisite files for each job using information contained in the metadata repository; determining a selected physical node to execute each job according to one of the two following cases: executing the job in a selected physical node, wherein the selected physical node is from the first physical node and the second physical node.