Patent ID: 11956330
Assignee: INTERNATIONAL BUSINESS MACHINES CORPORATION
Field: Digital communication (Electrical engineering)
Classification: CPC H | IPC H

Claim 0:
1. A method of adaptive data fetching from networked storage in a networked computing environment, the method comprising:
receiving, at a network storage system in the network computing environment, a request from an analytical node for a set of data for performing a defined job, the network storage system including a storage cluster of servers and data storage devices;
analyzing said request to determine whether said request includes a specification of the defined job;
determining whether another job having a specified relationship to said defined job can be identified;
when the request does not include a specification of the defined job and no other job having the specified relationship to said defined job is identified, loading the whole requested data from the networked storage in a response to said request for the one analytical node to use said whole requested data to perform the defined job, and
when the request includes the specification of the defined job or when said other job having the defined relationship with the defined job is identified, using said specification or using said other job having the defined relationship with the defined job to identify in the networked storage a subset of the set of data for performing the defined job, said subset of the data being less than the set of data;
after the identifying the subset of the set of data requested by the one remote analytical node, the network storage system adaptively fetching said subset of the data at the network storage system and transmitting the adaptively fetched subset of the data to the one analytical node; and
loading the subset of the data to the analytical node in response to the request in an order based on the sequence in which the subset of the data are projected to be accessed in the defined job, wherein the defined job is an analytics job; and the method further comprises:
storing a data usage pattern for the analytics job, the data usage pattern comprising a pattern of data fetched from the network storage system, said data comprising files and data blocks fetched from data tables stored in the network storage system;
updating the stored data usage pattern for the analytics job by monitoring data access by the remote analytical node during running of the analytics job; and
the identifying a subset of the set of data requested by the remote analytical node includes learning an empirical model to describe access patterns of the remote analytical node based on captured correlations between users and stored data, and wherein a subsequent request from the remote analytical node further comprise hints based on an analysis by the remote analytical node of earlier tasks run by the remote analytical node, about which data to send to the remote analytical node, the storage cluster analyzing the request and the hints to determine the subset of data needed to respond to the subsequent request,
wherein the request and subsequent analytical cluster request is a request for one or more data tables from the storage cluster of servers, each of the data tables including a plurality of blocks of data, the identifying a subset of the set of data includes identifying selected blocks of the data tables, said method further comprising:
conducting analytics based on an analytics template, said analytics template specifying one or more categories for the analytics, wherein multiple analytics that belong to a same category have similar data access patterns;
comparing the request with past runs in the same category, and
suggesting blocks that were frequently used in the past runs for a current analytics request; and the loading the subset of the data includes loading the selected blocks of the data tables in the response.