Patent ID: 11934655
Assignee: SALESFORCE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 0:
1. A cloud-based data repository system, the data repository system comprising:
memory;
at least one processor;
internal partitioned data storage for storing multiple petabytes of log data;
a data ingestion pipeline for use in ingesting multiple gigabytes of raw log data per second for storage in the internal partitioned data storage;
a configuration service for providing configuration metadata regarding source data buckets from which the raw log data are retrieved; and
a log partitioner service comprising a controller configured to:
deploy a log partitioner cluster comprising a plurality of log partitioner service instances for storing the raw log data in a partitioned manner; and
associate one or more of the source data buckets to each of a plurality of deployed log partitioner service instances, wherein to associate a log partitioner service instance the controller is configured to provide associated configuration metadata from the configuration service to a deployed log partitioner service instance to initiate the log partitioner service instance;
wherein each log partitioner service instance is configured to fetch raw log data from associated source data buckets based on the associated configuration metadata provided to the log partitioner service instance, wherein associated configuration metadata comprises source data bucket specific values for a partition key for fetched raw log data from a specific source data bucket;
wherein the associated configuration metadata provides instructions for use by a log partitioner service instance to onboard raw log data and place log data to destination storage on defined paths; and
wherein each log partitioner service instance is configured to place fetched raw log data in the internal partitioned log storage based on the partition key having a plurality of dimensions, wherein the dimensions comprise one or more of a destination storage root dimension, a source type dimension, a time key dimension, an environment dimension, and a hash group dimension.