Patent ID: 11934655
Assignee: SALESFORCE, INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G  H | IPC G

Claim 18:
19. A non-transitory computer readable medium encoded with programming instructions configurable to cause a processor to perform a method for storing multiple petabytes of raw log data from cloud-based source data buckets into internal partitioned data storage in a data lake, the method comprising:
deploying a log partitioner cluster comprising a plurality of log partitioner service instances for storing the raw log data in a partitioned manner;
associating one or more of the source data buckets to each of a plurality of deployed log partitioner service instances, the associating comprising providing associated configuration metadata from a configuration service to a deployed log partitioner service instance to initiate the log partitioner service instance;
fetching, via the log partitioner cluster, raw log data from associated source data buckets based on the associated configuration metadata provided to the log partitioner service instances, wherein associated configuration metadata comprises source data bucket specific values for a partition key for fetched raw log data from a specific source data bucket
wherein the associated configuration metadata provides instructions for use by a log partitioner service instance to onboard raw log data and place log data to destination storage on defined paths; and
placing fetched raw log data, via the log partitioner cluster, in the internal partitioned log storage based on the partition key having a plurality of dimensions, wherein the dimensions comprise one or more of a destination storage root dimension, a source type dimension, a time key dimension, an environment dimension, and a hash group dimension.