Patent ID: 11960505
Assignee: SNOWFLAKE INC.
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 24:
25. A non-transitory computer-storage medium embodying instructions that, when executed by one or more processors, cause the one or more hardware processors to perform operations comprising:
identifying a database statement request to export data from a database stored in a network-based data system to an external datastore in export files having a user-specified file size limit, the user-specified file size limit being an individual file size limit for the export files to be exported to the external datastore;
determining, by one or more processors in the network-based data system, that generating the export files having the user-specified file size limit using a first set of threads within respective nodes of the database will exceed available memory for the respective nodes;
in response to determining that the first set of threads will exceed the available memory in generating the export files according to the user-specified file size limit in the database statement request, activating a second set of the threads to export the data, the second set of the threads having fewer threads than the first set of threads and being a subset of the first set of threads;
generating, using the second set of the threads of the one or more processors, the export files according to the user-specified file size limit in the database statement request, generating the export files including:
generating, using the second set of threads within each node, a first set of merge files by performing a table scan, a projection operation, and a partition unload operation to generate the first set of merge files,
generating, using the second set of threads within each node, a second set of merge files including hashing the first set of merge files using a partition key specified in the database statement request based on respective destination partition, and
querying, using the second set of threads within each node, a corresponding portion of merged data in the second set of merge files to generate the export files; and
transmitting the export files to the external datastore.