A computer-implemented method for repartitioning data in a distributed computing system that includes: one or more sources, each source comprising an initial partition of the data, two or more sinks, each sink comprising a final partition of the data, and a transient memory comprising a data file including one or more segments and one or more blocks within each of the segments, wherein each block is associated to a particular sink of the two or more sinks, the transient memory configured to transiently store data in-memory, the method comprising: allocating, by a source of the one or more sources, a segment of the data file of the transient memory for exclusive access by the source; writing, by the source, data from the initial partition to two or more blocks within the allocated segment of the data file of the transient memory, wherein a portion of the initial partition of the data, the portion being destined by the repartitioning to the particular sink, is written to the first blocks of the two or more blocks associated with the particular sink of the two or more sinks, within the allocated segment of the data file of the transient memory; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by the two or more sinks; and reading, by the particular sink of the two or more sinks, the particular block of the published segment of the data file of the transient memory, wherein the transient memory comprises a marker file, wherein for every segment in the data file there is a segment marker in the marker file, wherein each segment marker is in the same order as the segments in the data file, wherein the marker file includes metadata used to indicate the status of each segment in the data file, wherein the metadata comprises information about segment length and segment commit status, and wherein publishing, by the source, the segment of the data file to be accessible for reading by the two or more sinks comprises updating the metadata in the marker file, and wherein the two or more sinks periodically check the updated metadata in the marker file to determine whether the segment of the data file is relevant to the two or more sinks, and if the segment of the data file is relevant to the two or more sinks, reading the segment of the data file by the two or more sinks, and sending, by the two or more sinks, the segment of the data file to a client receive queue to be consumed by a user. allocating, by a source of the one or more sources, a segment of the data file of the transient memory for exclusive access by the source; writing, by the source, data from the initial partition to two or more blocks within the allocated segment of the data file of the transient memory, wherein a portion of the initial partition of the data, the portion being destined by the repartitioning to the particular sink, is written to the first blocks of the two or more blocks associated with the particular sink of the two or more sinks, within the allocated segment of the data file of the transient memory; publishing, by the source, the segment of the data file of the transient memory to be accessible for reading by the two or more sinks; and reading, by the particular sink of the two or more sinks, the particular block of the published segment of the data file of the transient memory, wherein the transient memory comprises a marker file, wherein for every segment in the data file there is a segment marker in the marker file, wherein each segment marker is in the same order as the segments in the data file, wherein the marker file includes metadata used to indicate the status of each segment in the data file, wherein the metadata comprises information about segment length and segment commit status, and wherein publishing, by the source, the segment of the data file to be accessible for reading by the two or more sinks comprises updating the metadata in the marker file, and wherein the two or more sinks periodically check the updated metadata in the marker file to determine whether the segment of the data file is relevant to the two or more sinks, and if the segment of the data file is relevant to the two or more sinks, reading the segment of the data file by the two or more sinks, and sending, by the two or more sinks, the segment of the data file to a client receive queue to be consumed by a user.