Project: datamanagement
Issue DM-1692: Implement "unlimited" result size handling
DM-854 exposed an issue in handling large results. Result rows are returned from worker to czar in protobufs messages. However, protobufs messages should not be larger than some number of megabytes, according to protobufs documentation. IOW, protobufs is not designed to handle messages on the order of hundreds of megabytes. There may be some code in the protobufs implementation that does not scale beyond messages of a few megabytes. Hence, in order to send larger amounts of result rows, we need to use multiple messages. The current protobufs definition for result messages includes a placeholder for chaining result messages, but there is no code on the czar or worker that implements result chaining. The scope of this ticket is to implement message chaining for results on the worker and on the czar, in such a way that it places no limits on the overall number of result rows (if there is a limit, it should be no smaller than quadrillions and be well-documented).
Story points: 15