Project: datamanagement
Issue DM-3495: X16 Large Results
As described in [LDM-135|http://ldm-135.readthedocs.org/en/master/#query-access-related-requirements] high volume queries can return large results (~6 GB per query). Current version of Qserv poorly supports large results. We have observed that a single query that returns ~40 million or two-column rows (~0.5 GB) requires nearly 16 GB of RAM in czar, and czar uses 100% of CPU for extended period of time. This is most likely related to aggregating results - results are currently cached in memory before they are returned to the proxy. This epic involves redesigning this part of Qserv. One reasonable approach would be to write to separate tables from each worker, and deliver the final result as UNION of these tables. Or, perhaps we could use a shared files system and each worker would write results directly there, without burdening czar. Deliverable: demonstration involving 8 queries each returning 2 GB result handled through a single czar. Czar should not use excessive amount of RAM or CPU.
Story points: 19