nathanmarz/cascalog — Issue #20: Memory-source-tap should upgrade to a temp seqfile for large seqs
Author: sritchie
Created: 2011-12-06 02:56:20+00:00
State: open

My notes from another ticket:

I wanted to ask what you thought of allowing larger lazy sequences to be used with union and combine. I've got some code that you helped me with that takes "for" style destructuring bindings and a final tuple form, stages all tuples into HDFS (or LFS, if Hadoop's in local mode), and returns a tap into the temporary file.

One use case here might be an upgrade of the current memory-source-tap. The memory-source-tap, when handed a sequence, instead of calling vec could take up to some in-memory count limit of items. If no tuples are remaining after this, we're done, and the memory-source-tap proceeds as usual. If tuples are left, these tuples would get staged into a temp folder on the filesystem, and the user would get (combine memory-tap temp-seqfile-tap) to use in the query. Behind the scenes support for huge sequences!