RevolutionAnalytics/RHadoop — Issue #41: binary native format
Author: piccolbo
Created: 2012-01-11 22:50:57+00:00
State: open
Labels: complexity-medium, importance-medium, pkg-rmr, type-improvement

Now that the support for binary formats is becoming available (in branch binary-io) it is possible to change the native R format from a text format to a binary format. This would be more compact end efficient and avoid quoting issues and hopefully make #36 go away.
This is the plan: serialize R objects with serialize, then used the typed bytes writer with type code 0 or one alias for that (type codes above 50 are reserved for application specific serialization). Then there is the question of whether to write the typedbytes directly to disk or to write them in a sequence file. Sequence files are the HDFS standard, so that seems like a good reason to use it. It has a header with version information and a key value structure plus some sync information. One drawback is that we don't have a sequence file reader in R and the only way to access them is through streaming, so that would leave from.dfs and to.dfs out in the cold. A workaround for that is to replace their implementation with a streaming job (at least for the binary case). It looks like the new dumptb command could be exactly what is needed (so one would have to do a hadoop streaming dumptb instead of a hadoop dfs -get, but it would be the same from there.