NOJIRA Putting IndexedDataset, Readers, Writers, Drivers, CLI Parser into core math-scala

Still cleaning up comments and removing some now unneeded code but the general idea will be checked in unless someone finds problems. Note the H2O stubs and how they are flagged as unimplemented. Is this what we want?

Mail describing some naming conventions used in this PR

=====================================================

@pferrel

This is a reasonable train of though and your names seem fine. However text _is_ actually the persistent representation of what I was calling an extended-DRM, which should probably just be called an IndexedDataset.The difference between a DRM and IDS is that the later will probably need to support more than one persistence format and persists any external IDs for row and column.

Therefore following your train of thought I’d change the IndexedDataset names to:

Package level
// lines = rows as used by spark-rowsimilarity
indexedDatasetDfsRead(src: String, schema: Schema = DefaultReadSchema): IndexedDataset

// lines = elements as used by spark-itemsimilarity
indexedDatasetDfsReadElements(src: String, schema: Schema = DefaultElementsSchema): IndexedDataset
 
Method level
// lines = rows
indexedDataset.dfsWrite(dest: String, schema: Schema = DefaultWriteSchema)

Once the read is performed the DRM is a CheckpointedDrm contained in the IndexedDataset. This will in effect allow read/write as sequence file or simple CSV (uses only split for parsing)

BTW this turns out to be rather convoluted to implement in a core way (oh, bother) and leaves some empty stubs in H2O


On Sep 26, 2014, at 11:30 AM, Dmitriy Lyubimov <notifications@github.com> wrote:

to be a bit more concrete, there's indeed slight discrepancy between write and read names, but semantically they are what they say they are, i.e. they are persisting drm to hdfs.

To be even more concrete, i am probably for simply package-level drmDfsRead() and method-level dfsWrite() names.

The convention here is that all drm-related package-level routines start with drm prefix so we don't easily mix these things with other things in global scope.

Now, everything else, including reading/writing CSV formats, is an export operation (as opposed to persistence). Consequently, proper names are perhaps along the lines drmImportCSV and exportCSV respectively. Import and export emphasizes the fact that format is not native, loses a lot of coherency enforcement, and requires a lot of validation while parsing back.

—
Reply to this email directly or view it on GitHub.