Additions to JsonUtil.java, partial update of new kind and section changes.
Updates to JsonUtil
.
Metadata functionality fully implemented. Unknown fields in a Json object will be dumped to metadata fields. Partial commit, debugging.
Metadata functionality fully implemented. Unknown fields in a Json object will be dumped to metadata fields. Partial commit, debugging.
Partial commit, debugging.
Finished metadata and json to and from concrete.
Type changes and switching out the readers.
We are rolling out a new set of classes to read and write Concrete objects to/from files. This should be a simple change for most people. In Java, there are two new classes. You should do this for ALL reading/writing of Concrete objects, even if you are only storing one per file.

edu.jhu.hlt.concrete.io.ProtocolBufferReader- pass in a filename/object, including compressed files, and the Concrete object you expect to find in the file. It acts as an iterator (hasNext() and next()) to return Message objects. You then cast them to the expected Concrete type (Vertex, Communication, Discourse, etc.)

edu.jhu.hlt.concrete.io.ProtocolBufferWriter- pass in a filename/object and then call write(message) to save things to a file.

There is similar support in Python in the new concrete-python/src/concrete/proto_io in concrete.

Why are we doing this? ProtocolBuffers currently have poor support for writing multiple objects to the same file. They have support in Java, but not in any other language. Since we have users that write in other languages (Python) we need a cross-language solution.

These new classes implement such a solution. Furthermore, by making reading and writing easy, this means that we should move to storing all our concrete objects in a single file (where appropriate) as opposed to multiple files. This will speedup disk reads/writers since our file system is optimized for reading large files instead of many small files.

Along with this new file format comes several advantages:
- cat: you can now cat multiple protocol buffer files together and it will work just file.
- head/tail/shuffle/split/count/print/etc.- the python code supports all these operations. Fast and easy.

To use Python, you need to do two things:
1) Compile the Python protoc files and move Python source to target: compile_python.sh (this is in the root of the concrete repo)
2) Add the Python files to your path:
export PYTHONPATH="/path/to/repo/concrete/concrete-python/target:$PYTHONPATH"
Merge branch 'master' of github.com:hltcoe/concrete
Updated ProtocolBufferReader to implement Iterator<Message>. This seems
reasonable given that it was already implementing the relevants methods,
and makes dealing with it a little more pleasant.
Merge branch 'master' of https://github.com/hltcoe/concrete
Merge pull request #22 from tturpen/master

JsonUtil New delimited readers implemented and some bug fixes
Merge pull request #21 from JackSullivan/master

Updated ProtocolBufferReader to implement java.util.Iterator<Message>