JohnLangford/vowpal_wabbit — pull request #76 by nicknussbaum (2012-12-19 23:52:13+00:00)
Title: Allow multi pass training from Library API

it also adds a small test to the cs_test/Program.cs
The point of this code is to allow --passes to work from the library.
It requires enabling the cache.

Status As of now, prediction results seem off but it seems to do
the right number of operations.

This patch allows --passes to work from the library VW initialization
with a cache file specification

changed parsed_atomic_example to have an optional do_read that defaults to
true. If false it will not read the example (which the library loaded) but
will cache it and also do ngram processing which was missing.

parsed_atomic_example is now called from VW:read_example, and various
VW:import_example functions before the setup_example.

on VWdll.cpp VW_Finish if the pass count is > 1 some values are adjusted
and then the start_parser, driver end parser loop is called to read the
remaining passes from the cache.

the main_parse_loop was changed to set examples to the
parsed_examples_count which will be nonzero if the library did anything
and zero in the normal .exe call. It also skips the all_passes_complet ==
all->numpasses for the library call. I'm not sure what this does, but it
seems to double the number of pass reads. I'm using stdin_off as a proxy
for "called by library" heree. If this can't be avoided perhaps stdin_off
should be named "invoked by library".

By calling the parser loop after the reads, the multi pass from cache is
done, but possible other pre-read sequencing is missed for other
functionality. A more general fix would be to make a mock reader which is
really a co-routine for the library functions that are invoked between
init and finish. This would allow stuff like merge caches before
prediction and other stuff to work.

The prediction file values do appear off. I'm trying to figure out if this
is user error or something that happens from the library calling, or if
I'm missing some processing.

Also this hasn't been tested with Unix. I'm a uncertain if the
fiddling to get it to do the passes has all the multi-thread
initialization done right.