HUGE CHANGESET -- expect bugs, regressions, etc.
Main thing is the addition of the HTTP/WEB INTERFACE adatptor code.

Other changes are as follows.

New tasks-lists, task-delete, task-move messages for the use with network controller(s).
Deletion of the 'swapinqueue' code (on the server side), in place of a generic "move task in list" mechanism.

In addition to periodic save-points, the convergence-state message is now also saved at the end of the convergence process.

Message-ids have been somewhat re-partitioned to allow for backwards compatibility with sensitive apps such as update_client, whilst also allowing for the addition/refactoring of messages in other apps such as controller(s).

Further refactoring io_wrapper and io_driver class structure to allow for multiple io-drivers automatic keeping 'in-scope' of the driver object as per shared_from_this semantics in asio's programming model (this should allow http-adaptor functionality which mantains two sockets in a given peer connection -- one for binary interface to the processing server, another for the http/web-browser client).

Refactoring io_wrapper et. al. w.r.t. callback settings (now io_wrapper will not have a pre-installed virtual on_{read,handshake,write,etc.) methods -- client classe must set such things explicitly). This is not so much of a problem since most of the io-event chains are usually handled with explicit callback settings anyway (i.e. default 'on_read' is rarely enough by itself anyways).

Minor re-factoring of the error_callback_ usage (and callback's signature) -- allowing for the content of the recoverable-netcpu-message exception to be propagated, this allows cross-protocol communication of error (e.g. http protocol portion reporting to web-client any of the native protocol thrown exception content such as wrong arguments/their-values during controller's run when loading a job).

Refactoring of the controller code structures and mechanisms w.r.t. how the arguments are packaged and passed around -- this is due to the introduction of the HTTP code and intention of reusing as much of the common code/logic as possible. This has also lead to some changes in the name of some CLI args (e.g. --file to --filepath and --filedata, etc.)

Adding a how-to for converting .pem certificate and key to the .p12 format for use as a client certificate in the web browsers for cross-authentication purposes.

Minor change w.r.t. how spreadsheet/csv grid object deals with memory growth (should not be exponential any more -- otherwise a potential worst case scenario could have been that there'd be too much unused/overallocated memory by a factor of 4 -- double the columns and double the rows).

Also had changed the ctor for the csv grid object -- now accepts stream object (as opposed to the filepath string from which an internal filestream was created).

Moreover, minor change w.r.t to parsing code-flow in csv parsing class (don't recall why, probably just for minor elegance/efficiency reasons).

Replacing ::std::auto_ptr  with ::std::unique_ptr in io_driver.h (auto_ptr is deprecated).

Minor proactive bug-fixing in on_handshake (w.r.t. 'canceled_' and exceptions).

Bug-fix in converger_1/controller.h -- arguments list erasing was (I think) wrong...

Taken out some deprecated header include directives and stray, unused code dependencies (e.g. Eigen/Core, csv.h in model-specific controller.h files).

fastsize -- very minor addition of visibility to const_iterator type.

A minor addition on building libarchive to freebsd howto txt file. This is currently probably not used, but will be of benefit when http uploads will also include zipped (.zip) archived file format as opposed to currently the only-supported uncomrpessed .csv formats.

Bug-fix in the BSDmakefile -- the autodependency inclusion did not follow subdirectory (1 level deeper for the sub-projects such as gmnl, http, etc.)

Some of the very minor, transient (likely to be changed anyways), changes are the "defalting to localhost" addresses for the servers -- to enable quicker testing during development (since this is a coding repository afterall). But such things are not expected to remain consistent -- may well be a likely scenario when having changed the values such are accidentally commited to the repo -- no big deal.

On the server side the tasks numeric id is now a 64bit int -- a quick-hack type of a solution to prevent cases where old jobs are purged and the new ones added with the "same as old" taskname thereby possibly leading to a confusion with the person who had added the job(s) in the first place.
Thought in progress -- removing the mandatory nature of the 'sort' option in the controller during loading of the tasks.
Some main changesets (do expect bugs, regressions, etc.):

Allowing datasets to have a varying number of alternatives per any given choice set. For the time-being the requirement is that there shall be less than 255 alternatives in any given choice set (seems reasonable enough w.r.t. current deployment practices in our field). More testing is yet to be done as to the correctness of the implementation -- this will take some time indeed.

Subsequent re-factoring, culling and simplification of the simulation code everywhere.

Introducing a new feature -- processor is able to cache the bulk_msg in a local-to-processor computer/filesystem. This caters for situations where processors' network connection is interrupted/dropped and then is resumed again (e.g. user closes laptop and such goes into hibernation; or user's Internet connectivity is interrupted during laptopts transfer from home to office; and so on). In the previous code-revision, the processor would re-connect and re-download (from server, i.e. remotely) the bulk mesage. The caching feature allows processor to instead re-use the already(previously)-saved bulk message (which houses the original, to-be-estimated, dataset). In cases of laptops and user-paid-for Internet connections, this is of some importance to the user's Internet usage quate (esp. if in hotels, etc).

Making the approximation of the exponent calculations to be a user-controlled option at run-itme (during the act of adding of the new task). Allows for a more robust testing/validation of the convergence runs if need be.

The server now will automatically and periodically run wakeonlan.sh script (currently from the current working directory, todo -- make it more configurable, and test if ). The purpose being to automatically wake-up any of the needed machines (such machines are also expected to be configured to hibernate, go into suspension mode whenever appropriate). It is responsibily of the sysadmin to provide such a script. A demo one is supplied in demo_wakeonlan.sh, sysadmin should edit the file with the right values and rename it to wakeonlan.sh (if no waking up on LAN is needed, the sysdamin can provide a blank script).

-----------------------------------------------------------------

Other additions:

New value for the --del option: *:eq:5 where the first field is allowed to be a wildcard -- in such a case any cell represented by the attribute (--x option) will trigger a row deletion from the to-be-processed dataset.

Expanded some CLI options (--resp to --respondent) and have added new options --choice_set and --alternative. Deleted the --alts option (as number of alternatives is now deducted automatically from the '--alternative' column).

-----------------------------------------------------------------

Bug-fixes:

Bug-fix: in converger_1/server.h the end_process_writer did not explicitly set (overwrite from previous holder of the io driver) the read_callback, consequently an on_read callback was called on the 'already-deleted' object (processing_peer -- a previous holder of the io driver). Now the io_wrapper's ctor calls 'reset_callbacks' implicitly. This should make it easier for the new-in-chain objects not having to be aware of the callbacks-infrastructure set-up by the preceeding objects -- "each takes care of their own domain of concern" so to speak.

Bug-fix (i think): the reset method in scoped_membership_iterator should not have been 'asserting' for the to-be-reset iterator being non-blank... this is consistent with the 'op =' implementation for the same class. An alternative is to force the user-code to check the 'nulled' property on the scoped_membership_iterator object prior to calling any of the 'reset'-invoking methods (todo -- consider in future refactoring aspirations).

Minor bug-fix w.r.t. 64-bit size of the maximum-task counter (and subsequent simplification of the 'already-existing' conditions to throw exceptions -- no longer a check is made for old tasks, which were purged/deleted, having the same number/id as the new tasks... because the wraparound is separated by a huuuge 64 bit range and subsequently it's implicity timespan implicitly should be greater than that of the explicitly present 'purge-old-jobs' timer, or indeed the lifespan of a given server). The only forseable problem with current simplified code is that it may theoretically loop forever -- but this is only possible when all of the 64bit values (count of tasks) have been exhausted. This is never expected to take place by design... the NetCPU infrastructure is NOT indended for archiving of any, old, jobs and so purging is expected to take place regularly -- by virtue of NetCPU's design philosophy.

Minor, pedantic even, bug-fix/refactoring for the matching of the task/model types in server.cc: the end of the substring was previously including additional, irrelevant, chars for the floating-point and integer resolutions (mixed with underscore and 'x' chars). It still worked/works because censoc::lexicast is tolerant of the 1st substring content being compatible with the target datatype and whithout caring for the rest/following string contents. Now, the things are more 'correct' (the rest of the string is dealt-with/truncated explicitly).

Bug-fix in the BSDmakefile -- was referencing a stray file (2logit.cc) which is no longer used (was never, in fact, in the repository) -- instead 2csv is expected to be used as an in-between utility when moving datasets to/from different models (logit, mixed logit, gmnl et al).

Bugfix/refactoring -- renaming (inherited from older experimental code elsewhere) the  CENSOC_NETCPU_SCALED_OFFSET_MIXED_LOGIT_CONTROLLER_H hashdefs for header-inclusions in gmnl_2a to a more correct CENSOC_NETCPU_GMNL_2a_CONTROLLER_H (likewise for the processor.h and server.h in gmnl_2a directory).

-----------------------------------------------------------------

Quality-assurances:

Completion of a previous 'todo': server-side validation of meta and bulk messages during the addition of a new task (... although more verification is still in todo stage). Primarily done to ensure that subsequent processors' usage of the aforementioned meta and bulk messages shall not crash any of the processors running on a multitude of the computing nodes.

Light quality-assurance feature for the server's runtime integrity -- not processing the tasks that have been converged on (during the load of convergence_message in task_processor, the coefficients_rand_range_ended_wait value shall not be 0).

Light quality assurance feature asserting that complexity index in the combos_metadta in the combos builder shall not be 0.

Minor quality-assurance static assertion -- for rand.h (rand_half and rand_full) need to supply floating point numbers only.

-----------------------------------------------------------------

Refactoring

Sha checksums input args are now pointers to void as oppsode to being pointers to (unsigned)char -- for better verbosity and pedantic compliance with the underlying APIs in openssl.

Refactoring code w.r.t. throwing of recoverable exceptions in some cases when before a generic runtime exception was thrown thusly making the server exit.

Refactoring the stale/slow-calculating-peer code -- instead of throwing exception, will simply stop sending and re-sync when the peer echoes a more recent key/sequence sync point. This is very experimental and is not that important (mainly intended to result in lesser numebr of re-connections and re-initalisations if-when stale peer is present), but no big deal either way...

Refactoring composite_matrix into a simple array of (u)int8_t -- yet another simplification step in the post-Eigen era.

Clean-up w.r.t. unused/commented-out code (eigen et al.)

More concise code for counting respondents choice sets.

Move of the longer fields in meta_msg to the bulk_msg.

Some folding syntax for the comments, etc.

Deleting extra error injected rows error_injected_choicsets in simulating simulator.cc code -- would otherwise take more time to refactor something that is not used currently anyways. If need be, one can retrofit the thing later on from the repo history logs.

Some defficiencies in the simulator code remain -- but simulator code is not expected to be in any need of being efficient. Moreover, it is unclear if all of the currently supported models shall remain in the repo/active-use... so shall remain a todo consideration for the (possible distant) future.

Deleting simulator.cc from gmnl_2 -- adopting the one in gmnl_2a to test for model id and set appropriately irrelevant coefficients to harmless values.

Splitting models.h into models.h and models_ids.h (just need basic model ids in simulator.cc without any pre-declared model factory objects -- before all were coupled in the models.h file)

Refactoring respondents_choice_sets in dataset_1/controller to be a fastsize list and not a vector, more for coding consistency rather than anything else (either way not too much speed improvement w.r.t. overall state of things).

-----------------------------------------------------------------

Some HTTP-RELATED CHANGES:

Added 'this is beta so dont expect much' type of a label to the front/start page of the web interface (index.html).

Adding licensing section to index.html (forgot to do it the 1st commit of the http-related code).

Adding an explicit option to chose the sorting of .csv datafile during processing.

Added a very informative diagnostic in "Epic fail" string denoting the status code of the finished XMLHttpRequest processing.

Added in http/server.cc support for other-than gmnl_2a models (inline with the main controller.cc file).

Bug-fix in http 'adding a new task' page: when some values have not been filled, the alert box is displayed, but the 'send to server' button stays disabled (making it impossible to re-submit corrected values).

Bug-fix: contextmenu (right-click menu) fixed positioning of the menu if/when such would have been partially displayed outside of the scroll-area of the browsers viewport.

Bug-fix: in "add a new task" menu the help section for supported models had a broken hyperlink to the GMNL-II paper.

There is a spurious bug: when http server disconnects/drops due to timeouts, the next time firefox attempts to connect (with XMLHttpRequest with use-credentials set to true, with CORS et. al.) the connection makes it to the server but bails on ssl handshake. This could be because http server is currently disabling sslv2 and tlsv1 or for some other firefox-boost/asio/ssl setup/compatibility-in-behaviour issues... A quick observation such as: clicking on submit-to-server again appears to work just fine, so making this workaround automatic for the time-being (until there is more time to properly investigate the issue as a future TODO).

Bug-fix in http/server.cc: json formatting of tasks list was appending a coma after the last element in the array.

Bug-fix: (w.r.t. visual formatting/presentation) the http/index.html formatting of the values for the raw and range fields in the tasks list was not limiting to 3 decimal places or so, consequently the numbers were very wide and caused the overall table-display to extend wider than screen without any need.

Taking out the 'number of alternatives' field and instead adding the "wildcard deleting of rows" where any of the attribute's has a specific value -- these changes are essentially a propagation of the functionality features at the level of the lower NetCPU infrastructure (i.e. a new feature of variable number of alternatives per any given choice set in the dataset).

Deleting "id = xxx" label.id/checkbox.id linkage when checkbox element is already appendChild-ed to the label element.

Minor gui/visual-styling adjustments to the graphs in the 'show tasks' part of the inferface.

Deleted "not set" values for the preselected options in the 'add new task' gui (makes no sense to have 'blank' value as an option for those fields which have already bee pre-selected).

Re-styling -- moving the CLI-like text info to the very end of the page (after all of the submisison buttons), reflects a lower-priority status of this kinda info at this stage of the development/deployment.