Remove PPtext from token config file.
Fixed a bug with the calculation of atomic data complexity.
Michel Blockelet, Summer intern at Princeton University
in June-July 2008 developed the first implementation of
PADS for Haskell under the supervision of David Walker.

Some of the key design ideas:

* use of built-in GADTs to represent the abstract syntax of PADS
   * interestingly, we considered many options for type-directed
     programming (many different ones exist in Haskell).  We settled
     on using GADTs, forgetting that that is what Yitzhak had done for
     PADS/ML -- that probably means that Yitzhak really made the right
     choice!  Though the implementation in Haskell is trivial compared with
     the trickiness that Yitzhak had to figure out to do the encoding.

   * though it probably looks easy now that we've done it (and perhaps
     it was! :-)  The difficult part was coming up with a solution that
     allowed (1) arbitrary data types to be representations (2) dependency
     including dependency on values with arbitrary types (3) annotations
     carrying values with arbitrary types (4) keeping things "well-typed"
     so that the type checker actually enforces useful invariants (eg:
     avoiding universal datatype definitions) (5) making it possible to
     use the representation to write all generic tools

   * all tools, including parsers and printers are coded easily using
     induction over the structure of the GADT

   * thanks to Stephanie Weirich for discussions about the various options
     we could use and help with GADTs in Haskell

* encoding of arbitrary data types using existential types and lists of
  constructor encodings where each constructor is encoded as a pair of
  injection and projection functions.  Turns out we reinvented an idea
  Stephanie Weirich uses in her RepLib library for generic programming.

* inclusion of "annotations" in the PADS descriptions that can be used to
customize the actions of particular tools.  These annotations carry values
of arbitrary types.  Moreover, each new tool can define its own new set of
annotations.  The implementation technique involves using the type class
Typeable to simulate an extensible datatype.  To define a new set of
annotations for a new tool, merely define a new datatype and derive the
Typeable type class.  In the tool, given an annotation, use type-safe
cast to determine if the annotation has a type associated with the tool.
If so, process the annotation however desired.  If not, discard the
annotation.  Thanks to the Haskell-cafe mailing lists for suggestions
on how to encode extensible datatypes using Typeable and existential
types (we figured out the existentials ourselves but weren't aware of
the existence of Typeable until we asked the Haskell gurus.

* we reimplemented regular expressions and added "tags" as suggested
by Ville Laurikari:

http://citeseer.ist.psu.edu/350334.html

* we implemented parser, printer, xml generator.  All of these tools
were more or less trivial to implement once we figured out the right
representation for the PADS abstract syntax.  There are things we
need to fix with the tools (eg: the xml pds are done differently
than the other tools; the parser and error recovery is a mess) but
this is "just engineering" :-).  Basically, the overall implementation
strategy and all the typing now "just works".  The features of Haskell
give a huge leg up on C (obviously!) but also it would appear on O'Caml.

Dave
Amazingly, I can't create a file names "Aux.hs" on my computer.
Apparently it's reserved for use by some kind of driver.  Bizarre.

Hence all occurrences of Aux --> Utilities

I haven't tested this since Haskell isn't installed on this computeri
(it is on my other laptop ...).  Hopefully, I correctly changes all
occurrences of the name.

Note: to play with this you must use the ghc extensions. ie:

ghc -fglasgow-exts Main.hs

or to use the interpreter:

ghci -fglasgow-exts Main.hs
the old tokenizer Qian used
First check-in of blob-finding algorithm. This algorithm
updates sub-parts in a ty with Blobs in a bottom-up manner.

As a heuristic, we compute the following two ratios to
determine of a ty is a possible blob:

let variance (ty) = sum of number of union/option/array/enum branches in ty
let var_ratio (ty) = variance (ty) / ((numTokens ty)/(coverage ty))
let ty_ratio(ty) = type_comp (ty) / atomic_data_comp (ty)
(*atomic data complexity is the data comp per chunk *)

let is_blob ty = var_ratio > 1.0 and (ratio + ty_ratio > 5.0)
Update padsc and padsml printer to output blobs properly.
Fixed a bug in refine_equal function after adding blobs.
Some small tweaks...
I check in the support vector machine stuff and the blob finding algorithm. Kenny helped me to integrate his blob finding algorithm into the learning system. I fixed some more small bugs in the SVM version. SVM is much slower than Maximum Entropy. The department server is way to slow to run the experiments. I'll try on another faster Linux machine.
a few more changes to SVM
instead of using my implementation of svm predicting phase, use libsvm c++ library directly
Initial paper skeleton for PADL'09 paper. I put in a temperary title as
a place holder which can certainly be changed.
Put in some of the references from Qian's general.
Rough draft of introduction completed.
Fix asl.log.p labeling to eliminate a few parsing errors in asl.log.
cut svm's cost by reducing # of times calling c++ library
a small change to avoid fileio buffer overflow
title change
A rough draft of section 2. The figures better have frames but I haven't
figured out how to add the frames. Moving on to sec 3 algorithm.
wrote the abstract.
added related work to the introduction.
changed title -- the resulting tokenizer/description is not probabilistic;
  the mechanism used to infer them is -- but way more importantly, the
  title now fits on 1 line :-)