first commit, get POS-tagging initial code in
finished most of word_pos_vectorization, which gen vector representations of words in CTB5.0 in terms of POS tags
small bug fixed
pre_proc_annotation.py to pre-preocess the word-structure annotation, so it is compatible with NLTK-style of Tree data structure
finshed 2 programs:
2a_gen_tag_set_for_word_type.py: generate the new-tag for each word type. This Output shall be a map:  Word (type) --> set of possible tags

2b_tag_vector_propagation.py: Algorithms for tag vector propagation. The tag vector are propogated from higher node in the tree to its 'head children'. And the update is based on type and done in a accumulative manner, i.e. each (sub)tree of a string should have the same tag-vector.
started 2c_re_propagation.py, which is "Similarity-based 2nd-order Tag propagation". Finished the complete design and the coding of the first half.
re_propagation.py done, now we have a complete annotation/grammar.
considering refinement of the grammar and building parsing model now!
tested equivalance class construction for non-terminals (gen_equivalence_class_of_no_terminals.py) via quick union algorithm (quick_union_wpc).
negative result, as the equivalent relation defined by node and head_children(node) is not robust to noised and errors.
update yesterday's work, which started mini_tree_seq_gen.py
extracted utilities shared by module to utility_proj.py, to keep consistency
update of work Wed-Fri
tried using NLTK's CKY to parse with X_baseline_grammar.py and X_baseline_parse_seg.py, but it is too slow

so try to directly output PTB format treebanks with X_write_annotation.py and X_write_weighted_annotation.py instead to parse with Stanford parser.

A couple of small utilities: pos2seg.py, convert pos-tagged CTB corpus to only word-segmented; uniq_ngram_gen.py can genenrate all unique ngrams up to N
implemented
1. parser_prob_collector, which get the log probabilities from the Viterbi output of Stanford parser;
2. gen_string2parseProb, using the above result and a list of their corresponding string, generate a hashtable that maps a string to its log prob score;
3. table_seger, which is a segmenter that read the hashtable gen by above function as its score function and then segment a given corpus by running Viterbi algorithm.
Update.
And summary of Y_*.py modules:
Run Y_*.py in the following order to get the parser-based segmentation from the raw annotation of word structure

1. Y_write_weighted_annotation.py or Y_write_annotation.py, which output the pre-processed annotation to plain txt file.  The weighted one is preferred, which for word-postag type occur write N trees, N=frequency of the (word, pos-tag) pair

2. Training the stanford parser with the output above and parse the ngram up to length K for all the unique ngrams that will appear in the testing corpus.
this may call uniq_ngram_gen.py to generate all the unique n-grams of the testing cropus

3. Y_parser_prob_collector.py, which collect the log prob of top-n Viterbi parsing given by the stanford parser

4. Y_gen_string2parseProb.py, use the output above and the original ngram file to generate a hashtable that maps a ngram to its log-probability.

5. Y_table_seger.py, read the map from above, and initilaize a Viterbi segmenter, the score of each word candidate is from the above hash table. It then segment the corpus specified in this module and write the segmentation result
minor change yet wrap for the day
implemented vector_grammar_ptb_annotation_gen, which generate (3-versions of) annotaitons suitable for Stanford Parser training from the induced Vector Grammar; the annotation is w.r.t. word type with frequency awareness.

impelmented Penn2Dep.py, which converts Penn/CTB sylte constituent trees to dependency graphs.
readme file for Y_*.py scripts
finished Penn2Dep by adding function depgraph2bracket to further convert dependency tree to brackted representation
developed ctb5_split_4_joint_seg_tag.py, which split the Chinese Treebank 5 according to popular configurations in joint segmentation and pos tagging literatures
started developing global_constituent_parsing_model_training_data_gen, which read training corpus and generate flat trees using word structure annotation; then write the result to PTB bracket format.
* finished global_constituent_parsing_model_training_data_gen.py, which generates flat-tree corpora for training with Stanford parser (global constituent parsing model for CWS)
* adapted 4_*.py to generate standing-alone refined (or final) word structure annotation (5_*.py), may have many overlaps with Z_vector_grammar_ptb_annotation_gen.py
* finished B_global_parsing_result_collector.py, which collect the results from the parser to list of list data structure and serialized to a .pickle file
* finished pos2charSeq.py, which convert pos-tagged sentences into white-spaced character sequences, which makes input for parsers.