removed old files. started redesign of the pipeline (only mapper
for extraction and several mapreduce jobs for mapreduce features, sparse format)
Merge branch 'provenance' of https://github.com/jmp84/ruleXtract into provenance
Separated RuleWritable into RuleWritable and InfoWritable. Started
the source-to-target mapreduce feature.
implemented target-to-source probability mapreduce feature and binary provenance
feature. started the mapreduce feature merge job. jar not updated.
added a merger for the mapreduce features. jar not updated. still testing.
removed useless file, modified default constructor of RuleInfoWritable to avoid NPE, put correct settings in the merge job.
fixed bugs in source-to-target and target-to-source features. merger needs to output
to hfile rather than sequence file (may need to upgrade hbase and/or hadoop). need to check
correctness of hfile, implement missing mr features.
upgraded to hadoop 1.0.1 and hbase 0.92. first version working for
extraction and basic mr features (s2t, t2s, count). jar not updated.
Added the unaligned word feature, works. Added the binary provenance feature, needs testing.
Jar not updated.
changed binary provenance to have a value 1 rather than NullWritable.
-- made the binary provenance work
-- moved UnalignedWordReducer.java to the right directory
-- attempt to have the features in sorted order but doesn't work yet
file forgotten in the previous commit
prevent input from being tracked (rm --cached input and modify gitignore to contain "input")
Made the mapreduce feature merger output the features in sorted order. Started changes on rule retrieval to work with new design.
Moved forward rule retrieval and feature creation to match the new pipeline (one extraction, several mapreduce features, sparse format). need to add pattern translation probability features using custom partitioner and grouping comparator.
implemented most of the source2target pattern prob feature. need to work on the sort comparator.
finished implementing source-to-target pattern probability. need to implement the target-to-source pattern probability and test rule retrieval.
Added the target-to-source pattern probability feature. Made all mapreduce features implement HadoopJob. Created a class to handle hadoop workflow with JobControl. Put all the mappers and reducers as private static classes.
now extraction is only one program, from creation of input data to hfile creation.
need to add either more info in the config or a getNumberOfFeatures in each mr feature.
Main program Extraction now works to extract and merge basic mapreduce features. Features
to check: lexical, pattern.