Created folder skeleton. Added Project Proposal .doc and converted into LaTeX
Added examples of jokes
Final draft?
Final revisions to the draft.
Moved project proposal into project_proposal folder
Wrote extraction script for x, y, z
Wrote POS tagger and filter for tags and count. Still need to work on running speed (~5 hours per 10,000,000 lines right now, and ~25 of such files)
...and now also tracking the files
Sped up tagging by filtering on counts first. Tagged first full file in 50 mins
Finished tagging and filtering. Then also implemented sorting
Wrote more code for filtering, and wrote code for generating 1-3 adjectives. Started writing code for generating Y
Added fully tagged files
Finished code for generating X,Y,Z with 1-3 adjectives
Restructured generation code, which is now fully working. Added code to sort_ngrams that pluralizes every noun in the ngrams, so that we can merge singular and plural as well. Our jokes require them to be plural anyways, so this works well. There are possibly some issues with nouns that can both be plural and singular (we now pluralize them, so if they should have been plural already we misinterpreted them. POS tagger cannot detect this either).
Generated 1000 jokes from each category for labeling
Formatted xyz combinations into joke format for easier labeling
Wrote up features. Started PMI calculations.
Wrote method for combining all bigrams into a single file for easier searching. Wrote another method for calculating the total counts of the filtered/unfiltered bigrams/unigrams. Now backing up for changing all 'tagging' names to 'filtered' names because that is clearer.
Changed 'untagged'/'tagged' to 'unfiltered'/'filtered' for clarity. Created a global_definitions file containing all file locations and other global definitions, so that we can change it in one place rather than in all separate files.
Wrote methods for calculating prob_a_and_b and prob_a_given_b