NoahCarnahan/plagcomps — pull request #10 by NoahCarnahan (2013-12-19 06:10:34+00:00)
Title: Restructure

Hey team, I've been working on restructuring our project in the restructure branch. I'm opening this pull request to explain and discus the changes before merging into master. Unfortunately, due to file renaming and moving and stuff, its a bit hard to see what all has changed. I also had bad commit messages. Therefore, the following is a bit of an explanation of what I've done:

# Structural changes

In the restructure branch, I have rearranged and reorganized our code base. My goal was to create a [package](http://docs.python.org/2/tutorial/modules.html#packages) structure for the modules that would be easier to work with and understand. The old structure also had important functions sort of scattered all over the place, and there wasn't a clear entry point to just analyze text for plagiarism. The new structure supplies this with 'plagcomps.intrinsic.get_plagiarism()'.

# Additional (functional) changes

Changes were also made within modules, and what the tool actually does when extracting certain features, for example, has changed some. Unfortunately, I didn't keep track of all of these changes as well as I should have. But, I'll try to list the major ones here:
### Span array slicing
The function that gets all of the spans between two given character indices now returns indices into the span array such that the first index is inclusive and the second is exclusive (it used to be that the second was inclusive as well). This is consistent with the rest of our project and python.
### Sum tables
Indexing has changed for code enhanced readability. For example word_length_sum_table[i] is the sum of the lengths of words from 0 to i-1, whereas it used to be from 0 to i. This allows for the simplification of the feature functions that reference the sum tables. The sum table value getter method that used to exist is no logger needed to deal with edge cases.
### Features
Several features were renamed for internal consistency. I also added "external word freq class", which is the feature described in the Eissen paper. I also discovered that the stopword feature was missing many stopwords because the list of stopwords contained all of the words in lower case, but we weren't "lowering" our words before comparison or stripping punctuation. It now works as expected.
### ReducedDoc spans
Can't remember, gotta look into this...

# Running it

With the new package structure, you must run modules in a slightly different fashion than you would run a module "in the wild." Each module has (or will have) tests that are executed when you run the module directly (as opposed to importing). To run these, one must use the `-m` switch of the python interpreter. For example, to run the featureextraction test, one should do the following: Change to the directory _containing_ the plagcomps directory. Run `python -m plagcomps.intrinsic.featureextraction` in your terminal.

One little quirk exists with the `__init__.py` modules. One imports the contents of say, the `plagcomps/intrinsic/__init__.py` module with `import plagcomps.intrinsic`. But, to test this module, use `python -m plagcomps.intrinsic.__init__.py`.

# What still needs to be done?

I have left the code related to extrinsic detection almost untouched. But I did move the relavent files and change some import statements to work with the new package structure. Most notably, the actually extrinsic processing and the code that tests it should be separated. This might not be a very large task, but I haven't really looked into it. Also, the names of the files should change to the all lower case convention.

Additionally, I think that a `corpus` sub-package could make some things cleaner. That sub-package could pull together the things in the `corpus_partition`, `extrinsic_corpus_partition`, `sample_corpus`, and `shared` directories.

# Suggestions?
So, what do you guys think of all this? Good? Bad? some of both? How can this be better? I realize that I have made lots of changes, and that it may be difficult to see exactly whats going on. So, if you have questions about anything please let me know and I would glad to help or be challenged on any changes that I have made.