progress on input preprocessing
fixed lawrence's duplicate issue
Merge pull request #7 from jwayne/josh

progress on input preprocessing