Add support for non-language-specific term frequencies.

In this "language" (labeled 'xx'), we don't distinguish stopwords or
punctuation. This means that a comma can be a term. Therefore, we need
to revise the wordlist code slightly to make that syntactically legal.

The new wordlist for this purpose is called 'twitter2.txt'.
Add comments about the status of each module.

This determines what should happen to the various pieces of metanl
if we were to refactor it, with the goals of:

- Making it possible to port to Python 3
- Making it make sense as a package (or two)
Removing four files in preparation for code updating

Three have been moved to wordfreq. The fourth is snowball, which doesn't get
used and can thus safely go away.
Removing another unnecessary file

Everything in general.py is either obsolete (because of ftfy) or only used in
ConceptNet code (like un_camel_case), or just not used at all (like asciify).
The one exception is unicode_is_punctuation, which has been moved into the one
file (extprocess.py) that uses it.
xrange to range, for Py3
More Py3 updates and other revisions (near complete)

* Restoration of metanl.general, primarily for untokenize (which is
  still a mess, but it's all we have at the moment)

* Use of unicode_literals in most files to remove the need to mark
  strings as u''.

* Replacement of metanl.general.tokenize_list with
  nltk.wordpunct_tokenize.  Replacement of preprocess_text with
  ftfy.fix_text.

* Removal of various methods to get word frequency and wordlists.

* Removal of commented-out debugging lines, and some pep8 changes.
Further Py3 updates (still need Py2 checks)

* Some fixes to things missed: u'', imports, unicode() vs str(),
  xrange, etc.

* self.process.stdin.flush() added to extprocess.send_input()
Be more explicit about bytestrings
Filename changes for accuracy
Now that we're manually flushing, let the OS choose the best buffer size.

This is mostly out of paranoia that Python 3 is taking "bufsize=1" as literally "use a one-byte buffer", which would not be desirable.
Never mind the bytestring, just do Py2 and Py3 differently
Two more fixes for the Py3 branch

nltk.py should have been called something else so that imports from nltk,
the actual python package, wouldn't run into errors. (It now is.)

general.py was missing an "import re".
Merge branch 'Py3' of github.com:commonsense/metanl into Py3
Remove support for euc-jp
Restore un_camel_case function
Merge branch 'taglessness' into Py3

Conflicts:
	metanl/nltk_morphy.py
Merge branch 'Py3' of github.com:commonsense/metanl into Py3
Simplification of nltk_morphy tag_and_stem
Copy docstring tests into test files
Updates to setup.py for the new version