Remove dedendency on bs4 (BeautifulSoup4) and tidylib

tidylib caused problems on OS X Yosemite.
These two libraries have been replaced by lxml's html5parser
and Cleaner.

Two tests fail after these changes.

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104902 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Remove unused code

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104903 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
DocConverter now raises an exception when wvHtml fails

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104904 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Fix the code so that the two last tests also pass

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104905 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Get encoding from html headers

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104906 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Set a default value for all classes that inherit from HTMLContentConverter
Fix tests

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104907 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Put charset definition into a separate function

All tests pass

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104908 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Give the arguments to set_charset in the same order as in __init__

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104909 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Check that set_charset returns what it is supposed to

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104910 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Remove debug output

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104911 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
If charset is iso-8859-1, interpret it as Windows-1252

http://en.wikipedia.org/wiki/Windows-1252#cite_note-1

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104912 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
<center> to <div>, tidy-style

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104913 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
One more thing to remove from www.samediggi.fi documents

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104914 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Catch errors so the program is able to continue

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104915 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
A simple html formatter, to make it easier to see the structure of html documents

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104916 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
flake8 conformance

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104917 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
wvHtml packs headings into some html wool, this is the start of a function to do it.

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104918 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Remove dependency on pytidylib and beautifulsoup4

CorpusTools already depends heavily on lxml, so replacing
these two modules with lxml.html.clean and lxml.html.html5parser

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104919 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
classify now calls classify_full, which allows inspecting the scores

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104920 c7155fb1-f0a7-4240-a2fc-2600b6f42f90
Fix two faulty tests and write more tests to check all cases for set_charset

git-svn-id: https://victorio.uit.no/langtech/trunk/tools/CorpusTools@104930 c7155fb1-f0a7-4240-a2fc-2600b6f42f90