Repository: AnalogJ/lexicon

Issue:
Provider Hetzner KonsoleH  is broken 
When using the latest version of lexicon, i.e.

```
$ lexicon --version
lexicon 3.3.11
```

there is a problem parsing the website:
```
$ lexicon hetzner create evk.services TXT --log_level DEBUG --name "_foo.foo2" --content "foo2"
Arguments: Namespace(action='create', auth_account=None, auth_password=None, auth_username=None, config_dir='/home/work/it/lexicon-github', content='foo2', delegated=None, domain='evk.services', identifier=None, latency=30, linked='yes', log_level='DEBUG', name='_foo.foo2', output='TABLE', priority=None, propagated='yes', provider_name='hetzner', ttl=None, type='TXT')
Warning: Use of environment variable LEXICON_HETZNER_USERNAME is deprecated. Try LEXICON_HETZNER_AUTH_USERNAME instead.
Warning: Use of environment variable LEXICON_HETZNER_PASSWORD is deprecated. Try LEXICON_HETZNER_AUTH_PASSWORD instead.
Hetzner => Enable CNAME lookup (see --linked parameter)
DNS Lookup => evk.services. IN SOA ns1.your-server.de. postmaster.your-server.de. 2019121911 10800 3600 604800 3600
DNS Lookup => ns1.your-server.de. IN A 213.133.106.251
DNS Lookup => ns1.your-server.de. IN AAAA 2a01:4f8:d0a:2006::2
DNS Lookup => evk.services. IN NS ns.second-ns.com.
DNS Lookup => evk.services. IN NS ns3.second-ns.de.
DNS Lookup => evk.services. IN NS ns1.your-server.de.
DNS Lookup => ns.second-ns.com. IN A 213.239.204.242
DNS Lookup => ns.second-ns.com. IN AAAA 2a01:4f8:0:a101::b:1
DNS Lookup => ns3.second-ns.de. IN A 193.47.99.4
DNS Lookup => ns3.second-ns.de. IN AAAA 2001:67c:192c::add:b3
DNS Lookup => ns1.your-server.de. IN A 213.133.106.251
DNS Lookup => ns1.your-server.de. IN AAAA 2a01:4f8:d0a:2006::2
DNS Lookup => evk.services IN NS 213.133.106.251 2a01:4f8:d0a:2006::2 213.239.204.242 2a01:4f8:0:a101::b:1 193.47.99.4 2001:67c:192c::add:b3
DNS Lookup => The DNS response does not contain an answer to the question: _foo.foo2.evk.services. IN CNAME
Starting new HTTPS connection (1): konsoleh.your-server.de:443
https://konsoleh.your-server.de:443 "GET / HTTP/1.1" 200 2184
https://konsoleh.your-server.de:443 "POST /login.php HTTP/1.1" 302 0
https://konsoleh.your-server.de:443 "GET / HTTP/1.1" 200 2019
Hetzner => Authenticate session with konsoleh account 'CXXXXXXXXX'
https://konsoleh.your-server.de:443 "GET /?page=1 HTTP/1.1" 200 2019
Hetzner => Get ID D1290385919 for domain evk.services
https://konsoleh.your-server.de:443 "GET /logout.php HTTP/1.1" 302 0
https://konsoleh.your-server.de:443 "GET /?err=logout HTTP/1.1" 200 2114
Hetzner => Exit session
Hetzner => Enable CNAME lookup (see --linked parameter)
DNS Lookup => evk.services. IN SOA ns1.your-server.de. postmaster.your-server.de. 2019121911 10800 3600 604800 3600
DNS Lookup => ns1.your-server.de. IN A 213.133.106.251
DNS Lookup => ns1.your-server.de. IN AAAA 2a01:4f8:d0a:2006::2
DNS Lookup => evk.services. IN NS ns3.second-ns.de.
DNS Lookup => evk.services. IN NS ns1.your-server.de.
DNS Lookup => evk.services. IN NS ns.second-ns.com.
DNS Lookup => ns3.second-ns.de. IN A 193.47.99.4
DNS Lookup => ns3.second-ns.de. IN AAAA 2001:67c:192c::add:b3
DNS Lookup => ns1.your-server.de. IN A 213.133.106.251
DNS Lookup => ns1.your-server.de. IN AAAA 2a01:4f8:d0a:2006::2
DNS Lookup => ns.second-ns.com. IN A 213.239.204.242
DNS Lookup => ns.second-ns.com. IN AAAA 2a01:4f8:0:a101::b:1
DNS Lookup => evk.services IN NS 213.133.106.251 2a01:4f8:d0a:2006::2 193.47.99.4 2001:67c:192c::add:b3 213.239.204.242 2a01:4f8:0:a101::b:1
DNS Lookup => The DNS response does not contain an answer to the question: _foo.foo2.evk.services. IN CNAME
Starting new HTTPS connection (1): konsoleh.your-server.de:443
https://konsoleh.your-server.de:443 "GET / HTTP/1.1" 200 2182
https://konsoleh.your-server.de:443 "POST /login.php HTTP/1.1" 302 0
https://konsoleh.your-server.de:443 "GET / HTTP/1.1" 200 2019
Hetzner => Authenticate session with konsoleh account 'CXXXXXXXX'
https://konsoleh.your-server.de:443 "GET /?domain_number=DXXXXXXX HTTP/1.1" 200 3285
https://konsoleh.your-server.de:443 "GET /dns.php?dnsaction2=editintextarea HTTP/1.1" 200 3744
https://konsoleh.your-server.de:443 "GET /logout.php HTTP/1.1" 302 0
https://konsoleh.your-server.de:443 "GET /?err=logout HTTP/1.1" 200 2115
Hetzner => Exit session
Traceback (most recent call last):
  File "/home/work/it/lexicon-github/venv/bin/lexicon", line 11, in <module>
    load_entry_point('dns-lexicon', 'console_scripts', 'lexicon')()
  File "/home/work/it/lexicon-github/lexicon/cli.py", line 117, in main
    results = client.execute()
  File "/home/work/it/lexicon-github/lexicon/client.py", line 78, in execute
    return self.provider.create_record(record_type, name, content)
  File "/home/work/it/lexicon-github/lexicon/providers/base.py", line 80, in create_record
    return self._create_record(rtype, name, content)
  File "/home/work/it/lexicon-github/lexicon/providers/hetzner.py", line 180, in _create_record
    with self._session(self.domain, self.domain_id) as ddata:
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/work/it/lexicon-github/lexicon/providers/hetzner.py", line 606, in _session
    raise exc
  File "/home/work/it/lexicon-github/lexicon/providers/hetzner.py", line 603, in _session
    zone = self._get_zone(qdomain, qdomain_id)
  File "/home/work/it/lexicon-github/lexicon/providers/hetzner.py", line 691, in _get_zone
    zone_file = Provider._filter_dom(dom, zone_file_filter).renderContents().decode('UTF-8')
AttributeError: 'NoneType' object has no attribute 'renderContents'
```

I do have some problems running the actual test-suite as well:
```
>                   cassette=self.cassette, failed_request=self._vcr_request
                )
E               vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('tests/fixtures/cassettes/hetzner/KonsoleH-IntegrationTests/test_provider_authenticate.yaml') in your current record mode ('none').
E               No match for the request (<Request (GET) https://konsoleh.your-server.de/>) was found.
E               No similar requests, that have not been played, found.

../venv/lib/python3.7/site-packages/vcr/stubs/__init__.py:232: CannotOverwriteExistingCassetteException
_ HetznerKonsoleHProviderTests.test_provider_when_calling_create_record_for_A_with_valid_name_and_content _
```
Please can someone provide short guidance how to set the config yaml file or environment variables correctly for Hetzner KonsoleH - thanks.


Discussion:
Further investigation revealed that the response from https://konsoleh.your-server.de/dns.php?dnsaction2=editintextarea is invalid HTML. Please note the second closing tag `</strong>` in the excerpt below:
```
<div id="content">
  <div class="warning">
    Please note that it can take up to 24 hours until the changes      
    become effective.<br/><br/>
    <strong>
      You should also take into account that faulty changes to DNS records can negatively 
      affect the operation of your website and/or email account. We take no responsibility for damages which might result.
    </strong>
    <br/><br/>
    If you want to use another name server, it is imperative that an update is done with the registrar. 
    Only changing the dns servers name in the zonefile is <u>not</u> sufficient!
  </strong>
  </div>
```
This causes the actual HTML parser `BeautifulSoup(dom, 'html.parser')` to substitute the second `</strong>` by a `</div>`. Consequently, parsing the subsequently following zonefile fails. 

Using the [more robust parser](https://beautiful-soup-4.readthedocs.io/en/latest/#installing-a-parser) `BeautifulSoup(dom, 'html5lib')` fixes this issue with *Hetzner KonsoleH*.

Resolving patch:
diff --git a/lexicon/providers/hetzner.py b/lexicon/providers/hetzner.py
--- a/lexicon/providers/hetzner.py
+++ b/lexicon/providers/hetzner.py
@@ -547,7 +547,7 @@ def _filter_dom(dom, filters, last_find_all=False):
         have no match.
         """
         if isinstance(dom, string_types):
-            dom = BeautifulSoup(dom, 'html.parser')
+            dom = BeautifulSoup(dom, 'html5lib')
         for idx, find in enumerate(filters, start=1):
             if not dom:
                 break
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -39,7 +39,7 @@
     'transip': ['transip>=0.3.0'],
     'plesk': ['xmltodict'],
     'henet': ['beautifulsoup4'],
-    'hetzner': ['dnspython>=1.15.0', 'beautifulsoup4'],
+    'hetzner': ['dnspython>=1.15.0', 'beautifulsoup4', 'html5lib'],
     'easyname': ['beautifulsoup4'],
     'localzone': ['localzone'],
     'gratisdns': ['beautifulsoup4'],


Test patch: