Repository: mdn/kuma

Issue:
scrape_document with trailing slash fails
```
▶ docker-compose exec web ./manage.py scrape_document https://developer.mozilla.org/en-US/docs/Web/CSS/ --depth=2
/usr/local/lib/python2.7/site-packages/django/utils/encoding.py:6: ImportWarning: Not importing directory '/app/locale': missing __init__.py
  import locale
/usr/local/lib/python2.7/site-packages/django/template/backends/jinja2.py:6: ImportWarning: Not importing directory '/app/jinja2': missing __init__.py
  import jinja2
/app/kuma/core/i18n.py:18: ImportWarning: Not importing directory '/app/kuma/core/jinja2': missing __init__.py
  from jinja2 import nodes
/usr/local/lib/python2.7/site-packages/decorator_include.py:20: RemovedInDjango20Warning: Importing from django.core.urlresolvers is deprecated in favor of django.urls.
  from django.core.urlresolvers import RegexURLPattern as URLPattern, RegexURLResolver as URLResolver
/usr/local/lib/python2.7/site-packages/soapbox/templatetags/soapbox.py:9: RemovedInDjango20Warning: assignment_tag() is deprecated. Use simple_tag() instead
  @register.assignment_tag(takes_context=True)
INFO: Scrape progress, cycle 1: 1 Initializing, 1 Gathering Requirements
INFO: Scrape progress, cycle 2: 3 Initializing, 1 Gathering Requirements, 1 Done
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/app/kuma/scrape/management/commands/scrape_document.py", line 47, in handle
    scraper.scrape()
  File "/app/kuma/scrape/scraper.py", line 182, in scrape
    dependencies = source.gather(self.requester, self.storage)
  File "/app/kuma/scrape/sources/base.py", line 192, in gather
    has_prereqs, data = self.load_prereqs(requester, storage)
  File "/app/kuma/scrape/sources/document_children.py", line 31, in load_prereqs
    data = response.json()
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 4 column 1 (char 3)
```

Note that if I retry without the trailing slash things work. I.e. `docker-compose exec web ./manage.py scrape_document https://developer.mozilla.org/en-US/docs/Web/CSS --depth=2`

Somewhere in there it assumes that the response is JSON when in fact the response status code is not OKish. 
scrape_document with trailing slash fails
```
▶ docker-compose exec web ./manage.py scrape_document https://developer.mozilla.org/en-US/docs/Web/CSS/ --depth=2
/usr/local/lib/python2.7/site-packages/django/utils/encoding.py:6: ImportWarning: Not importing directory '/app/locale': missing __init__.py
  import locale
/usr/local/lib/python2.7/site-packages/django/template/backends/jinja2.py:6: ImportWarning: Not importing directory '/app/jinja2': missing __init__.py
  import jinja2
/app/kuma/core/i18n.py:18: ImportWarning: Not importing directory '/app/kuma/core/jinja2': missing __init__.py
  from jinja2 import nodes
/usr/local/lib/python2.7/site-packages/decorator_include.py:20: RemovedInDjango20Warning: Importing from django.core.urlresolvers is deprecated in favor of django.urls.
  from django.core.urlresolvers import RegexURLPattern as URLPattern, RegexURLResolver as URLResolver
/usr/local/lib/python2.7/site-packages/soapbox/templatetags/soapbox.py:9: RemovedInDjango20Warning: assignment_tag() is deprecated. Use simple_tag() instead
  @register.assignment_tag(takes_context=True)
INFO: Scrape progress, cycle 1: 1 Initializing, 1 Gathering Requirements
INFO: Scrape progress, cycle 2: 3 Initializing, 1 Gathering Requirements, 1 Done
Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/app/kuma/scrape/management/commands/scrape_document.py", line 47, in handle
    scraper.scrape()
  File "/app/kuma/scrape/scraper.py", line 182, in scrape
    dependencies = source.gather(self.requester, self.storage)
  File "/app/kuma/scrape/sources/base.py", line 192, in gather
    has_prereqs, data = self.load_prereqs(requester, storage)
  File "/app/kuma/scrape/sources/document_children.py", line 31, in load_prereqs
    data = response.json()
  File "/usr/local/lib/python2.7/site-packages/requests/models.py", line 897, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/site-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 4 column 1 (char 3)
```

Note that if I retry without the trailing slash things work. I.e. `docker-compose exec web ./manage.py scrape_document https://developer.mozilla.org/en-US/docs/Web/CSS --depth=2`

Somewhere in there it assumes that the response is JSON when in fact the response status code is not OKish. 


Discussion:

Resolving patch:
diff --git a/kuma/scrape/management/commands/__init__.py b/kuma/scrape/management/commands/__init__.py
--- a/kuma/scrape/management/commands/__init__.py
+++ b/kuma/scrape/management/commands/__init__.py
@@ -18,12 +18,12 @@ def parse_url_or_path(self, url_or_path):
         if url_or_path.startswith('http'):
             bits = urlparse(url_or_path)
             host = bits.netloc
-            path = bits.path
+            path = bits.path.rstrip('/')
             ssl = (bits.scheme == 'https')
         else:
             host = 'wiki.developer.mozilla.org'
             ssl = True
-            path = url_or_path
+            path = url_or_path.rstrip('/')
         return host, ssl, path
 
     def setup_logging(self, verbosity):


Test patch: