Repository: encode/django-rest-framework

Issue:
Backslashes added to paths at docs from `include_docs_urls`
## Checklist

- [x] I have verified that that issue exists against the `master` branch of Django REST framework.
- [x] I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
- [x] This is not a usage question. (Those should be directed to the [discussion group](https://groups.google.com/forum/#!forum/django-rest-framework) instead.)
- [x] This cannot be dealt with as a third party library. (We prefer new functionality to be [in the form of third party libraries](http://www.django-rest-framework.org/topics/third-party-resources/#about-third-party-packages) where possible.)
- [ ] I have reduced the issue to the simplest possible case.
- [ ] I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

## Steps to reproduce

Using DRF installed from github, Django 2.0.

Included the following at `urls.py`:

    urlpatterns = [
        ...
        path('^api/token-auth/', authtoken_views.obtain_auth_token)
        path('^docs/', include_docs_urls(title='API Docs', authentication_classes=[], permission_classes=[], public=True)),
        ...
    ]

## Expected behavior

Urls in docs should `api/endpoint-name/`

## Actual behavior

Urls are `api\/endpoint\-name/`, when using the "interact" button you get a 404 (request URL is `/api%5C/profiles/`)

<img width="619" alt="captura de tela 2017-12-16 as 23 49 05" src="https://user-images.githubusercontent.com/1397214/34075891-fd2cf026-e2bb-11e7-90d0-61d7dc7c7558.png">

Backslashes added to paths at docs from `include_docs_urls`
## Checklist

- [x] I have verified that that issue exists against the `master` branch of Django REST framework.
- [x] I have searched for similar issues in both open and closed tickets and cannot find a duplicate.
- [x] This is not a usage question. (Those should be directed to the [discussion group](https://groups.google.com/forum/#!forum/django-rest-framework) instead.)
- [x] This cannot be dealt with as a third party library. (We prefer new functionality to be [in the form of third party libraries](http://www.django-rest-framework.org/topics/third-party-resources/#about-third-party-packages) where possible.)
- [ ] I have reduced the issue to the simplest possible case.
- [ ] I have included a failing test as a pull request. (If you are unable to do so we can still accept the issue.)

## Steps to reproduce

Using DRF installed from github, Django 2.0.

Included the following at `urls.py`:

    urlpatterns = [
        ...
        path('^api/token-auth/', authtoken_views.obtain_auth_token)
        path('^docs/', include_docs_urls(title='API Docs', authentication_classes=[], permission_classes=[], public=True)),
        ...
    ]

## Expected behavior

Urls in docs should `api/endpoint-name/`

## Actual behavior

Urls are `api\/endpoint\-name/`, when using the "interact" button you get a 404 (request URL is `/api%5C/profiles/`)

<img width="619" alt="captura de tela 2017-12-16 as 23 49 05" src="https://user-images.githubusercontent.com/1397214/34075891-fd2cf026-e2bb-11e7-90d0-61d7dc7c7558.png">



Discussion:
Downgrading to Django 1.11 fixed the problem. Would `tests/test_renderers.py` be the appropriate file to add a test for this issue?
Hi @scardine.

I need to look into this but I'm not sure it is a renderer issue. See @axnsan12's commit referencing this issue and #5672. 

#5609 (although distinct) is in a similar ballpark here. 
More specifically, I think this should be addressed by Django in `simplify_regex`, which should, quote,

```python
    r"""
    Clean up urlpattern regexes into something more readable by humans. For
    example, turn "^(?P<sport_slug>\w+)/athletes/(?P<athlete_slug>\w+)/$"
    into "/<sport_slug>/athletes/<athlete_slug>/".
    """
```

[get_path_from_regex](https://github.com/encode/django-rest-framework/blob/d12005cf902e3969b5002593eab83bb63feda840/rest_framework/schemas/generators.py#L191) in drf makes use of that function.

I filed a bug in their tracker - https://code.djangoproject.com/ticket/28936

Good follow-up @axnsan12. Thanks. 
Thanks, if it is not fixed by next week I will try to submit a patch for `simplify_regex` on my holiday's recess.
Note: Discussion on #5686 has details worth viewing. 
I think one should do `str(pattern)` before passing it to `simplify_regex`. In fact, you don't need to pass a `RoutePattern` into `simplify_regex` at all, as it's not a regex (but you may for code simplicity reasons).

The _bigger_ problem here seems not the escaping (can be dealt with calling `str(pattern)`), but that the result may contain type coercion tags.

Before in Django 1.x, the output of `simplify_regex` would always look like: `/example/<id>/`.
Whereas now, it can also be: `/example/<int:id>/`. Which may be unexpected for some follow-up code.
@tiltec I +1 your assessment.

Calling `str(pattern)` in Python 2.7 may trigger UnicodeError. BTW, are there any plans to do the same in DRF now that Django 2.0 dropped support for 2.7? Life is easier when you don't have to think about 2.7 compatibility.

Django 1.11 is the last version supporting 2.7 but it is an LTS version and support ends only in April 2020.

Seems only tangentially related, but I wonder if there's a bug in Django 2.0 when using `str(pattern)` together with [translatable URL patterns](https://docs.djangoproject.com/en/2.0/topics/i18n/translation/#translating-url-patterns).
It clearly returns the gettext_lazy function instead of the translated string. Will set up a more comprehensive testcase for it...

(EDIT) Upstream bug filed: https://code.djangoproject.com/ticket/28947
> In Django 1.x, the output of simplify_regex would always look like: `/example/<id>/`.
Whereas now, it can also be: `/example/<int:id>/`. Which may be unexpected for some follow-up code.

I encountered a follow-up bug.

DRF uses [uritemplate](https://uritemplate.readthedocs.io) to parse the URL variables, but unfortunately the colon means something else there. [RFC 6570](https://tools.ietf.org/html/rfc6570#section-2.4.1) says that `id:5` specifies a parameter with max length of 5. 

Doesn't really fit `int:id`, where `int` is the _converter_ and `id` the parameter name. Therefore I get `ValueError: invalid literal for int() with base 10: 'id'` from uritemplate.

Maybe it's easiest to strip the converter part out for SchemaGenerator purposes?
I think the converter part could be useful for the typing information it could provide about the path parameter. Perhaps the most complete solution would be to [try and parse it just like django does](https://github.com/django/django/blob/78247b80a8cbf1ebfb7a54624de7dc92e1a1f888/django/urls/resolvers.py#L193)?

EDIT: or strip it and use the already parsed `converters` on the `RoutePattern`
Downgrading to Django 1.11 fixed the problem. Would `tests/test_renderers.py` be the appropriate file to add a test for this issue?
Hi @scardine.

I need to look into this but I'm not sure it is a renderer issue. See @axnsan12's commit referencing this issue and #5672. 

#5609 (although distinct) is in a similar ballpark here. 
More specifically, I think this should be addressed by Django in `simplify_regex`, which should, quote,

```python
    r"""
    Clean up urlpattern regexes into something more readable by humans. For
    example, turn "^(?P<sport_slug>\w+)/athletes/(?P<athlete_slug>\w+)/$"
    into "/<sport_slug>/athletes/<athlete_slug>/".
    """
```

[get_path_from_regex](https://github.com/encode/django-rest-framework/blob/d12005cf902e3969b5002593eab83bb63feda840/rest_framework/schemas/generators.py#L191) in drf makes use of that function.

I filed a bug in their tracker - https://code.djangoproject.com/ticket/28936

Good follow-up @axnsan12. Thanks. 
Thanks, if it is not fixed by next week I will try to submit a patch for `simplify_regex` on my holiday's recess.
Note: Discussion on #5686 has details worth viewing. 
I think one should do `str(pattern)` before passing it to `simplify_regex`. In fact, you don't need to pass a `RoutePattern` into `simplify_regex` at all, as it's not a regex (but you may for code simplicity reasons).

The _bigger_ problem here seems not the escaping (can be dealt with calling `str(pattern)`), but that the result may contain type coercion tags.

Before in Django 1.x, the output of `simplify_regex` would always look like: `/example/<id>/`.
Whereas now, it can also be: `/example/<int:id>/`. Which may be unexpected for some follow-up code.
@tiltec I +1 your assessment.

Calling `str(pattern)` in Python 2.7 may trigger UnicodeError. BTW, are there any plans to do the same in DRF now that Django 2.0 dropped support for 2.7? Life is easier when you don't have to think about 2.7 compatibility.

Django 1.11 is the last version supporting 2.7 but it is an LTS version and support ends only in April 2020.

Seems only tangentially related, but I wonder if there's a bug in Django 2.0 when using `str(pattern)` together with [translatable URL patterns](https://docs.djangoproject.com/en/2.0/topics/i18n/translation/#translating-url-patterns).
It clearly returns the gettext_lazy function instead of the translated string. Will set up a more comprehensive testcase for it...

(EDIT) Upstream bug filed: https://code.djangoproject.com/ticket/28947
> In Django 1.x, the output of simplify_regex would always look like: `/example/<id>/`.
Whereas now, it can also be: `/example/<int:id>/`. Which may be unexpected for some follow-up code.

I encountered a follow-up bug.

DRF uses [uritemplate](https://uritemplate.readthedocs.io) to parse the URL variables, but unfortunately the colon means something else there. [RFC 6570](https://tools.ietf.org/html/rfc6570#section-2.4.1) says that `id:5` specifies a parameter with max length of 5. 

Doesn't really fit `int:id`, where `int` is the _converter_ and `id` the parameter name. Therefore I get `ValueError: invalid literal for int() with base 10: 'id'` from uritemplate.

Maybe it's easiest to strip the converter part out for SchemaGenerator purposes?
I think the converter part could be useful for the typing information it could provide about the path parameter. Perhaps the most complete solution would be to [try and parse it just like django does](https://github.com/django/django/blob/78247b80a8cbf1ebfb7a54624de7dc92e1a1f888/django/urls/resolvers.py#L193)?

EDIT: or strip it and use the already parsed `converters` on the `RoutePattern`

Resolving patch:
diff --git a/rest_framework/compat.py b/rest_framework/compat.py
--- a/rest_framework/compat.py
+++ b/rest_framework/compat.py
@@ -32,7 +32,7 @@
 def get_regex_pattern(urlpattern):
     if hasattr(urlpattern, 'pattern'):
         # Django 2.0
-        return urlpattern.pattern.regex.pattern
+        return str(urlpattern.pattern)
     else:
         # Django < 2.0
         return urlpattern.regex.pattern
@@ -255,6 +255,14 @@ def md_filter_add_syntax_highlight(md):
 except ImportError:
     InvalidTimeError = Exception
 
+# Django 1.x url routing syntax. Remove when dropping Django 1.11 support.
+try:
+    from django.urls import include, path, re_path # noqa
+except ImportError:
+    from django.conf.urls import include, url # noqa
+    path = None
+    re_path = url
+
 
 # `separators` argument to `json.dumps()` differs between 2.x and 3.x
 # See: http://bugs.python.org/issue22767
diff --git a/rest_framework/schemas/generators.py b/rest_framework/schemas/generators.py
--- a/rest_framework/schemas/generators.py
+++ b/rest_framework/schemas/generators.py
@@ -3,6 +3,7 @@
 
 See schemas.__init__.py for package overview.
 """
+import re
 import warnings
 from collections import Counter, OrderedDict
 from importlib import import_module
@@ -135,6 +136,11 @@ def endpoint_ordering(endpoint):
     return (path, method_priority)
 
 
+_PATH_PARAMETER_COMPONENT_RE = re.compile(
+    r'<(?:(?P<converter>[^>:]+):)?(?P<parameter>\w+)>'
+)
+
+
 class EndpointEnumerator(object):
     """
     A class to determine the available API endpoints that a project exposes.
@@ -189,7 +195,9 @@ def get_path_from_regex(self, path_regex):
         Given a URL conf regex, return a URI template string.
         """
         path = simplify_regex(path_regex)
-        path = path.replace('<', '{').replace('>', '}')
+
+        # Strip Django 2.0 convertors as they are incompatible with uritemplate format
+        path = re.sub(_PATH_PARAMETER_COMPONENT_RE, r'{\g<parameter>}', path)
         return path
 
     def should_include_endpoint(self, path, callback):


Test patch:
diff --git a/tests/test_schemas.py b/tests/test_schemas.py
--- a/tests/test_schemas.py
+++ b/tests/test_schemas.py
@@ -9,7 +9,7 @@
 from rest_framework import (
     filters, generics, pagination, permissions, serializers
 )
-from rest_framework.compat import coreapi, coreschema, get_regex_pattern
+from rest_framework.compat import coreapi, coreschema, get_regex_pattern, path
 from rest_framework.decorators import (
     api_view, detail_route, list_route, schema
 )
@@ -361,6 +361,59 @@ def test_schema_for_regular_views(self):
         assert schema == expected
 
 
+@unittest.skipUnless(coreapi, 'coreapi is not installed')
+@unittest.skipUnless(path, 'needs Django 2')
+class TestSchemaGeneratorDjango2(TestCase):
+    def setUp(self):
+        self.patterns = [
+            path('example/', ExampleListView.as_view()),
+            path('example/<int:pk>/', ExampleDetailView.as_view()),
+            path('example/<int:pk>/sub/', ExampleDetailView.as_view()),
+        ]
+
+    def test_schema_for_regular_views(self):
+        """
+        Ensure that schema generation works for APIView classes.
+        """
+        generator = SchemaGenerator(title='Example API', patterns=self.patterns)
+        schema = generator.get_schema()
+        expected = coreapi.Document(
+            url='',
+            title='Example API',
+            content={
+                'example': {
+                    'create': coreapi.Link(
+                        url='/example/',
+                        action='post',
+                        fields=[]
+                    ),
+                    'list': coreapi.Link(
+                        url='/example/',
+                        action='get',
+                        fields=[]
+                    ),
+                    'read': coreapi.Link(
+                        url='/example/{id}/',
+                        action='get',
+                        fields=[
+                            coreapi.Field('id', required=True, location='path', schema=coreschema.String())
+                        ]
+                    ),
+                    'sub': {
+                        'list': coreapi.Link(
+                            url='/example/{id}/sub/',
+                            action='get',
+                            fields=[
+                                coreapi.Field('id', required=True, location='path', schema=coreschema.String())
+                            ]
+                        )
+                    }
+                }
+            }
+        )
+        assert schema == expected
+
+
 @unittest.skipUnless(coreapi, 'coreapi is not installed')
 class TestSchemaGeneratorNotAtRoot(TestCase):
     def setUp(self):