internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: readthedocs/readthedocs.org

Issue:
Etag + md5 question
I'm playing with python3 conversion, and came across the following : 

I see that [here](https://github.com/rtfd/readthedocs.org/blob/f829ef0ebf93447318c93ff9ad45dfe7ed22ce98/readthedocs/projects/views/public.py#L117) the Etag is set to a `md5.new(url)` object. Shouldn't it be the `md5.new(url).hexdigest()` ? Or does django do some magic ?

Thanks.



Discussion:
Perhaps -- It would be great to get a test for this showing it failing, and a patch that fixes it. 

Actually, it should use the `hashlib` module. `md5` has been deprecated since Python 2.5. However, it seems like the else branch in `_badge_return` is unused. `_badge_return` used by the `project_badge` view which passes `redirect=True` by default. I will open a pull request to simply it.


Resolving patch:
diff --git a/readthedocs/projects/views/public.py b/readthedocs/projects/views/public.py
--- a/readthedocs/projects/views/public.py
+++ b/readthedocs/projects/views/public.py
@@ -5,7 +5,6 @@
 import json
 import logging
 import mimetypes
-import md5
 
 from django.core.urlresolvers import reverse
 from django.core.cache import cache
@@ -106,20 +105,8 @@ def get_context_data(self, **kwargs):
         return context
 
 
-def _badge_return(redirect, url):
-    if redirect:
-        return HttpResponseRedirect(url)
-    else:
-        response = requests.get(url)
-        http_response = HttpResponse(response.content,
-                                     content_type="image/svg+xml")
-        http_response['Cache-Control'] = 'no-cache'
-        http_response['Etag'] = md5.new(url)
-        return http_response
-
-
 @cache_control(no_cache=True)
-def project_badge(request, project_slug, redirect=True):
+def project_badge(request, project_slug):
     """Return a sweet badge for the project"""
     version_slug = request.GET.get('version', LATEST)
     style = request.GET.get('style', 'flat')
@@ -130,13 +117,13 @@ def project_badge(request, project_slug, redirect=True):
         url = (
             'https://img.shields.io/badge/docs-unknown%20version-yellow.svg?style={style}'
             .format(style=style))
-        return _badge_return(redirect, url)
+        return HttpResponseRedirect(url)
     version_builds = version.builds.filter(type='html', state='finished').order_by('-date')
     if not version_builds.exists():
         url = (
             'https://img.shields.io/badge/docs-no%20builds-yellow.svg?style={style}'
             .format(style=style))
-        return _badge_return(redirect, url)
+        return HttpResponseRedirect(url)
     last_build = version_builds[0]
     if last_build.success:
         color = 'brightgreen'
@@ -144,7 +131,7 @@ def project_badge(request, project_slug, redirect=True):
         color = 'red'
     url = 'https://img.shields.io/badge/docs-%s-%s.svg?style=%s' % (
         version.slug.replace('-', '--'), color, style)
-    return _badge_return(redirect, url)
+    return HttpResponseRedirect(url)
 
 
 def project_downloads(request, project_slug):


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Document support of .readthedocs.yml
see rtfd/readthedocs-build#17



Discussion:

Resolving patch:
diff --git a/readthedocs/core/adapters.py b/readthedocs/core/adapters.py
--- a/readthedocs/core/adapters.py
+++ b/readthedocs/core/adapters.py
@@ -1,5 +1,8 @@
 """Allauth overrides"""
 
+import pickle
+import logging
+
 from allauth.account.adapter import DefaultAccountAdapter
 from django.template.loader import render_to_string
 
@@ -10,6 +13,8 @@
 except ImportError:
     from django.utils.encoding import force_unicode as force_text
 
+log = logging.getLogger(__name__)
+
 
 class AccountAdapter(DefaultAccountAdapter):
 
@@ -25,6 +30,19 @@ def send_mail(self, template_prefix, email, context):
         subject = " ".join(subject.splitlines()).strip()
         subject = self.format_email_subject(subject)
 
+        # Allauth sends some additional data in the context, remove it if the
+        # pieces can't be pickled
+        removed_keys = []
+        for key in context.keys():
+            try:
+                _ = pickle.dumps(context[key])
+            except pickle.PickleError:
+                removed_keys.append(key)
+                del context[key]
+        if removed_keys:
+            log.debug('Removed context we were unable to serialize: %s',
+                      removed_keys)
+
         send_email(
             recipient=email,
             subject=subject,


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Document that RTD uses `rel` branch for production
Hi, i'd like to add a new builder for doxygen documentation (but native, not with breath). Since there are a lot of branches like real/relcorp which a far ahead of master, i'd like to know, which branch to choose for development.

Thanks in advance!
Oli



Discussion:
Thanks for your interest. We likely wouldn't include a doxygen builder in code, but we'd be happy to make it easy to implement as a third party builder, with a good API in RTD. We'd also be happy to list the extension in our docs! :)

The main site is deployed from `rel`, which shouldn't be ahead of `master`. 
Note that there are a few sphinx extensions for integrating doxygen content, notably https://github.com/michaeljones/breathe, you might want to check that out before starting from scratch.
Should I close this issue? The original question was answered (regarding to the branch) :grin: 

Do you want to continue the discussion about how to do/implement it here? If so, we should change the title of the issue since it will be confusing.
We should probably document how we do deployments and that the `rel` branch is in production. 

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -32,7 +32,7 @@
 
 master_doc = 'index'
 project = u'Read The Docs'
-copyright = u'2010, Eric Holscher, Charlie Leifer, Bobby Grace'
+copyright = u'2010-2017, Read the Docs, Inc & contributors'
 version = '1.0'
 release = '1.0'
 exclude_patterns = ['_build']


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Support latest Pygments (for improved Haskell syntax, and more)
### Change Requested
The use of `Pygments==2.2.0` (or newer, potentially) to highlight code.

### Background
As part of a [syntax highlighting problem noted on Haskell-Servant](https://github.com/haskell-servant/servant/issues/359), @teh pushed a [PR for Pygments](https://bitbucket.org/birkenfeld/pygments-main/pull-requests/685/two-haskell-fixes/diff) that was merged a while ago. 

I've submitted [a Pygments dependency bump in the Servant codebase](https://github.com/haskell-servant/servant/compare/master...declension:359-upgrade-pygments) which works locally, but as we use https://readthedocs.org for actually hosting, it won't help solve the real problem unless Pygments is upgraded here too...


Discussion:

Resolving patch:
diff --git a/readthedocs/doc_builder/python_environments.py b/readthedocs/doc_builder/python_environments.py
--- a/readthedocs/doc_builder/python_environments.py
+++ b/readthedocs/doc_builder/python_environments.py
@@ -103,7 +103,7 @@ def setup_base(self):
     def install_core_requirements(self):
         requirements = [
             'sphinx==1.5.3',
-            'Pygments==2.1.3',
+            'Pygments==2.2.0',
             'setuptools==28.8.0',
             'docutils==0.13.1',
             'mkdocs==0.15.0',
@@ -194,7 +194,7 @@ def install_core_requirements(self):
         # Use conda for requirements it packages
         requirements = [
             'sphinx==1.3.5',
-            'Pygments==2.1.1',
+            'Pygments==2.2.0',
             'docutils==0.12',
             'mock',
             'pillow==3.0.0',


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Update conda requirements for Pillow
## Details

* Project URL: https://readthedocs.org/projects/jupyterhub/
* Build URL (if applicable): https://readthedocs.org/projects/jupyterhub/builds/5242702/
* Read the Docs username (if applicable): willingc

## Expected Result

I didn't expect the stable/0.7.2 docs to build. If they did run, I expected the docs to build without failing. 

## Actual Result

Build is failing since Pillow 3.0.0 is not compatible with Python 3.6. I'm not sure what triggered the build for JupyterHub stable/0.7.2 docs. I triggered the last failure but the other two just appeared :bat: 👻 

My recommendation would be allow Pillow to float >=3.0.0 for conda builds to prevent mismatches when there is a newer Python release like 3.6.

Happy to submit the PR if needed.


Discussion:

Resolving patch:
diff --git a/readthedocs/doc_builder/python_environments.py b/readthedocs/doc_builder/python_environments.py
--- a/readthedocs/doc_builder/python_environments.py
+++ b/readthedocs/doc_builder/python_environments.py
@@ -197,7 +197,7 @@ def install_core_requirements(self):
             'Pygments==2.2.0',
             'docutils==0.12',
             'mock',
-            'pillow==3.0.0',
+            'pillow>=3.0.0',
             'sphinx_rtd_theme==0.1.7',
             'alabaster>=0.7,<0.8,!=0.7.5',
         ]


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Support `maxAge` query parameter to Shields.io in badge redirect
## Details

Shields.io allows passing a `maxAge` query parameter that should address #1612, but this is not supported by the RTD redirect.

## Expected Result

The `Location` header from `curl -v https://readthedocs.org/projects/pyphi/badge/?version=latest&style=flat-square&maxAge=600` should be:
```
Location: https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat-square&maxAge=600
```

## Actual Result

`maxAge` is ignored, though `style` is respected.

The `Location` header from `curl -v https://readthedocs.org/projects/pyphi/badge/?version=latest&style=flat-square&maxAge=600` is:
```
Location: https://img.shields.io/badge/docs-latest-brightgreen.svg?style=flat-square
```


Discussion:
This would be a great contribution! The code is here: https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/projects/views/public.py#L119

Resolving patch:
diff --git a/readthedocs/projects/views/public.py b/readthedocs/projects/views/public.py
--- a/readthedocs/projects/views/public.py
+++ b/readthedocs/projects/views/public.py
@@ -107,27 +107,34 @@ def project_badge(request, project_slug):
     """Return a sweet badge for the project"""
     version_slug = request.GET.get('version', LATEST)
     style = request.GET.get('style', 'flat')
+    # Default to 24 hour cache lifetime
+    max_age = request.GET.get('maxAge', 86400)
     try:
         version = Version.objects.public(request.user).get(
             project__slug=project_slug, slug=version_slug)
     except Version.DoesNotExist:
         url = (
-            'https://img.shields.io/badge/docs-unknown%20version-yellow.svg?style={style}'
-            .format(style=style))
+            'https://img.shields.io/badge/docs-unknown%20version-yellow.svg'
+            '?style={style}&maxAge={max_age}'
+            .format(style=style, max_age=max_age))
         return HttpResponseRedirect(url)
     version_builds = version.builds.filter(type='html', state='finished').order_by('-date')
     if not version_builds.exists():
         url = (
-            'https://img.shields.io/badge/docs-no%20builds-yellow.svg?style={style}'
-            .format(style=style))
+            'https://img.shields.io/badge/docs-no%20builds-yellow.svg'
+            '?style={style}&maxAge={max_age}'
+            .format(style=style, max_age=max_age))
         return HttpResponseRedirect(url)
     last_build = version_builds[0]
     if last_build.success:
         color = 'brightgreen'
     else:
         color = 'red'
-    url = 'https://img.shields.io/badge/docs-%s-%s.svg?style=%s' % (
-        version.slug.replace('-', '--'), color, style)
+    url = (
+        'https://img.shields.io/badge/docs-{version}-{color}.svg'
+        '?style={style}&maxAge={max_age}'
+        .format(version=version.slug.replace('-', '--'), color=color,
+                style=style, max_age=max_age))
     return HttpResponseRedirect(url)
 
 


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Build failing since sphinx-rtd-theme can't be found
I believe a recent PR introduced an error which is preventing the theme from being loaded.

https://readthedocs.org/projects/jupyterhub/builds/5713289/

I think that perhaps [this line](https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/doc_builder/python_environments.py#L229) should be `sphinx_rtd_theme`.

cc/ @ericholscher 


Discussion:

Resolving patch:
diff --git a/readthedocs/doc_builder/python_environments.py b/readthedocs/doc_builder/python_environments.py
--- a/readthedocs/doc_builder/python_environments.py
+++ b/readthedocs/doc_builder/python_environments.py
@@ -226,7 +226,7 @@ def install_core_requirements(self):
             pip_requirements.append('mkdocs')
         else:
             pip_requirements.append('readthedocs-sphinx-ext')
-            requirements.extend(['sphinx', 'sphinx-rtd-theme'])
+            requirements.extend(['sphinx', 'sphinx_rtd_theme'])
 
         cmd = [
             'conda',


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Click twice in the 404 sustainability link doesn't work
## Steps to reproduce it

1. Go to https://readthedocs.org/humitos
2. You will see a 404 page with a sustainability link (https://readthedocs.org/sustainability/click/90/EdfO7Jed1YQr/)
3. Click on it
4. It goes to Sentry home page
5. Go back and click it again

## Expected Result

Go to Sentry again.

## Actual Result

You get **a new** 404 page with a new link :)



Discussion:

Resolving patch:
diff --git a/readthedocs/donate/views.py b/readthedocs/donate/views.py
--- a/readthedocs/donate/views.py
+++ b/readthedocs/donate/views.py
@@ -135,7 +135,6 @@ def click_proxy(request, promo_id, hash):
             )
         )
         cache.incr(promo.cache_key(type=CLICKS, hash=hash))
-        raise Http404('Invalid click. This has been logged.')
     return redirect(promo.link)
 
 
@@ -165,7 +164,6 @@ def view_proxy(request, promo_id, hash):
             )
         )
         cache.incr(promo.cache_key(type=VIEWS, hash=hash))
-        raise Http404('Invalid click. This has been logged.')
     return redirect(promo.image)
 
 


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Use `--upgrade` flag for user environment `pip install`
Addresses #3027.

I'm not sure if this will cause undesirable side-effects, so more discussion is needed—but I figured I'd make the PR in any case.


Discussion:
@ericholscher, by the way, I'd be happy to help out triaging tickets—would you grant me the necessary permissions?
Sure thing. I added you. I know there is a reason we removed the call to `-U` on pip, but I can't remember what it was. I'll do a bit of research and see if I can remember what issue it was causing. 
Sounds good. Thanks for the permissions!
I vaguely recall a problem with pip package caching while forcing upgrades. I'm going to bump up plans to test out feature flagging on projects this week. This is yet another case where we could push out a change on a sample and see if it blows up.
@agjohnson, sounds good! Let me know if I can help.

Resolving patch:
diff --git a/readthedocs/doc_builder/python_environments.py b/readthedocs/doc_builder/python_environments.py
--- a/readthedocs/doc_builder/python_environments.py
+++ b/readthedocs/doc_builder/python_environments.py
@@ -152,7 +152,7 @@ def install_core_requirements(self):
             self.venv_bin(filename='pip'),
             'install',
             '--use-wheel',
-            '-U',
+            '--upgrade',
             '--cache-dir',
             self.project.pip_cache_path,
         ]
@@ -182,14 +182,21 @@ def install_user_requirements(self):
                         break
 
         if requirements_file_path:
-            self.build_env.run(
+            args = [
                 'python',
                 self.venv_bin(filename='pip'),
                 'install',
+            ]
+            if self.project.has_feature(Feature.PIP_ALWAYS_UPGRADE):
+                args += ['--upgrade']
+            args += [
                 '--exists-action=w',
                 '--cache-dir',
                 self.project.pip_cache_path,
                 '-r{0}'.format(requirements_file_path),
+            ]
+            self.build_env.run(
+                *args,
                 cwd=self.checkout_path,
                 bin_path=self.venv_bin()
             )
diff --git a/readthedocs/projects/models.py b/readthedocs/projects/models.py
--- a/readthedocs/projects/models.py
+++ b/readthedocs/projects/models.py
@@ -1003,10 +1003,12 @@ def add_features(sender, **kwargs):
     # may be added by other packages
     USE_SPHINX_LATEST = 'use_sphinx_latest'
     USE_SETUPTOOLS_LATEST = 'use_setuptools_latest'
+    PIP_ALWAYS_UPGRADE = 'pip_always_upgrade'
 
     FEATURES = (
         (USE_SPHINX_LATEST, _('Use latest version of Sphinx')),
         (USE_SETUPTOOLS_LATEST, _('Use latest version of setuptools')),
+        (PIP_ALWAYS_UPGRADE, _('Always run pip install --upgrade')),
     )
 
     projects = models.ManyToManyField(


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Build version forms hit deprecated /build endpoint
The `/build` endpoint was deprecated recently, as it was a security problem allowing builds without authorization. We are still posting to this endpoint on the project dashboard and the build listing page "build version" forms.

We should replace this with an endpoint that uses CSRF token and stop posting to `/build/`.

Right now, posting to this endpoint on a project that does not have the `Feature.ALLOW_DEPRECATED_WEBHOOKS` feature will result in "This endpoint is deprecated" error.


Discussion:

Resolving patch:
diff --git a/readthedocs/builds/views.py b/readthedocs/builds/views.py
--- a/readthedocs/builds/views.py
+++ b/readthedocs/builds/views.py
@@ -6,11 +6,14 @@
 
 from django.shortcuts import get_object_or_404
 from django.views.generic import ListView, DetailView
-from django.http import HttpResponsePermanentRedirect
+from django.http import HttpResponsePermanentRedirect, HttpResponseRedirect
 from django.conf import settings
+from django.contrib.auth.decorators import login_required
 from django.core.urlresolvers import reverse
+from django.utils.decorators import method_decorator
 
 from readthedocs.builds.models import Build, Version
+from readthedocs.core.utils import trigger_build
 from readthedocs.projects.models import Project
 
 from redis import Redis, ConnectionError
@@ -33,7 +36,26 @@ def get_queryset(self):
         return queryset
 
 
-class BuildList(BuildBase, ListView):
+class BuildTriggerMixin(object):
+
+    @method_decorator(login_required)
+    def post(self, request, project_slug):
+        project = get_object_or_404(
+            Project.objects.for_admin_user(self.request.user),
+            slug=project_slug
+        )
+        version_slug = request.POST.get('version_slug')
+        version = get_object_or_404(
+            Version,
+            project=project,
+            slug=version_slug,
+        )
+
+        trigger_build(project=project, version=version)
+        return HttpResponseRedirect(reverse('builds_project_list', args=[project.slug]))
+
+
+class BuildList(BuildBase, BuildTriggerMixin, ListView):
 
     def get_context_data(self, **kwargs):
         context = super(BuildList, self).get_context_data(**kwargs)
diff --git a/readthedocs/projects/views/public.py b/readthedocs/projects/views/public.py
--- a/readthedocs/projects/views/public.py
+++ b/readthedocs/projects/views/public.py
@@ -26,6 +26,7 @@
 from .base import ProjectOnboardMixin
 from readthedocs.builds.constants import LATEST
 from readthedocs.builds.models import Version
+from readthedocs.builds.views import BuildTriggerMixin
 from readthedocs.projects.models import Project, ImportedFile
 from readthedocs.search.indexes import PageIndex
 from readthedocs.search.views import LOG_TEMPLATE
@@ -67,7 +68,7 @@ def get_context_data(self, **kwargs):
 project_index = ProjectIndex.as_view()
 
 
-class ProjectDetailView(ProjectOnboardMixin, DetailView):
+class ProjectDetailView(BuildTriggerMixin, ProjectOnboardMixin, DetailView):
 
     """Display project onboard steps"""
 


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Build passed but there is an error on the build page
## Details

* Project URL: https://readthedocs.org/projects/mujoco/
* Build URL (if applicable): https://readthedocs.org/projects/mujoco/builds/6291303/
* Read the Docs username (if applicable):  ethanabrooks

## Expected Result

Matching the build status with the message

## Actual Result

It says the build Passed but it has Failed

![captura de pantalla_2017-11-19_16-43-48](https://user-images.githubusercontent.com/244656/32995885-007a55b2-cd49-11e7-8dca-8381fdb1d90a.png)

Related: #3284 


Discussion:
This might be related to #3280 

Resolving patch:
diff --git a/readthedocs/projects/tasks.py b/readthedocs/projects/tasks.py
--- a/readthedocs/projects/tasks.py
+++ b/readthedocs/projects/tasks.py
@@ -196,6 +196,7 @@ def run_setup(self, record=True):
             if not isinstance(self.setup_env.failure, vcs_support_utils.LockTimeout):
                 self.send_notifications()
 
+            self.setup_env.update_build(state=BUILD_STATE_FINISHED)
             return False
 
         if self.setup_env.successful and not self.project.has_valid_clone:
@@ -257,6 +258,8 @@ def run_build(self, docker=False, record=True):
 
         if self.build_env.failed:
             self.send_notifications()
+
+        self.build_env.update_build(state=BUILD_STATE_FINISHED)
         build_complete.send(sender=Build, build=self.build_env.build)
 
     @staticmethod


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Can't edit canonical domain
## Details

Putting in a canonical domain and then trying to change the domain name results in "Only 1 Domain can be canonical at a time."

## Expected Result

Domain is updated with new domain name

## Actual Result

Form errors.



Discussion:

Resolving patch:
diff --git a/readthedocs/projects/forms.py b/readthedocs/projects/forms.py
--- a/readthedocs/projects/forms.py
+++ b/readthedocs/projects/forms.py
@@ -570,9 +570,10 @@ def clean_domain(self):
 
     def clean_canonical(self):
         canonical = self.cleaned_data['canonical']
+        _id = self.initial.get('id')
         if canonical and Domain.objects.filter(
-                project=self.project, canonical=True).exclude(
-                    domain=self.cleaned_data['domain']).exists():
+                project=self.project, canonical=True
+        ).exclude(pk=_id).exists():
             raise forms.ValidationError(
                 _('Only 1 Domain can be canonical at a time.'))
         return canonical


Test patch:
diff --git a/readthedocs/rtd_tests/tests/test_domains.py b/readthedocs/rtd_tests/tests/test_domains.py
--- a/readthedocs/rtd_tests/tests/test_domains.py
+++ b/readthedocs/rtd_tests/tests/test_domains.py
@@ -81,6 +81,26 @@ def test_https(self):
                           project=self.project)
         self.assertFalse(form.is_valid())
 
+    def test_canonical_change(self):
+        """Make sure canonical can be properly changed"""
+        form = DomainForm({'domain': 'example.com', 'canonical': True},
+                          project=self.project)
+        self.assertTrue(form.is_valid())
+        domain = form.save()
+        self.assertEqual(domain.domain, 'example.com')
+
+        form = DomainForm({'domain': 'example2.com', 'canonical': True},
+                          project=self.project)
+        self.assertFalse(form.is_valid())
+        self.assertEqual(form.errors['canonical'][0], 'Only 1 Domain can be canonical at a time.')
+
+        form = DomainForm({'domain': 'example2.com', 'canonical': True},
+                          project=self.project,
+                          instance=domain)
+        self.assertTrue(form.is_valid())
+        domain = form.save()
+        self.assertEqual(domain.domain, 'example2.com')
+
 
 class TestAPI(TestCase):