internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: readthedocs/readthedocs.org

Issue:
Always zero value for the first day in traffic analytics.
https://github.com/readthedocs/readthedocs.org/blob/78e6c08e8155f98a976ac7eeeac6ba06ef9cbf61/readthedocs/analytics/models.py#L102-L116

I think because the `date__gt` filter is used (instead of `date__gte`) the first value (a day exactly a month ago) doesn't get to the output dictionary `count_dict`. So `count_dict.get(date) or 0` is evaluated to zero every time.


Discussion:

Resolving patch:
diff --git a/readthedocs/analytics/models.py b/readthedocs/analytics/models.py
--- a/readthedocs/analytics/models.py
+++ b/readthedocs/analytics/models.py
@@ -104,7 +104,7 @@ def page_views_by_date(cls, project_slug, since=None):
 
         queryset = cls.objects.filter(
             project__slug=project_slug,
-            date__gt=since,
+            date__gte=since,
         ).values('date').annotate(total_views=Sum('view_count')).order_by('date')
 
         count_dict = dict(


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Improve performance for concurrency builds DB query
We start getting timeout when calling `/api/v2/builds/concurrent/` from builders to know `(limit_reached, concurrent, max_concurrent)` for a particular project.

We need to improve the queryset that works over the `Build` model because sometimes it takes ~4s or more. It usually takes ~200ms, so I guess this is because the project is/has subprojects/translations since in that case the query gets more complex.

This is the query we need to optimize,

https://github.com/readthedocs/readthedocs.org/blob/2e9cb8dd85c5f8a55ab085abe58082fe3f2c6799/readthedocs/builds/querysets.py#L129-L177

Sentry issues: https://sentry.io/organizations/read-the-docs/issues/1719520575/?project=148442&query=is%3Aunresolved&statsPeriod=14d


Discussion:

Resolving patch:
diff --git a/readthedocs/builds/querysets.py b/readthedocs/builds/querysets.py
--- a/readthedocs/builds/querysets.py
+++ b/readthedocs/builds/querysets.py
@@ -1,8 +1,10 @@
 """Build and Version QuerySet classes."""
+import datetime
 import logging
 
 from django.db import models
 from django.db.models import Q
+from django.utils import timezone
 
 from readthedocs.core.utils.extend import SettingsOverrideObject
 from readthedocs.projects import constants
@@ -143,7 +145,11 @@ def concurrent(self, project):
         :returns: limit_reached, number of concurrent builds, number of max concurrent
         """
         limit_reached = False
-        query = Q(project__slug=project.slug)
+        query = Q(
+            project__slug=project.slug,
+            # Limit builds to 5 hours ago to speed up the query
+            date__gte=timezone.now() - datetime.timedelta(hours=5),
+        )
 
         if project.main_language_project:
             # Project is a translation, counts all builds of all the translations


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Enable intersphinx support for hoverxref in our documentation
While writing #8283, I realized that we still do not enable intersphinx support in our sphinx-hoverxref documentation. More info here:

https://blog.readthedocs.com/hoverxref-intersphinx/

I think it would be nice to do so.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -53,7 +53,23 @@ def get_version():
     'django': ('https://docs.djangoproject.com/en/1.11/', 'https://docs.djangoproject.com/en/1.11/_objects/'),
     'sphinx': ('https://www.sphinx-doc.org/en/master/', None),
     'pip': ('https://pip.pypa.io/en/stable/', None),
+    'nbsphinx': ('https://nbsphinx.readthedocs.io/en/0.8.6/', None),
+    'myst-nb': ('https://myst-nb.readthedocs.io/en/v0.12.3/', None),
+    'ipywidgets': ('https://ipywidgets.readthedocs.io/en/7.6.3/', None),
+    'jupytext': ('https://jupytext.readthedocs.io/en/stable/', None),
+    'ipyleaflet': ('https://ipyleaflet.readthedocs.io/en/stable/', None),
+    'poliastro': ('https://docs.poliastro.space/en/v0.15.2/', None),
+    'qiskit': ('https://qiskit.org/documentation/', None),
+    'myst-parser': ('https://myst-parser.readthedocs.io/en/v0.15.1/', None),
 }
+hoverxref_intersphinx = [
+   "sphinx",
+   "pip",
+   "nbsphinx",
+   "myst-nb",
+   "ipywidgets",
+   "jupytext",
+]
 htmlhelp_basename = 'ReadTheDocsdoc'
 latex_documents = [
     ('index', 'ReadTheDocs.tex', 'Read the Docs Documentation',
@@ -107,8 +123,6 @@ def get_version():
 # Activate autosectionlabel plugin
 autosectionlabel_prefix_document = True
 
-numfig = True
-
 # sphinx-notfound-page
 # https://github.com/readthedocs/sphinx-notfound-page
 notfound_context = {


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Failed Pull Request Build on RTD shows as pending on GitHub
Currently when a pull request build fails on RTD. It appears the GitHub status for that pull request is not updated as having failed, but remains marked as pending.

## Details

* GitHub PR: https://github.com/rapidsai/ucx-py/pull/734
* Read the Docs project URL: https://readthedocs.org/projects/ucx-py/
* Build URL (if applicable): https://readthedocs.org/projects/ucx-py/builds/13749560/
* Read the Docs username (if applicable):

## Expected Result

Once the RTD pull request build fails, it would be good to have the GitHub pull request status also marked as having failed.

## Actual Result

Here's a screenshot from RTD for the pull request. Note the build has failed

<img width="1792" alt="Screen Shot 2021-05-12 at 12 15 34 PM" src="https://user-images.githubusercontent.com/3019665/118031781-c412fe00-b31b-11eb-8c70-f3c79df6d97b.png">

Here's the screenshot of the GitHub pull request's CI statuses. Note RTD is shown as still pending

<img width="1792" alt="Screen Shot 2021-05-12 at 12 15 13 PM" src="https://user-images.githubusercontent.com/3019665/118031731-b8bfd280-b31b-11eb-849d-7c2c8c0afc19.png">





Discussion:
(apologies if this was in another issue somewhere. was unable to find it after searching. also feel free to move if this is not the right repo for this issue)
This may be a temporary failure when contacting GitHub. We could probably re-try the task up to 3 times if it fails at https://github.com/readthedocs/readthedocs.org/blob/f32ba4d4ca4e935b68bacf337cd88ac4ffdd12a4/readthedocs/projects/tasks.py#L1880 (it returns `False`)

Resolving patch:
diff --git a/readthedocs/builds/tasks.py b/readthedocs/builds/tasks.py
--- a/readthedocs/builds/tasks.py
+++ b/readthedocs/builds/tasks.py
@@ -346,7 +346,11 @@ def sync_versions_task(project_pk, tags_data, branches_data, **kwargs):
     return True
 
 
-@app.task(queue='web')
+@app.task(
+    max_retries=3,
+    default_retry_delay=60,
+    queue='web'
+)
 def send_build_status(build_pk, commit, status, link_to_build=False):
     """
     Send Build Status to Git Status API for project external versions.
@@ -367,7 +371,7 @@ def send_build_status(build_pk, commit, status, link_to_build=False):
 
     provider_name = build.project.git_provider_name
 
-    log.info('Sending build status. build=%s, project=%s', build.pk, build.project.slug)
+    log.info('Sending build status. build=%s project=%s', build.pk, build.project.slug)
 
     if provider_name in [GITHUB_BRAND, GITLAB_BRAND]:
         # get the service class for the project e.g: GitHubService.
@@ -445,7 +449,7 @@ def send_build_status(build_pk, commit, status, link_to_build=False):
             notification.send()
 
         log.info(
-            'No social account or repository permission available for %s',
+            'No social account or repository permission available. project=%s',
             build.project.slug
         )
         return False


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Project slugs with multiple dashes can't load PR builds
For a period of time, projects could end up with a slightly invalid slug, with double dashes. This could happen if the project name included a utf8 character that would not slugify or even a dash/underscore. New projects seem to slug properly now, but for any project that was created in the period where these slugs were possible, PR builds will not work due to pattern matching bugs.

The logic here simply breaks up the first `--`, which for a project slug like `slug--bug--1.org.readthedocs.build`, would not parse correctly. Bug is here:

https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/proxito/middleware.py#L82


Discussion:

Resolving patch:
diff --git a/readthedocs/proxito/middleware.py b/readthedocs/proxito/middleware.py
--- a/readthedocs/proxito/middleware.py
+++ b/readthedocs/proxito/middleware.py
@@ -86,7 +86,7 @@ def map_host_to_project_slug(request):  # pylint: disable=too-many-return-statem
         # Serve custom versions on external-host-domain
         if external_domain_parts == host_parts[1:]:
             try:
-                project_slug, version_slug = host_parts[0].split('--', 1)
+                project_slug, version_slug = host_parts[0].rsplit('--', 1)
                 request.external_domain = True
                 request.host_version_slug = version_slug
                 log.debug('Proxito External Version Domain: host=%s', host)


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Use `django-storages`'s manifest files class
`django-storages` already implemented the manifest files class (see https://github.com/jschneier/django-storages/issues/733#issuecomment-989005340) so we can use it directly and remove our override for this in:

https://github.com/readthedocs/readthedocs.org/blob/933c316f5228423f9f871b3da858a01491f79925/readthedocs/storage/s3_storage.py#L55-L61


Discussion:

Resolving patch:
diff --git a/readthedocs/storage/s3_storage.py b/readthedocs/storage/s3_storage.py
--- a/readthedocs/storage/s3_storage.py
+++ b/readthedocs/storage/s3_storage.py
@@ -10,9 +10,8 @@
 # Disable abstract method because we are not overriding all the methods
 # pylint: disable=abstract-method
 from django.conf import settings
-from django.contrib.staticfiles.storage import ManifestFilesMixin
 from django.core.exceptions import ImproperlyConfigured
-from storages.backends.s3boto3 import S3Boto3Storage
+from storages.backends.s3boto3 import S3Boto3Storage, S3ManifestStaticStorage
 
 from readthedocs.builds.storage import BuildMediaStorageMixin
 
@@ -52,7 +51,11 @@ def __init__(self, *args, **kwargs):
             )
 
 
-class S3StaticStorage(OverrideHostnameMixin, ManifestFilesMixin, S3Boto3Storage):
+class S3StaticStorage(
+    OverrideHostnameMixin,
+    S3ManifestStaticStorage,
+    S3Boto3Storage
+):  # pylint: disable=too-many-ancestors
 
     """
     An AWS S3 Storage backend for static media.


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
OSError: File does not exist: staticfiles.json - Dev Install
## Details

* Read the Docs project URL: **N/A**
* Build URL (if applicable): **N/A**
* Read the Docs username (if applicable): **N/A**

## Expected Result

Show the http://community.dev.readthedocs.io/ local site and allow you to login and generate docs.

## Actual Result

![Screenshot 2022-01-14 002309](https://user-images.githubusercontent.com/12134329/149456476-6e2d0d65-9d7b-4880-b798-1ee74985870c.png)

This error also shows up when you try to go to http://community.dev.readthedocs.io/ but in the form of a nicely formatted django error with details about it.



Discussion:
Hi! Did you follow the steps from https://docs.readthedocs.io/en/latest/development/install.html? Are you using the latest commit from `master` branch of this repository?

I'm not able to reproduce your error locally by following the steps from that link. We may need more debugging information to be able to trace down your error.
Same with me on latest/master release, fallback to 6.3.3 works.
Followed all step as per https://docs.readthedocs.io/en/latest/development/install.html 
> Hi! Did you follow the steps from https://docs.readthedocs.io/en/latest/development/install.html? Are you using the latest commit from `master` branch of this repository?
> 
> I'm not able to reproduce your error locally by following the steps from that link. We may need more debugging information to be able to trace down your error.

Yes I used the latest commit from the master branch and followed https://docs.readthedocs.io/en/latest/development/install.html

Resolving patch:
diff --git a/readthedocs/settings/docker_compose.py b/readthedocs/settings/docker_compose.py
--- a/readthedocs/settings/docker_compose.py
+++ b/readthedocs/settings/docker_compose.py
@@ -167,9 +167,6 @@ def show_debug_toolbar(request):
     S3_STATIC_STORAGE_OVERRIDE_HOSTNAME = 'community.dev.readthedocs.io'
     S3_MEDIA_STORAGE_OVERRIDE_HOSTNAME = 'community.dev.readthedocs.io'
 
-    AWS_AUTO_CREATE_BUCKET = True
-    AWS_DEFAULT_ACL = 'public-read'
-    AWS_BUCKET_ACL = 'public-read'
     AWS_S3_ENCRYPTION = False
     AWS_S3_SECURE_URLS = False
     AWS_S3_USE_SSL = False


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Notification on duplicated builds
We deployed #7123 recently.

Now, I started receiving email notifications about the build being duplicated.

![Screenshot_2020-06-10_17-39-09](https://user-images.githubusercontent.com/244656/84288304-4c53b680-ab41-11ea-8a84-1bc080d821fa.png)

Build: https://readthedocs.org/projects/test-builds/builds/11228342/

I think we should not send these notifications to users.


Discussion:
Weird. I received only one notification and I haven't received anymore. I'm sure there were lot of duplicated builds recently and no notifications in my inbox.
Check if this is still present after #8815 
I recently started seeing this for all of [my projects](https://readthedocs.org/profiles/bmw/). It's pretty annoying because I'd like to be notified about build failures, but I'm getting notifications about duplicated builds on every commit.

I tried recreating my webhooks on GitHub but it didn't seem to help. It's not clear to me what's causing the duplicated build, although that seems to be a known problem https://github.com/readthedocs/readthedocs.org/issues/8015.

Resolving patch:
diff --git a/readthedocs/projects/tasks/builds.py b/readthedocs/projects/tasks/builds.py
--- a/readthedocs/projects/tasks/builds.py
+++ b/readthedocs/projects/tasks/builds.py
@@ -228,6 +228,12 @@ class UpdateDocsTask(SyncRepositoryMixin, Task):
         YAMLParseError,
     )
 
+    # Do not send notifications on failure builds for these exceptions.
+    exceptions_without_notifications = (
+        DuplicatedBuildError,
+        ProjectBuildsSkippedError,
+    )
+
     acks_late = True
     track_started = True
 
@@ -394,11 +400,12 @@ def on_failure(self, exc, task_id, args, kwargs, einfo):
             )
 
         # Send notifications for unhandled errors
-        self.send_notifications(
-            self.data.version.pk,
-            self.data.build['id'],
-            event=WebHookEvent.BUILD_FAILED,
-        )
+        if not isinstance(exc, self.exceptions_without_notifications):
+            self.send_notifications(
+                self.data.version.pk,
+                self.data.build['id'],
+                event=WebHookEvent.BUILD_FAILED,
+            )
 
         # NOTE: why we wouldn't have `self.data.build_commit` here?
         # This attribute is set when we get it after clonning the repository


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Logging: filter Sentry user build errors
Partial regression of #4590

During release of the task refactoring work, we are noticing a few exception classes filling up Sentry:

* BuildUserError
* RepositoryError

Just noting to come back to this to filter those out.


Discussion:
Today I found 

* MkDocsYAMLParseError
* ProjectConfigurationError

Resolving patch:
diff --git a/readthedocs/projects/tasks/builds.py b/readthedocs/projects/tasks/builds.py
--- a/readthedocs/projects/tasks/builds.py
+++ b/readthedocs/projects/tasks/builds.py
@@ -53,6 +53,7 @@
     DuplicatedBuildError,
     ProjectBuildsSkippedError,
     YAMLParseError,
+    MkDocsYAMLParseError,
 )
 from readthedocs.doc_builder.loader import get_builder_class
 from readthedocs.doc_builder.python_environments import Conda, Virtualenv
@@ -62,7 +63,7 @@
 from readthedocs.worker import app
 
 
-from ..exceptions import RepositoryError
+from ..exceptions import RepositoryError, ProjectConfigurationError
 from ..models import APIProject, Feature, WebHookEvent, HTMLFile, ImportedFile, Project
 from ..signals import (
     after_build,
@@ -220,12 +221,21 @@ class UpdateDocsTask(SyncRepositoryMixin, Task):
     max_retries = 5  # 5 per normal builds, 25 per concurrency limited
     default_retry_delay = 7 * 60
 
-    # Expected exceptions that will be logged as info only and not retried
+    # Expected exceptions that will be logged as info only and not retried.
+    # These exceptions are not sent to Sentry either because we are using
+    # ``SENTRY_CELERY_IGNORE_EXPECTED=True``.
+    #
+    # All exceptions generated by a user miss-configuration should be listed
+    # here. Actually, every subclass of ``BuildUserError``.
     throws = (
         DuplicatedBuildError,
         ProjectBuildsSkippedError,
         ConfigError,
         YAMLParseError,
+        BuildUserError,
+        RepositoryError,
+        MkDocsYAMLParseError,
+        ProjectConfigurationError,
     )
 
     acks_late = True


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Duplicate build sends GH PR status, showing failure until the first build finishes
## Details

## Expected Result

I expect GH to show "build running" with a link pointing to the first build for that commit

## Actual Result

GH shows a failed build in the status result, pointing to the build that failed with `Duplicated build`. This is replaced with a ✔️ success after the first build finishes and overwrites the status. 


Discussion:
Good catch! I think we should probably avoid sending the github notification status when we detect the build is duplicated. This happens at `on_failure` or `after_return`, I think.

Resolving patch:
diff --git a/readthedocs/projects/tasks/builds.py b/readthedocs/projects/tasks/builds.py
--- a/readthedocs/projects/tasks/builds.py
+++ b/readthedocs/projects/tasks/builds.py
@@ -231,6 +231,9 @@ class UpdateDocsTask(SyncRepositoryMixin, Task):
         ProjectBuildsSkippedError,
     )
 
+    # Do not send external build status on failure builds for these exceptions.
+    exceptions_without_external_build_status = (DuplicatedBuildError,)
+
     acks_late = True
     track_started = True
 
@@ -407,7 +410,9 @@ def on_failure(self, exc, task_id, args, kwargs, einfo):
         # Oh, I think this is to differentiate a task triggered with
         # `Build.commit` than a one triggered just with the `Version` to build
         # the _latest_ commit of it
-        if self.data.build_commit:
+        if self.data.build_commit and not isinstance(
+            exc, self.exceptions_without_external_build_status
+        ):
             send_external_build_status(
                 version_type=self.data.version.type,
                 build_pk=self.data.build['id'],


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Data Science landing page in docs
This is an initial start of a single page to link people when we talk about our grant work and data science outreach.

This was converted from a note, I'm sure there is more context somewhere though.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -34,6 +34,7 @@
     "sphinx.ext.autodoc",
     "sphinx.ext.intersphinx",
     "sphinxcontrib.httpdomain",
+    "sphinxcontrib.video",
     "djangodocs",
     "doc_extensions",
     "sphinx_tabs.tabs",
@@ -42,6 +43,7 @@
     "hoverxref.extension",
     "sphinx_search.extension",
     "sphinxemoji.sphinxemoji",
+    "sphinx_design",
     "myst_parser",
 ]
 
@@ -89,9 +91,11 @@
     "myst-parser": ("https://myst-parser.readthedocs.io/en/stable/", None),
     "writethedocs": ("https://www.writethedocs.org/", None),
     "jupyterbook": ("https://jupyterbook.org/en/stable/", None),
+    "executablebook": ("https://executablebooks.org/en/latest/", None),
     "rst-to-myst": ("https://rst-to-myst.readthedocs.io/en/stable/", None),
     "rtd": ("https://docs.readthedocs.io/en/stable/", None),
     "rtd-dev": ("https://dev.readthedocs.io/en/latest/", None),
+    "jupyter": ("https://docs.jupyter.org/en/latest/", None),
 }
 # Redundant in Sphinx 5.0
 # https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html#confval-intersphinx_disabled_reftypes


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Telemetry: track `extensions` and `html_theme` Sphinx variables
While working on https://github.com/sphinx-doc/sphinx/issues/10070, @AA-Turner ask for the content of `extensions` and `html_theme` and I think it could be useful to store in the Telemetry database somehow. Considering these values may be dynamically populated, we should find a way to execute this code and grab the final value of them.


Discussion:
If your metrics are able to work with Sphinx extensions, this could be read as `app.extensions` and `app.config.html_theme` by hooking into the `config-inited` event.

A

Resolving patch:
diff --git a/readthedocs/telemetry/collectors.py b/readthedocs/telemetry/collectors.py
--- a/readthedocs/telemetry/collectors.py
+++ b/readthedocs/telemetry/collectors.py
@@ -1,6 +1,7 @@
 """Data collectors."""
 
 import json
+import os
 
 import dparse
 import structlog
@@ -89,6 +90,37 @@ def collect(self):
                 "all": all_apt_packages,
             },
         }
+        data["doctool"] = self._get_doctool()
+        return data
+
+    def _get_doctool_name(self):
+        if self.version.is_sphinx_type:
+            return "sphinx"
+
+        if self.version.is_mkdocs_type:
+            return "mkdocs"
+
+        return "generic"
+
+    def _get_doctool(self):
+        data = {
+            "name": self._get_doctool_name(),
+            "extensions": [],
+            "html_theme": "",
+        }
+
+        if self._get_doctool_name() != "sphinx":
+            return data
+
+        conf_py_dir = os.path.join(
+            self.checkout_path,
+            os.path.dirname(self.config.sphinx.configuration),
+        )
+        filepath = os.path.join(conf_py_dir, "_build", "json", "telemetry.json")
+        if os.path.exists(filepath):
+            with open(filepath, "r") as json_file:
+                content = json_file.read()
+            data.update(self._safe_json_loads(content, {}))
         return data
 
     def _get_all_conda_packages(self):


Test patch:
Repository: readthedocs/readthedocs.org

Issue:
Docs: use Plausible for analytics
We started using Plausible on our new website, we should eventually consolidate our own docs site there as well.

This might take a little bit of thought, but isn't a lot of work otherwise. Is there anything we care to preserve from GA or replicate from GA before migrating?


Discussion:
Would it be sufficient to simply fetch and archive a few statistical reports on statistics we would want to refer back to? I'm thinking specifically about metrics we can use to assess how things are developing?

* Total site usage
* Individual page hits
* Geographical region
* Browser
* Search terms
* Top entry page / exit page

fin.
I'm bumping this up for the next sprint. We have been talking about this lately. We want to add Plausible in:

- [x] blog
- [ ] our documentation

That way, we will have everything in one place (together with about.readthedocs.com) and also be able to define some goals.
It's already on the blog, but adding it to our docs shouldn't be hard. 👍 

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -160,6 +160,8 @@
     # TODO: remove once we support different rtd config
     # files per project.
     "conf_py_path": f"/docs/{docset}/",
+    # Use to generate the Plausible "data-domain" attribute from the template
+    "plausible_domain": f"{os.environ.get('READTHEDOCS_PROJECT')}.readthedocs.io",
 }
 
 hoverxref_auto_ref = True


Test patch: