Repository: pypi/warehouse

Issue:
Fixup indexing query
Our current indexing query, https://github.com/pypa/warehouse/blob/f6deb957b37a5c2f742ab9e97019b3828671a6d3/warehouse/search/tasks.py#L29-L54

Is kind of abhorrent in performance as it pulls far to many rows (one for each project/version/classifier) and on top of that makes 2 queries to pull version history.

I have some SQL:

```sql
SELECT
  releases.description AS releases_description,
  releases.name AS releases_name,
  releases.version AS releases_version,
  (select array_agg(version) from releases as r where r.name = releases.name) as all_versions,
  releases.author AS releases_author,
  releases.author_email AS releases_author_email,
  releases.maintainer AS releases_maintainer,
  releases.maintainer_email AS releases_maintainer_email,
  releases.home_page AS releases_home_page,
  releases.summary AS releases_summary,
  releases.keywords AS releases_keywords,
  releases.platform AS releases_platform,
  releases.download_url AS releases_download_url,
  releases.created AS releases_created,
  (select array_agg(id) from release_classifiers join trove_classifiers on trove_classifiers.id = release_classifiers.trove_id where releases.name = release_classifiers.name and releases.version = release_classifiers.version) as trove_classifiers_1_id,
  (select array_agg(trove_classifiers.classifier) from release_classifiers join trove_classifiers on trove_classifiers.id = release_classifiers.trove_id where releases.name = release_classifiers.name and releases.version = release_classifiers.version) as trove_classifiers_1_id,
  normalize_pep426_name(packages_1.name) AS normalize_pep426_name_1,
  packages_1.name AS packages_1_name
FROM (
  SELECT DISTINCT ON (releases.name) releases.name AS name, releases.version AS version FROM releases ORDER BY releases.name, pep440_is_prerelease(releases.version) NULLS LAST, releases._pypi_ordering DESC) AS release_list, releases
  LEFT OUTER JOIN packages AS packages_1 ON packages_1.name = releases.name
WHERE releases.name = release_list.name AND releases.version = release_list.version
ORDER BY releases.name
```
Thanks to the help of @eeeebbbbrrrr that pulls what we need in a single row!

Can anyone help me convert the above query into SQLAlchemy so we can continue to use the windowed_query helper https://github.com/pypa/warehouse/blob/f6deb957b37a5c2f742ab9e97019b3828671a6d3/warehouse/search/tasks.py#L56


Discussion:
```shell
warehouse=> SELECT
warehouse->   -- releases.description AS releases_description,
warehouse->   releases.name AS releases_name,
warehouse->   releases.version AS releases_version,
warehouse->   (select array_agg(version) from releases as r where r.name = releases.name) as all_versions,
warehouse->   releases.author AS releases_author,
warehouse->   releases.author_email AS releases_author_email,
warehouse->   releases.maintainer AS releases_maintainer,
warehouse->   releases.maintainer_email AS releases_maintainer_email,
warehouse->   releases.home_page AS releases_home_page,
warehouse->   releases.summary AS releases_summary,
warehouse->   releases.keywords AS releases_keywords,
warehouse->   releases.platform AS releases_platform,
warehouse->   releases.download_url AS releases_download_url,
warehouse->   releases.created AS releases_created,
warehouse->   (select array_agg(id) from release_classifiers join trove_classifiers on trove_classifiers.id = release_classifiers.trove_id where releases.name = release_classifiers.name and releases.version = release_classifiers.version) as trove_classifiers_1_id,
warehouse->   (select array_agg(trove_classifiers.classifier) from release_classifiers join trove_classifiers on trove_classifiers.id = release_classifiers.trove_id where releases.name = release_classifiers.name and releases.version = release_classifiers.version) as trove_classifiers_1_id,
warehouse->   normalize_pep426_name(packages_1.name) AS normalize_pep426_name_1,
warehouse->   packages_1.name AS packages_1_name
warehouse-> FROM (
warehouse(>   SELECT DISTINCT ON (releases.name) releases.name AS name, releases.version AS version FROM releases ORDER BY releases.name, pep440_is_prerelease(releases.version) NULLS LAST, releases._pypi_ordering DESC) AS release_list, releases
warehouse->   LEFT OUTER JOIN packages AS packages_1 ON packages_1.name = releases.name
warehouse-> WHERE releases.name = release_list.name AND releases.version = release_list.version AND releases.name = 'pip'
warehouse-> ORDER BY releases.name;
-[ RECORD 1 ]-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
releases_name             | pip
releases_version          | 10.0.0
all_versions              | {9.0.2,9.0.1,0.8.1,0.6.1,0.6.3,0.5,1.2.1,6.0.7,1.3,8.0.2,6.1.0,7.0.3,0.7.2,7.0.1,6.0.3,0.7.1,0.7,1.4,6.0.8,0.8.3,6.0.4,0.5.1,1.0.2,1.5.1,1.5.6,8.0.1,0.6.2,0.8,1.5.2,1.5,6.0.2,1.3.1,8.1.1,6.0.1,8.0.0,8.0.3,0.8.2,7.1.2,6.1.1,0.4,7.1.1,6.0,0.2,0.3.1,7.1.0,1.2,1.4.1,1.1,0.2.1,1.5.4,0.3,8.1.0,0.6,1.5.3,9.0.0,8.1.2,6.0.6,1.5.5,6.0.5,1.0.1,1.0,7.0.2,7.0.0,9.0.3,10.0.0b1,10.0.0b2,10.0.0}
releases_author           | The pip developers
releases_author_email     | python-virtualenv@groups.google.com
releases_maintainer       | 
releases_maintainer_email | 
releases_home_page        | https://pip.pypa.io/
releases_summary          | The PyPA recommended tool for installing Python packages.
releases_keywords         | easy_install distutils setuptools egg virtualenv
releases_platform         | 
releases_download_url     | 
releases_created          | 2018-04-14 11:38:05.342706
trove_classifiers_1_id    | {611,607,587,566,554,553,533,532,527,30,5,69,214,408}
trove_classifiers_1_id    | {"Programming Language :: Python :: 3.6","Programming Language :: Python :: 3.5","Programming Language :: Python :: 3.4","Programming Language :: Python :: 3.3","Programming Language :: Python :: Implementation :: PyPy","Programming Language :: Python :: Implementation :: CPython","Programming Language :: Python :: 3","Programming Language :: Python :: 2.7","Programming Language :: Python :: 2","Intended Audience :: Developers","Development Status :: 5 - Production/Stable","License :: OSI Approved :: MIT License","Programming Language :: Python","Topic :: Software Development :: Build Tools"}
normalize_pep426_name_1   | pip
packages_1_name           | pip
```
Query pulls everything we need in a single row! Performs well! (Description not queried for clarity)
I don't think the `all_versions` column requires ordering as to my knowledge it isn't exposed to users, just used for search.

But if it does, we can just reorder it when constructing the document.
I'm taking a look at this

Resolving patch:
diff --git a/warehouse/packaging/search.py b/warehouse/packaging/search.py
--- a/warehouse/packaging/search.py
+++ b/warehouse/packaging/search.py
@@ -55,11 +55,11 @@ class Meta:
 
     @classmethod
     def from_db(cls, release):
-        obj = cls(meta={"id": release.project.normalized_name})
-        obj["name"] = release.project.name
-        obj["normalized_name"] = release.project.normalized_name
-        obj["version"] = [r.version for r in release.project.all_versions]
-        obj["latest_version"] = release.project.latest_version.version
+        obj = cls(meta={"id": release.normalized_name})
+        obj["name"] = release.name
+        obj["normalized_name"] = release.normalized_name
+        obj["version"] = release.all_versions
+        obj["latest_version"] = release.latest_version
         obj["summary"] = release.summary
         obj["description"] = release.description
         obj["author"] = release.author
@@ -71,6 +71,6 @@ def from_db(cls, release):
         obj["keywords"] = release.keywords
         obj["platform"] = release.platform
         obj["created"] = release.created
-        obj["classifiers"] = list(release.classifiers)
+        obj["classifiers"] = release.classifiers
 
         return obj
diff --git a/warehouse/search/tasks.py b/warehouse/search/tasks.py
--- a/warehouse/search/tasks.py
+++ b/warehouse/search/tasks.py
@@ -14,10 +14,11 @@
 import os
 
 from elasticsearch.helpers import parallel_bulk
-from sqlalchemy import and_
-from sqlalchemy.orm import lazyload, joinedload, load_only
+from sqlalchemy import and_, func
+from sqlalchemy.orm import aliased
 
-from warehouse.packaging.models import Release
+from warehouse.packaging.models import (
+    Classifier, Project, Release, release_classifiers)
 from warehouse.packaging.search import Project as ProjectDocType
 from warehouse.search.utils import get_index
 from warehouse import tasks
@@ -27,30 +28,63 @@
 def _project_docs(db):
 
     releases_list = (
-        db.query(Release)
-          .options(load_only(
-                   "name", "version"))
-          .distinct(Release.name)
-          .order_by(
-              Release.name,
-              Release.is_prerelease.nullslast(),
-              Release._pypi_ordering.desc())
-    ).subquery('release_list')
+        db.query(Release.name, Release.version)
+        .order_by(
+            Release.name,
+            Release.is_prerelease.nullslast(),
+            Release._pypi_ordering.desc(),
+        )
+        .distinct(Release.name)
+        .subquery("release_list")
+    )
+
+    r = aliased(Release, name="r")
+
+    all_versions = (
+        db.query(func.array_agg(r.version))
+        .filter(r.name == Release.name)
+        .correlate(Release)
+        .as_scalar()
+        .label("all_versions")
+    )
+
+    classifiers = (
+        db.query(func.array_agg(Classifier.classifier))
+        .select_from(release_classifiers)
+        .join(Classifier, Classifier.id == release_classifiers.c.trove_id)
+        .filter(Release.name == release_classifiers.c.name)
+        .filter(Release.version == release_classifiers.c.version)
+        .correlate(Release)
+        .as_scalar()
+        .label("classifiers")
+    )
 
     release_data = (
-        db.query(Release)
-          .options(load_only(
-                   "summary", "description", "author",
-                   "author_email", "maintainer", "maintainer_email",
-                   "home_page", "download_url", "keywords", "platform",
-                   "created"))
-          .options(lazyload("*"),
-                   (joinedload(Release.project)
-                    .load_only("normalized_name", "name")),
-                   joinedload(Release._classifiers).load_only("classifier"))
-          .filter(and_(
-              Release.name == releases_list.c.name,
-              Release.version == releases_list.c.version))
+        db.query(
+            Release.description,
+            Release.name,
+            Release.version.label("latest_version"),
+            all_versions,
+            Release.author,
+            Release.author_email,
+            Release.maintainer,
+            Release.maintainer_email,
+            Release.home_page,
+            Release.summary,
+            Release.keywords,
+            Release.platform,
+            Release.download_url,
+            Release.created,
+            classifiers,
+            Project.normalized_name,
+            Project.name,
+        )
+        .select_from(releases_list)
+        .join(Release, and_(
+            Release.name == releases_list.c.name,
+            Release.version == releases_list.c.version))
+        .outerjoin(Release.project)
+        .order_by(Release.name)
     )
 
     for release in windowed_query(release_data, Release.name, 1000):


Test patch:
diff --git a/tests/unit/packaging/test_search.py b/tests/unit/packaging/test_search.py
--- a/tests/unit/packaging/test_search.py
+++ b/tests/unit/packaging/test_search.py
@@ -18,18 +18,16 @@
 
 def test_build_search():
     release = pretend.stub(
-        project=pretend.stub(
-            name="Foobar",
-            normalized_name="foobar",
-            all_versions=[
-                pretend.stub(version="5.0.dev0"),
-                pretend.stub(version="4.0"),
-                pretend.stub(version="3.0"),
-                pretend.stub(version="2.0"),
-                pretend.stub(version="1.0"),
-            ],
-            latest_version=pretend.stub(version="4.0"),
-        ),
+        name="Foobar",
+        normalized_name="foobar",
+        all_versions=[
+            "5.0.dev0",
+            "4.0",
+            "3.0",
+            "2.0",
+            "1.0",
+        ],
+        latest_version="4.0",
         summary="This is my summary",
         description="This is my description",
         author="Jane Author",