Repository: pulp/pulpcore

Issue:
Querying content in repository versions with lots of content is slow
**Version**
All

**Describe the bug**
When a repository version has upwards of 50k content in it, querying the content in the repository using `repo_version.get_content()` becomes extremely slow. In my testing the results haven't been consistent, however I have notice queries that can take anywhere from 200ms to 1000ms. This appears to be due to the complex joins that have to be made on version_added and version_removed.

**To Reproduce**
- Load up a repository with a lot of content
- Perform a query to fetch the last 100 objects or count how many objects are in it
- Observe query speeds that would make sonic the hedgehog faint

**Expected behavior**
Database queries should be fast. Hopefully less than 50ms for complex ones, ideally lower than that.

**Some possible solutions**
- Adding the repo version numbers directly to the RepositoryContent table speeds up queries greatly. My testing showed that this can lower the queryset speed to 80ms with 65k content in some cases.
- pulp_ansible is experimenting with querying the repository version table to look up RepositoryContent (https://github.com/pulp/pulp_ansible/blob/main/pulp_ansible/app/utils.py#L5). Results have been mixed with this. In some cases the query efficiency is substantially higher (on the order of 8x improvement). In other cases, it's roughly the same.
- Some form of memoization/caching of querysets. 



Discussion:
>Adding the repo version numbers directly to the RepositoryContent table speeds up queries greatly. My testing showed that this can lower the queryset speed to 80ms with 65k content in some cases.

I agree there's an issue, but just to elaborate on why it is the way it is:

There's a tradeoff between space, read and write speeds here.  Storing 60,000 `Content` with 100 repo versions under the current schema only requires 60,000 `RepositoryContent` presuming you don't remove and re-add any content across versions.  Additions and removals only require the saving of one new `RepositoryContent` per addition or removal.

Storing 60,000 `Content` with 100 repo versions if each `RepositoryContent` was associated with exactly one repository version would require 6,000,000 `RepositoryContent`, and each additional version would add another ~60,000.

Now apply that to larger repos or registries, such as PyPI or Dockerhub, where the number of content is in the hundred thousands or millions. 

At the time this was mostly based on theory and theassumptions might have been incorrect, but that's where we figured this schema might be safer.
One other thing is that we might want to be careful about how we do the benchmarking and generalize the results.  pulpcore < 3.25 used completely random UUID4s whereas pulpcore >3.25 uses lexicographically sortable UUID7s.  So the latter ought to be dramatically more efficient on the indexes - but those speedups probably won't apply to  content saved previously.
> > Adding the repo version numbers directly to the RepositoryContent table speeds up queries greatly. My testing showed that this can lower the queryset speed to 80ms with 65k content in some cases.
> 
> I agree there's an issue, but just to elaborate on why it is the way it is:
> 
> There's a tradeoff between space, read and write speeds here. Storing 60,000 `Content` with 100 repo versions under the current schema only requires 60,000 `RepositoryContent` presuming you don't remove and re-add any content across versions. Additions and removals only require the saving of one new `RepositoryContent` per addition or removal.
> 
> Storing 60,000 `Content` with 100 repo versions if each `RepositoryContent` was associated with exactly one repository version would require 6,000,000 `RepositoryContent`, and each additional version would add another ~60,000.
> 
> Now apply that to larger repos or registries, such as PyPI or Dockerhub, where the number of content is in the hundred thousands or millions.
> 
> At the time this was mostly based on theory and theassumptions might have been incorrect, but that's where we figured this schema might be safer.

I think the suggestion here is not to move all content references into all versions they are in (and that suggestion has been made in other places), but to instead of needing the foreign key to identify the added / removed in version, we could rely on the numerical value of it. This ought to remove the join of the RepositoryVersion table when you are interested in content for one version you already know the number of.
Aaaah. Ok.  Interesting idea, I can see how that could be an improvement
@newswangerd, I think I am not able to reproduce the observations (pulpcore:main|pulp_rpm:main).

I synced a remote fedora repository with 60k+ packages in total with pulp_rpm. I have one repository with two versions (0 - init, 1 - synced) in the testing system. I started by running db queries purely in django-admin/pulpcore-admin shell.

I am unsure whether I experience the same performance degradation as you do. There might be a case where content types in pulp_ansible differ from those in pulp_rpm. I also think that using `globals()` in `timeit.timeit` does not impact executed queries. Would you share queries you have played with? Can you tell me which of the queries I have posted resemble the ones you ran?

This is what I have tried so far:
```
In [3]: v = RpmRepository.objects.get().latest_version()

In [72]: v.get_content(Package.objects).count()
Out[72]: 69222
```
```
In [67]: timeit.timeit("v.get_content()", number=100, globals=globals())
Out[67]: 0.07517607201589271

In [68]: timeit.timeit("len(v.get_content())", number=10, globals=globals())
Out[68]: 16.806439982028678

In [69]: timeit.timeit("v.get_content().count()", number=10, globals=globals())
Out[69]: 0.7796547850011848

In [70]: timeit.timeit("len(v.get_content(Package.objects))", number=10, globals=globals())
Out[70]: 152.61631407699315     <--------- is this the bottleneck?

In [71]: timeit.timeit("v.get_content(Package.objects).count()", number=10, globals=globals())
Out[71]: 1.1137214149930514
```
```
# copied filter_content_for_repo_version from pulp_ansible; no performance difference spotted

In [75]: timeit.timeit("filter_content_for_repo_version(Content.objects, v)", number=100, globals=globals())
Out[75]: 0.1258076920057647

In [76]: timeit.timeit("len(filter_content_for_repo_version(Content.objects, v))", number=10, globals=globals())
Out[76]: 17.80737394897733

In [77]: timeit.timeit("filter_content_for_repo_version(Content.objects, v).count()", number=10, globals=globals())
Out[77]: 1.1327312650391832

In [78]: timeit.timeit("len(filter_content_for_repo_version(Package.objects, v))", number=10, globals=globals())
Out[78]: 158.9172281989595     <--------- is this the bottleneck?

In [79]: timeit.timeit("filter_content_for_repo_version(Package.objects, v).count()", number=10, globals=globals())
Out[79]: 1.1048942319466732
```

I agree with the idea of improving the performance. However, I need to get a baseline measurement for further work.
The performance is relatively the same with pulpcore 3.24 considering the above queries.
To better understand what is going on under the hood, I enabled the `DEBUG` setting in the settings file. I ran the same queries as before.

```
In [2]: v.get_content().count()
(0.082)
SELECT COUNT(*) AS "__count"
FROM "core_content"
INNER JOIN "core_repositorycontent" ON ("core_content"."pulp_id" = "core_repositorycontent"."content_id")
WHERE "core_repositorycontent"."pulp_id" IN (
    SELECT U0."pulp_id"
    FROM "core_repositorycontent" U0
    INNER JOIN "core_repositoryversion" U2 ON (U0."version_added_id" = U2."pulp_id")
    LEFT OUTER JOIN "core_repositoryversion" U3 ON (U0."version_removed_id" = U3."pulp_id")
    WHERE (U0."repository_id" = '018993dbde647f7b87c5086164f04feb'::uuid AND U2."number" <= 1 AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))
);
args = (UUID('018993db-de64-7f7b-87c5-086164f04feb'), Int4(1), Int4(1));
alias = default;
Out[2]: 69415
```
```
In [3]: v.get_content(Package.objects).count()
(0.107)
SELECT COUNT(*) AS "__count"
FROM "rpm_package"
INNER JOIN "core_content" ON ("rpm_package"."content_ptr_id" = "core_content"."pulp_id")
INNER JOIN "core_repositorycontent" ON ("core_content"."pulp_id" = "core_repositorycontent"."content_id")
WHERE "core_repositorycontent"."pulp_id" IN (
    SELECT U0."pulp_id"
    FROM "core_repositorycontent" U0
    INNER JOIN "core_repositoryversion" U2 ON (U0."version_added_id" = U2."pulp_id")
    LEFT OUTER JOIN "core_repositoryversion" U3 ON (U0."version_removed_id" = U3."pulp_id")
    WHERE (U0."repository_id" = '018993dbde647f7b87c5086164f04feb'::uuid AND U2."number" <= 1 AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))
);
args = (UUID('018993db-de64-7f7b-87c5-086164f04feb'), Int4(1), Int4(1));
alias = default;
Out[3]: 69222
```
```
In [4]: len(v.get_content())
(0.150)
SELECT
    "core_content"."pulp_id",
    "core_content"."pulp_created",
    "core_content"."pulp_last_updated",
    "core_content"."pulp_type",
    "core_content"."upstream_id",
    "core_content"."timestamp_of_interest",
    "core_content"."pulp_domain_id"
FROM
    "core_content"
INNER JOIN
    "core_repositorycontent"
    ON ("core_content"."pulp_id" = "core_repositorycontent"."content_id")
WHERE
    "core_repositorycontent"."pulp_id" IN (
        SELECT U0."pulp_id"
        FROM "core_repositorycontent" U0
        INNER JOIN "core_repositoryversion" U2
            ON (U0."version_added_id" = U2."pulp_id")
        LEFT OUTER JOIN "core_repositoryversion" U3
            ON (U0."version_removed_id" = U3."pulp_id")
        WHERE
            (U0."repository_id" = '018993dbde647f7b87c5086164f04feb'::uuid
            AND U2."number" <= 1
            AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))
    );
args = (UUID('018993db-de64-7f7b-87c5-086164f04feb'), Int4(1), Int4(1));
alias = default;
Out[4]: 69415
```
```
In [5]: len(v.get_content(Package.objects))
(4.896)
SELECT
    "core_content"."pulp_id",
    "core_content"."pulp_created",
    "core_content"."pulp_last_updated",
    "core_content"."pulp_type",
    "core_content"."upstream_id",
    "core_content"."timestamp_of_interest",
    "core_content"."pulp_domain_id",
    "rpm_package"."content_ptr_id",
    "rpm_package"."name",
    "rpm_package"."epoch",
    "rpm_package"."version",
    "rpm_package"."release",
    "rpm_package"."arch",
    "rpm_package"."evr",
    "rpm_package"."pkgId",
    "rpm_package"."checksum_type",
    "rpm_package"."summary",
    "rpm_package"."description",
    "rpm_package"."url",
    "rpm_package"."changelogs",
    "rpm_package"."files",
    "rpm_package"."requires",
    "rpm_package"."provides",
    "rpm_package"."conflicts",
    "rpm_package"."obsoletes",
    "rpm_package"."suggests",
    "rpm_package"."enhances",
    "rpm_package"."recommends",
    "rpm_package"."supplements",
    "rpm_package"."location_base",
    "rpm_package"."location_href",
    "rpm_package"."rpm_buildhost",
    "rpm_package"."rpm_group",
    "rpm_package"."rpm_license",
    "rpm_package"."rpm_packager",
    "rpm_package"."rpm_sourcerpm",
    "rpm_package"."rpm_vendor",
    "rpm_package"."rpm_header_start",
    "rpm_package"."rpm_header_end",
    "rpm_package"."size_archive",
    "rpm_package"."size_installed",
    "rpm_package"."size_package",
    "rpm_package"."time_build",
    "rpm_package"."time_file",
    "rpm_package"."is_modular",
    "rpm_package"."_pulp_domain_id"
FROM
    "rpm_package"
INNER JOIN
    "core_content"
    ON ("rpm_package"."content_ptr_id" = "core_content"."pulp_id")
INNER JOIN
    "core_repositorycontent"
    ON ("core_content"."pulp_id" = "core_repositorycontent"."content_id")
WHERE
    "core_repositorycontent"."pulp_id" IN (
        SELECT U0."pulp_id"
        FROM "core_repositorycontent" U0
        INNER JOIN "core_repositoryversion" U2
            ON (U0."version_added_id" = U2."pulp_id")
        LEFT OUTER JOIN "core_repositoryversion" U3
            ON (U0."version_removed_id" = U3."pulp_id")
        WHERE
            (U0."repository_id" = '018993dbde647f7b87c5086164f04feb'::uuid
            AND U2."number" <= 1
            AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))
    );
args = (UUID('018993db-de64-7f7b-87c5-086164f04feb'), Int4(1), Int4(1));
alias = default;
Out[5]: 69222
```

Seeing the last two queries, the only difference can be observed in one additional INNER JOIN operation and the list of selected fields.

![image](https://github.com/pulp/pulpcore/assets/8740962/04a7dccd-7789-4144-95d9-8bc4bc335903)

When I selected only one field from the table, the performance improved a lot (the query remained the same):
```
In [15]: len(v.get_content(Package.objects).only('name'))
(0.202)
SELECT
    "rpm_package"."content_ptr_id",
    "rpm_package"."name"
FROM
    "rpm_package"
INNER JOIN
    "core_content"
    ON ("rpm_package"."content_ptr_id" = "core_content"."pulp_id")
INNER JOIN
    "core_repositorycontent"
    ON ("core_content"."pulp_id" = "core_repositorycontent"."content_id")
WHERE
    "core_repositorycontent"."pulp_id" IN (
        SELECT U0."pulp_id"
        FROM "core_repositorycontent" U0
        INNER JOIN "core_repositoryversion" U2
            ON (U0."version_added_id" = U2."pulp_id")
        LEFT OUTER JOIN "core_repositoryversion" U3
            ON (U0."version_removed_id" = U3."pulp_id")
        WHERE
            (U0."repository_id" = '018993dbde647f7b87c5086164f04feb'::uuid
            AND U2."number" <= 1
            AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))
    );
```

```
In [5]: timeit.timeit("len(v.get_content(Package.objects).only('name'))", number=10, globals=globals())
Out[5]: 11.05222433299059

In [6]: timeit.timeit("len(v.get_content(Package.objects))", number=10, globals=globals())
Out[6]: 183.93376222799998
```

As demonstrated, the problem is fundamentally related to the number of selected fields in pulp_rpm.
We first need to establish a baseline measurement to track the performance improvements. I am continuing with the investigation in pulp_ansible. The content type used in tests mentioned by @newswangerd is [`AnsibleNamespaceMetadata`](https://github.com/pulp/pulp_ansible/blob/1b66045db608b1f7a54713b118252daa1c7b3cd6/pulp_ansible/app/models.py#L294). The model of the said content type contains significantly fewer fields compared to the [`Package`](https://github.com/pulp/pulp_rpm/blob/8f6b258b3d3b27d54bc3665a705447b6efb8e999/pulp_rpm/app/models/package.py#L59) model in pulp_rpm.
Additionally, I am attaching the execution plan for completeness:

```
In [7]: v.get_content().explain(verbose=True, analyze=True)
```
```
Hash Join  (cost=4167.26..6054.58 rows=7794 width=84) (actual time=114.960..162.652 rows=69415 loops=1)
  Output: core_content.pulp_id, core_content.pulp_created, core_content.pulp_last_updated, core_content.pulp_type, core_content.upstream_id, core_content.timestamp_of_interest, core_content.pulp_domain_id
  Hash Cond: (core_content.pulp_id = core_repositorycontent.content_id)
  ->  Seq Scan on public.core_content  (cost=0.00..1550.11 rows=69211 width=84) (actual time=0.450..13.670 rows=69415 loops=1)
        Output: core_content.pulp_id, core_content.pulp_created, core_content.pulp_last_updated, core_content.pulp_type, core_content.upstream_id, core_content.timestamp_of_interest, core_content.pulp_domain_id
  ->  Hash  (cost=4069.84..4069.84 rows=7794 width=16) (actual time=114.420..114.432 rows=69415 loops=1)
        Output: core_repositorycontent.content_id
        Buckets: 131072 (originally 8192)  Batches: 2 (originally 1)  Memory Usage: 3073kB
        ->  Hash Semi Join  (cost=2180.32..4069.84 rows=7794 width=16) (actual time=64.381..101.531 rows=69415 loops=1)
              Output: core_repositorycontent.content_id
              Hash Cond: (core_repositorycontent.pulp_id = u0.pulp_id)
              ->  Seq Scan on public.core_repositorycontent  (cost=0.00..1620.50 rows=69450 width=32) (actual time=0.421..4.965 rows=69415 loops=1)
                    Output: core_repositorycontent.pulp_id, core_repositorycontent.pulp_created, core_repositorycontent.pulp_last_updated, core_repositorycontent.content_id, core_repositorycontent.repository_id, core_repositorycontent.version_added_id, core_repositorycontent.version_removed_id
              ->  Hash  (cost=2082.90..2082.90 rows=7794 width=16) (actual time=63.881..63.892 rows=69415 loops=1)
                    Output: u0.pulp_id
                    Buckets: 131072 (originally 8192)  Batches: 2 (originally 1)  Memory Usage: 3073kB
                    ->  Hash Join  (cost=44.67..2082.90 rows=7794 width=16) (actual time=0.656..46.721 rows=69415 loops=1)
                          Output: u0.pulp_id
                          Inner Unique: true
                          Hash Cond: (u0.version_added_id = u2.pulp_id)
                          ->  Hash Left Join  (cost=24.18..2000.61 rows=23382 width=32) (actual time=0.114..34.120 rows=69415 loops=1)
                                Output: u0.version_added_id, u0.pulp_id
                                Inner Unique: true
                                Hash Cond: (u0.version_removed_id = u3.pulp_id)
                                Filter: ((u3.number > 1) OR (u3.number IS NULL))
                                ->  Seq Scan on public.core_repositorycontent u0  (cost=0.00..1794.12 rows=69450 width=48) (actual time=0.016..21.858 rows=69415 loops=1)
                                      Output: u0.pulp_id, u0.pulp_created, u0.pulp_last_updated, u0.content_id, u0.repository_id, u0.version_added_id, u0.version_removed_id
                                      Filter: (u0.repository_id = '0189abfe-606a-7006-b21c-e144742984bf'::uuid)
                                ->  Hash  (cost=16.30..16.30 rows=630 width=20) (actual time=0.060..0.069 rows=2 loops=1)
                                      Output: u3.pulp_id, u3.number
                                      Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                      ->  Seq Scan on public.core_repositoryversion u3  (cost=0.00..16.30 rows=630 width=20) (actual time=0.008..0.012 rows=2 loops=1)
                                            Output: u3.pulp_id, u3.number
                          ->  Hash  (cost=17.88..17.88 rows=210 width=16) (actual time=0.527..0.527 rows=2 loops=1)
                                Output: u2.pulp_id
                                Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                ->  Seq Scan on public.core_repositoryversion u2  (cost=0.00..17.88 rows=210 width=16) (actual time=0.514..0.517 rows=2 loops=1)
                                      Output: u2.pulp_id
                                      Filter: (u2.number <= 1)
Planning Time: 3.848 ms
Execution Time: 164.707 ms
```

```
In [8]: v.get_content(Package.objects).explain(verbose=True, analyze=True)
```
```
Nested Loop  (cost=4167.55..15719.79 rows=7781 width=1763) (actual time=89.768..586.934 rows=69222 loops=1)
  Output: core_content.pulp_id, core_content.pulp_created, core_content.pulp_last_updated, core_content.pulp_type, core_content.upstream_id, core_content.timestamp_of_interest, core_content.pulp_domain_id, rpm_package.content_ptr_id, rpm_package.name, rpm_package.epoch, rpm_package.version, rpm_package.release, rpm_package.arch, rpm_package.evr, rpm_package."pkgId", rpm_package.checksum_type, rpm_package.summary, rpm_package.description, rpm_package.url, rpm_package.changelogs, rpm_package.files, rpm_package.requires, rpm_package.provides, rpm_package.conflicts, rpm_package.obsoletes, rpm_package.suggests, rpm_package.enhances, rpm_package.recommends, rpm_package.supplements, rpm_package.location_base, rpm_package.location_href, rpm_package.rpm_buildhost, rpm_package.rpm_group, rpm_package.rpm_license, rpm_package.rpm_packager, rpm_package.rpm_sourcerpm, rpm_package.rpm_vendor, rpm_package.rpm_header_start, rpm_package.rpm_header_end, rpm_package.size_archive, rpm_package.size_installed, rpm_package.size_package, rpm_package.time_build, rpm_package.time_file, rpm_package.is_modular, rpm_package._pulp_domain_id
  Inner Unique: true
  ->  Hash Join  (cost=4167.26..6054.58 rows=7794 width=100) (actual time=88.933..166.249 rows=69415 loops=1)
        Output: core_repositorycontent.content_id, core_content.pulp_id, core_content.pulp_created, core_content.pulp_last_updated, core_content.pulp_type, core_content.upstream_id, core_content.timestamp_of_interest, core_content.pulp_domain_id
        Hash Cond: (core_content.pulp_id = core_repositorycontent.content_id)
        ->  Seq Scan on public.core_content  (cost=0.00..1550.11 rows=69211 width=84) (actual time=0.022..14.862 rows=69415 loops=1)
              Output: core_content.pulp_id, core_content.pulp_created, core_content.pulp_last_updated, core_content.pulp_type, core_content.upstream_id, core_content.timestamp_of_interest, core_content.pulp_domain_id
        ->  Hash  (cost=4069.84..4069.84 rows=7794 width=16) (actual time=88.847..88.851 rows=69415 loops=1)
              Output: core_repositorycontent.content_id
              Buckets: 131072 (originally 8192)  Batches: 2 (originally 1)  Memory Usage: 3073kB
              ->  Hash Semi Join  (cost=2180.32..4069.84 rows=7794 width=16) (actual time=43.669..78.114 rows=69415 loops=1)
                    Output: core_repositorycontent.content_id
                    Hash Cond: (core_repositorycontent.pulp_id = u0.pulp_id)
                    ->  Seq Scan on public.core_repositorycontent  (cost=0.00..1620.50 rows=69450 width=32) (actual time=0.008..4.498 rows=69415 loops=1)
                          Output: core_repositorycontent.pulp_id, core_repositorycontent.pulp_created, core_repositorycontent.pulp_last_updated, core_repositorycontent.content_id, core_repositorycontent.repository_id, core_repositorycontent.version_added_id, core_repositorycontent.version_removed_id
                    ->  Hash  (cost=2082.90..2082.90 rows=7794 width=16) (actual time=43.613..43.615 rows=69415 loops=1)
                          Output: u0.pulp_id
                          Buckets: 131072 (originally 8192)  Batches: 2 (originally 1)  Memory Usage: 3073kB
                          ->  Hash Join  (cost=44.67..2082.90 rows=7794 width=16) (actual time=0.047..31.974 rows=69415 loops=1)
                                Output: u0.pulp_id
                                Inner Unique: true
                                Hash Cond: (u0.version_added_id = u2.pulp_id)
                                ->  Hash Left Join  (cost=24.18..2000.61 rows=23382 width=32) (actual time=0.021..21.140 rows=69415 loops=1)
                                      Output: u0.version_added_id, u0.pulp_id
                                      Inner Unique: true
                                      Hash Cond: (u0.version_removed_id = u3.pulp_id)
                                      Filter: ((u3.number > 1) OR (u3.number IS NULL))
                                      ->  Seq Scan on public.core_repositorycontent u0  (cost=0.00..1794.12 rows=69450 width=48) (actual time=0.005..9.893 rows=69415 loops=1)
                                            Output: u0.pulp_id, u0.pulp_created, u0.pulp_last_updated, u0.content_id, u0.repository_id, u0.version_added_id, u0.version_removed_id
                                            Filter: (u0.repository_id = '0189abfe-606a-7006-b21c-e144742984bf'::uuid)
                                      ->  Hash  (cost=16.30..16.30 rows=630 width=20) (actual time=0.007..0.008 rows=2 loops=1)
                                            Output: u3.pulp_id, u3.number
                                            Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                            ->  Seq Scan on public.core_repositoryversion u3  (cost=0.00..16.30 rows=630 width=20) (actual time=0.003..0.004 rows=2 loops=1)
                                                  Output: u3.pulp_id, u3.number
                                ->  Hash  (cost=17.88..17.88 rows=210 width=16) (actual time=0.015..0.015 rows=2 loops=1)
                                      Output: u2.pulp_id
                                      Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                      ->  Seq Scan on public.core_repositoryversion u2  (cost=0.00..17.88 rows=210 width=16) (actual time=0.010..0.011 rows=2 loops=1)
                                            Output: u2.pulp_id
                                            Filter: (u2.number <= 1)
  ->  Index Scan using rpm_package_pkey on public.rpm_package  (cost=0.29..1.24 rows=1 width=1679) (actual time=0.006..0.006 rows=1 loops=69415)
        Output: rpm_package.content_ptr_id, rpm_package.name, rpm_package.epoch, rpm_package.version, rpm_package.release, rpm_package.arch, rpm_package.evr, rpm_package."pkgId", rpm_package.checksum_type, rpm_package.summary, rpm_package.description, rpm_package.url, rpm_package.changelogs, rpm_package.files, rpm_package.requires, rpm_package.provides, rpm_package.conflicts, rpm_package.obsoletes, rpm_package.suggests, rpm_package.enhances, rpm_package.recommends, rpm_package.supplements, rpm_package.location_base, rpm_package.location_href, rpm_package.rpm_buildhost, rpm_package.rpm_group, rpm_package.rpm_license, rpm_package.rpm_packager, rpm_package.rpm_sourcerpm, rpm_package.rpm_vendor, rpm_package.rpm_header_start, rpm_package.rpm_header_end, rpm_package.size_archive, rpm_package.size_installed, rpm_package.size_package, rpm_package.time_build, rpm_package.time_file, rpm_package.is_modular, rpm_package._pulp_domain_id
        Index Cond: (rpm_package.content_ptr_id = core_content.pulp_id)
Planning Time: 1.537 ms
Execution Time: 590.110 ms
```
At our last performance working group meeting, we acknowledged the need for refactoring the executed query in general (regardless of the model). I will use the `v.get_content(Package.objects)` call as the baseline. I am starting to optimize the query as suggested.
1. First attempt for improvement:
```
    def filter1(self, qs, repo_version):
        return qs.filter(pk__in=repo_version._content_relationships().values_list("content_id"))
```

By using this query, we omit a couple of joins and/or perform the **index-only scan** first when using `only("pk")` (https://www.postgresql.org/docs/current/indexes-index-only-scans.html). The speedup is significant (twice as fast).

```
In [5]: timeit.timeit("filter1(Package.objects, v).only('pk').count()", globals=globals(), number=100)
Out[5]: 4.320918750017881

In [6]: timeit.timeit("v.get_content(Package.objects).only('pk').count()", globals=globals(), number=100)
Out[6]: 10.077250079018995
```
```
In [6]: timeit.timeit("v.get_content(Package.objects).count()", globals=globals(), number=100)
Out[6]: 10.635628992982674

In [7]: timeit.timeit("filter1(Package.objects, v).count()", globals=globals(), number=100)
Out[7]: 4.526120177994017
```
```
In [6]: filter1(Package.objects, v).count()
(0.081) SELECT COUNT(*) AS "__count" FROM "rpm_package" WHERE "rpm_package"."content_ptr_id" IN (SELECT U0."content_id" FROM "core_repositorycontent" U0 INNER JOIN "core_repositoryversion" U2 ON (U0."version_added_id" = U2."pulp_id") LEFT OUTER JOIN "core_repositoryversion" U3 ON (U0."version_removed_id" = U3."pulp_id") WHERE (U0."repository_id" = '0189b6fd82f878a1a9a0b3dab55e8684'::uuid AND U2."number" <= 1 AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))); args=(UUID('0189b6fd-82f8-78a1-a9a0-b3dab55e8684'), Int4(1), Int4(1)); alias=default
Out[6]: 69222

In [7]: v.get_content(Package.objects).count()
(0.124) SELECT COUNT(*) AS "__count" FROM "rpm_package" INNER JOIN "core_content" ON ("rpm_package"."content_ptr_id" = "core_content"."pulp_id") INNER JOIN "core_repositorycontent" ON ("core_content"."pulp_id" = "core_repositorycontent"."content_id") WHERE "core_repositorycontent"."pulp_id" IN (SELECT U0."pulp_id" FROM "core_repositorycontent" U0 INNER JOIN "core_repositoryversion" U2 ON (U0."version_added_id" = U2."pulp_id") LEFT OUTER JOIN "core_repositoryversion" U3 ON (U0."version_removed_id" = U3."pulp_id") WHERE (U0."repository_id" = '0189b6fd82f878a1a9a0b3dab55e8684'::uuid AND U2."number" <= 1 AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))); args=(UUID('0189b6fd-82f8-78a1-a9a0-b3dab55e8684'), Int4(1), Int4(1)); alias=default
Out[7]: 69222
```

EXPLAIN:
```
In [36]: v.get_content(Package.objects).only('pk').explain(analyze=True)
```
```
(0.198) EXPLAIN (ANALYZE true) SELECT "rpm_package"."content_ptr_id"
FROM "rpm_package"
INNER JOIN "core_content" ON ("rpm_package"."content_ptr_id" = "core_content"."pulp_id")
INNER JOIN "core_repositorycontent" ON ("core_content"."pulp_id" = "core_repositorycontent"."content_id")
WHERE "core_repositorycontent"."pulp_id" IN (
  SELECT U0."pulp_id"
  FROM "core_repositorycontent" U0
  INNER JOIN "core_repositoryversion" U2 ON (U0."version_added_id" = U2."pulp_id")
  LEFT OUTER JOIN "core_repositoryversion" U3 ON (U0."version_removed_id" = U3."pulp_id")
  WHERE (U0."repository_id" = '0189b6fd82f878a1a9a0b3dab55e8684'::uuid AND U2."number" <= 1 AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))
);
args=(UUID('0189b6fd-82f8-78a1-a9a0-b3dab55e8684'), Int4(1), Int4(1));
alias=default
```
```
Nested Loop  (cost=4166.41..8591.28 rows=7790 width=16) (actual time=83.957..174.258 rows=69222 loops=1)
  ->  Hash Join  (cost=4166.11..6056.54 rows=7790 width=32) (actual time=83.921..111.440 rows=69415 loops=1)
        Hash Cond: (core_content.pulp_id = core_repositorycontent.content_id)
        ->  Seq Scan on core_content  (cost=0.00..1552.20 rows=69420 width=16) (actual time=0.016..8.547 rows=69420 loops=1)
        ->  Hash  (cost=4068.74..4068.74 rows=7790 width=16) (actual time=83.579..83.585 rows=69415 loops=1)
              Buckets: 131072 (originally 8192)  Batches: 2 (originally 1)  Memory Usage: 3073kB
              ->  Hash Semi Join  (cost=2179.71..4068.74 rows=7790 width=16) (actual time=48.852..75.283 rows=69415 loops=1)
                    Hash Cond: (core_repositorycontent.pulp_id = u0.pulp_id)
                    ->  Seq Scan on core_repositorycontent  (cost=0.00..1620.15 rows=69415 width=32) (actual time=0.195..3.727 rows=69415 loops=1)
                    ->  Hash  (cost=2082.33..2082.33 rows=7790 width=16) (actual time=48.383..48.387 rows=69415 loops=1)
                          Buckets: 131072 (originally 8192)  Batches: 2 (originally 1)  Memory Usage: 3073kB
                          ->  Hash Join  (cost=44.67..2082.33 rows=7790 width=16) (actual time=0.640..33.826 rows=69415 loops=1)
                                Hash Cond: (u0.version_added_id = u2.pulp_id)
                                ->  Hash Left Join  (cost=24.18..2000.08 rows=23370 width=32) (actual time=0.140..24.105 rows=69415 loops=1)
                                      Hash Cond: (u0.version_removed_id = u3.pulp_id)
                                      Filter: ((u3.number > 1) OR (u3.number IS NULL))
                                      ->  Seq Scan on core_repositorycontent u0  (cost=0.00..1793.69 rows=69415 width=48) (actual time=0.005..14.728 rows=69415 loops=1)
                                            Filter: (repository_id = '0189b6fd-82f8-78a1-a9a0-b3dab55e8684'::uuid)
                                      ->  Hash  (cost=16.30..16.30 rows=630 width=20) (actual time=0.021..0.022 rows=2 loops=1)
                                            Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                            ->  Seq Scan on core_repositoryversion u3  (cost=0.00..16.30 rows=630 width=20) (actual time=0.005..0.005 rows=2 loops=1)
                                ->  Hash  (cost=17.88..17.88 rows=210 width=16) (actual time=0.067..0.068 rows=2 loops=1)
                                      Buckets: 1024  Batches: 1  Memory Usage: 9kB
                                      ->  Seq Scan on core_repositoryversion u2  (cost=0.00..17.88 rows=210 width=16) (actual time=0.032..0.034 rows=2 loops=1)
                                            Filter: (number <= 1)
  ->  Index Only Scan using rpm_package_pkey on rpm_package  (cost=0.29..0.33 rows=1 width=16) (actual time=0.001..0.001 rows=1 loops=69415)
        Index Cond: (content_ptr_id = core_content.pulp_id)
        Heap Fetches: 0
Planning Time: 4.956 ms
Execution Time: 176.278 ms
```
VS
```
In [37]: filter1(Package.objects, v).only('pk').explain(analyze=True)
```
```
(0.073) EXPLAIN (ANALYZE true) SELECT "rpm_package"."content_ptr_id"
FROM "rpm_package"
WHERE "rpm_package"."content_ptr_id" IN (
  SELECT U0."content_id"
  FROM "core_repositorycontent" U0
  INNER JOIN "core_repositoryversion" U2 ON (U0."version_added_id" = U2."pulp_id")
  LEFT OUTER JOIN "core_repositoryversion" U3 ON (U0."version_removed_id" = U3."pulp_id")
  WHERE (U0."repository_id" = '0189b6fd82f878a1a9a0b3dab55e8684'::uuid AND U2."number" <= 1 AND NOT (U3."number" <= 1 AND U3."number" IS NOT NULL))
);
args=(UUID('0189b6fd-82f8-78a1-a9a0-b3dab55e8684'), Int4(1), Int4(1));
alias=default
```
```
Hash Semi Join  (cost=2180.00..4561.38 rows=7790 width=16) (actual time=47.981..70.716 rows=69222 loops=1)
  Hash Cond: (rpm_package.content_ptr_id = u0.content_id)
  ->  Index Only Scan using rpm_package_pkey on rpm_package  (cost=0.29..2112.31 rows=69601 width=16) (actual time=0.011..5.430 rows=69222 loops=1)
        Heap Fetches: 0
  ->  Hash  (cost=2082.33..2082.33 rows=7790 width=16) (actual time=47.963..47.965 rows=69415 loops=1)
        Buckets: 131072 (originally 8192)  Batches: 2 (originally 1)  Memory Usage: 3073kB
        ->  Hash Join  (cost=44.67..2082.33 rows=7790 width=16) (actual time=0.045..31.664 rows=69415 loops=1)
              Hash Cond: (u0.version_added_id = u2.pulp_id)
              ->  Hash Left Join  (cost=24.18..2000.08 rows=23370 width=32) (actual time=0.019..20.769 rows=69415 loops=1)
                    Hash Cond: (u0.version_removed_id = u3.pulp_id)
                    Filter: ((u3.number > 1) OR (u3.number IS NULL))
                    ->  Seq Scan on core_repositorycontent u0  (cost=0.00..1793.69 rows=69415 width=48) (actual time=0.004..9.634 rows=69415 loops=1)
                          Filter: (repository_id = '0189b6fd-82f8-78a1-a9a0-b3dab55e8684'::uuid)
                    ->  Hash  (cost=16.30..16.30 rows=630 width=20) (actual time=0.004..0.005 rows=2 loops=1)
                          Buckets: 1024  Batches: 1  Memory Usage: 9kB
                          ->  Seq Scan on core_repositoryversion u3  (cost=0.00..16.30 rows=630 width=20) (actual time=0.003..0.003 rows=2 loops=1)
              ->  Hash  (cost=17.88..17.88 rows=210 width=16) (actual time=0.012..0.012 rows=2 loops=1)
                    Buckets: 1024  Batches: 1  Memory Usage: 9kB
                    ->  Seq Scan on core_repositoryversion u2  (cost=0.00..17.88 rows=210 width=16) (actual time=0.009..0.010 rows=2 loops=1)
                          Filter: (number <= 1)
Planning Time: 0.477 ms
Execution Time: 72.341 ms
```

To this moment, I could find only one disadvantage. If we want to perform the index-only scan, we should force plugin-writers to use `only("pk")` which is not convenient.

I am now continuing with the changes to the DB model, as suggested.

(credits to @newswangerd for sharing the filter)
Side note: the above query seems to be robust inside the environment where the same content is shared across multiple repositories:
```
>>> filter1(None, Package.objects, v).count()
268653
>>> import timeit
>>> timeit.timeit("filter1(None, Package.objects, v).count()", globals=globals(), number=10)
24.24022313579917
>>> timeit.timeit("v.get_content(Package.objects).count()", globals=globals(), number=10)
64.38561939820647
```
I have measured the time spent querying the database for the attached pulp_ansible filter (https://github.com/pulp/pulp_ansible/blob/fb6e12d1546c52ef7c6b892a2e8743d477b1e3e5/pulp_ansible/app/utils.py#L5) in the very same environment. The performance is even better than for the `filter1` query in 3 out of 4 trials. 
```
>>> timeit.timeit("filter1(Package.objects, v).count()", globals=globals(), number=10)
27.624995853751898
>>> timeit.timeit("filter_content_for_repo_version(Package.objects, v).count()", globals=globals(), number=10)
18.918790597468615 --------------> BEST
>>> timeit.timeit("v.get_content(Package.objects).count()", globals=globals(), number=10)
75.1577151119709
>>> 
>>> timeit.timeit("filter1(Package.objects, v).only('pk').count()", globals=globals(), number=10)
24.04796067252755
>>> timeit.timeit("filter_content_for_repo_version(Package.objects, v).only('pk').count()", globals=globals(), number=10)
16.870408695191145 --------------> BEST
>>> timeit.timeit("v.get_content(Package.objects).only('pk').count()", globals=globals(), number=10)
71.92632512748241
>>> 
>>> timeit.timeit("len(filter1(Package.objects, v))", globals=globals(), number=1)
75.4389706030488
>>> timeit.timeit("len(filter_content_for_repo_version(Package.objects, v))", globals=globals(), number=1)
66.47706608846784 --------------> BEST
>>> timeit.timeit("len(v.get_content(Package.objects))", globals=globals(), number=1)
81.08162821829319
>>> 
>>> timeit.timeit("len(filter1(Package.objects, v).only('pk'))", globals=globals(), number=10)
91.59041200578213 --------------> BEST
>>> timeit.timeit("len(filter_content_for_repo_version(Package.objects, v).only('pk'))", globals=globals(), number=10)
98.93004615977407
>>> timeit.timeit("len(v.get_content(Package.objects).only('pk'))", globals=globals(), number=10)
152.07668563723564
```

@daviddavis , can you give us some insights about the observed "mixed results" of this query? In which case was it slow? Is this the reason for the concerns:
```
>>> timeit.timeit("filter1(Package.objects, v)[0:100]", globals=globals(), number=100)
0.08750724419951439
>>> timeit.timeit("filter_content_for_repo_version(Package.objects, v)[0:100]", globals=globals(), number=100)
0.18397120386362076
>>> timeit.timeit("v.get_content(Package.objects)[0:100]", globals=globals(), number=100)
0.08487076684832573
```


Resolving patch:
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -617,7 +617,7 @@ def get_content(self, content_qs=None):
         if content_qs is None:
             content_qs = Content.objects
 
-        return content_qs.filter(version_memberships__in=self._content_relationships())
+        return content_qs.filter(pk__in=self._content_relationships().values_list("content_id"))
 
     @property
     def content(self):


Test patch: