Repository: pulp/pulpcore

Issue:
Add ability to turn any Pulp instance into a replica
**Version**
Python PYPI installation
	PulpCore 3.32.0
	CertGuard 1.6.5
	file 1.14.4
	python 3.10.0
	rpm 3.22.3


**Describe the bug**
Replication task fails to maintain the publication attribute in a existing RPM distribution artifact. Instead it tries and fail to update the repository attribute in distribution artifact.

In a replication task group, I am getting a error for "general_update" task of a RPM distribution artifact
https://github.com/pulp/pulpcore/blob/main/pulpcore/app/tasks/base.py

ErrorDetail(string="Only one of the attributes 'repository' and 'publication' may used simultaneously.", code='invalid')


**To Reproduce**
In a upstream pulp server(Pulp1 server), create a repository with AutoPublish as false and it's distribution is based on Publications and repository field value is null.
In a downstream pulp server(Pulp2 server), do the same like create a repository with AutoPublish as false and it's distribution is based on Publications and repository field value is null.
Create a upstream object in Pulp2 server to replicate from upstream server (Pulp1 server)
Start the replication task in downstream pulp server(Pulp2 server) to replicate content from upstream server (Pulp1 server).
Replication task will fail to update the distribution artifact of the repository.


**Expected behavior**
Why Pulp replication doesn't replicate the publication from upstream pulp server when RPM repository distribution is based on publication. 


Discussion:
This may still be a bug, but for the workflow, one is not supposed to create the repositories downstream by hand. Replicate is meant to handle _all_ of that.
What does mean 'distribution is based on publication'? Does it mean that you have manually triggered publish task for the repo and then updated the distribution wth the created href of publication?

@mdellweg I agree that the feature as currently designed requires the user to only use the replication API to create repositories/publications/distributions in the system. However, it would be better if you could use the replicate command to turn an existing Pulp installation into a replica with a single replicate task. The replicate task would need to be able to handle situations such as the one described in this issue. 

This is a request for a feature and not a bug. 
Hi,

Based on my knowledge I see this as a bug not as a feature request. Apologize, as I haven't provided an example. I'm doing this now.

In the Upstream server, we sync/add RPM packages to the repository and then publish the repository so distribution object is based on publication not by repository version.

` $ pulp rpm distribution list --name "rh-el7-extras"
{
  "pulp_href": "/pulp/api/v3/distribution/rpm/rpm/XXXXXXXXXXX",
  "pulp_created": "DATE",
  "base_path": "rh/el7/extras",
  "base_url": "Upstream Pulp Content Server Distribution URL",
  "content_guard": null,
  "hidden": false,
  "pulp_labels": {},
  "name": "rh-el7-extras",
  "repository": null,
  "publication: "/pulp/api/v3/publications/rpm/rpm/XXXXXXXXXXXX"
}`


Now I initiate a pulp replication task in the downstream server. The distribution object in downstream server is created based on repository attribute and not based on publication attribute.
`$ pulp rpm distribution list --name "rh-el7-extras"
{
  "pulp_href": "/pulp/api/v3/distribution/rpm/rpm/XXXXXXXXXXX",
  "pulp_created": "DATE",
  "base_path": "rh/el7/extras",
  "base_url": "Downstream Pulp Content Server Distribution URL",
  "content_guard": null,
  "hidden": false,
  "pulp_labels": {},
  "name": "rh-el7-extras",
  "repository": "/pulp/api/v3/repositories/rpm/rpm/XXXXXXXXXXXX",
  "publication: null
}`
I see the confusion here, however, do you have any actual problem?

Rpm replicator task triggers MIRROR_COMPLETE sync, which automatically re-creates bit-by-bit publication. Then when content is served it makes sure to find publication serving the latest version https://github.com/pulp/pulpcore/blob/main/pulpcore/content/handler.py#L598
Even though our Handler code seems to handle the serving of content, this not really intuitive https://github.com/pulp/pulpcore/blob/main/pulpcore/app/replica.py#L176
Maybe replica code should take into account publications and set them for the publication based distributions? @dkliban you have any insights here?
> I see the confusion here, however, do you have any actual problem?
> 
> Rpm replicator task triggers MIRROR_COMPLETE sync, which automatically re-creates bit-by-bit publication. Then when content is served it makes sure to find publication serving the latest version https://github.com/pulp/pulpcore/blob/main/pulpcore/content/handler.py#L598

Yes, I am facing a problem. In our setup, the role of upstream and downstream is swapped. During the maintenance window of the upstream server, one of the existing downstream servers will be promoted as upstream server and start supplying comtent to rest of the downstream servers.

At this time in the downsteam server (whose role is due to change to upstream), I have destroy all distribution artifacts and recreate them based on publication attribute and keep repository attribute as null. 
To clarify, based on recent pulpcore mtg to discuss: The ask here, is for the replication code to notice this problem, and for the replication to "win" (ie reset/'fix' the downstream to be 'replicable') instead of failing.  

Should be a straightforward fix, and easy to have test cases for.

Resolving patch:
diff --git a/pulp_file/pytest_plugin.py b/pulp_file/pytest_plugin.py
--- a/pulp_file/pytest_plugin.py
+++ b/pulp_file/pytest_plugin.py
@@ -308,7 +308,7 @@ def _file_remote_client_cert_req_factory(*, manifest_path, policy, **kwargs):
 
 
 @pytest.fixture(scope="class")
-def file_repository_factory(file_repository_api_client, gen_object_with_cleanup):
+def file_repository_factory(file_bindings, gen_object_with_cleanup):
     """A factory to generate a File Repository with auto-deletion after the test run."""
 
     def _file_repository_factory(pulp_domain=None, **body):
@@ -316,11 +316,26 @@ def _file_repository_factory(pulp_domain=None, **body):
         if pulp_domain:
             kwargs["pulp_domain"] = pulp_domain
         body.setdefault("name", str(uuid.uuid4()))
-        return gen_object_with_cleanup(file_repository_api_client, body, **kwargs)
+        return gen_object_with_cleanup(file_bindings.RepositoriesFileApi, body, **kwargs)
 
     return _file_repository_factory
 
 
+@pytest.fixture(scope="class")
+def file_publication_factory(file_bindings, gen_object_with_cleanup):
+    """A factory to generate a File Publication with auto-deletion after the test run."""
+
+    def _file_publication_factory(**kwargs):
+        extra_args = {}
+        if pulp_domain := kwargs.pop("pulp_domain", None):
+            extra_args["pulp_domain"] = pulp_domain
+        # XOR check on repository and repository_version
+        assert bool("repository" in kwargs) ^ bool("repository_version" in kwargs)
+        return gen_object_with_cleanup(file_bindings.PublicationsFileApi, kwargs, **extra_args)
+
+    return _file_publication_factory
+
+
 @pytest.fixture
 def gen_bad_response_fixture_server(gen_threaded_aiohttp_server):
     """
diff --git a/pulpcore/app/replica.py b/pulpcore/app/replica.py
--- a/pulpcore/app/replica.py
+++ b/pulpcore/app/replica.py
@@ -1,18 +1,36 @@
 from django.conf import settings
 from django.db.models import Model
+import logging
 
 from pulp_glue.common.context import PulpContext
 from pulpcore.tasking.tasks import dispatch
-from pulpcore.app.tasks.base import general_update, general_create, general_delete
+from pulpcore.app.tasks.base import (
+    general_update,
+    general_create,
+    general_multi_delete,
+)
 from pulpcore.plugin.util import get_url, get_domain
 
+_logger = logging.getLogger(__name__)
 
-class ReplicaContext(PulpContext):
-    def prompt(self, *args, **kwargs):
-        pass
 
-    def echo(self, *args, **kwargs):
-        pass
+class ReplicaContext(PulpContext):
+    def __init__(self, **kwargs):
+        super().__init__(**kwargs)
+        self.out_buf = ""
+        self.err_buf = ""
+
+    def echo(self, message: str, nl: bool = True, err: bool = False) -> None:
+        if err:
+            self.err_buf += message
+            if nl:
+                _logger.warn("{}", self.err_buf)
+                self.err_buf = ""
+        else:
+            self.out_buf += message
+            if nl:
+                _logger.info("{}", self.out_buf)
+                self.out_buf = ""
 
 
 class Replicator:
@@ -39,6 +57,7 @@ def __init__(self, pulp_ctx, task_group, tls_settings):
         self.tls_settings = tls_settings
         self.domain = get_domain()
         uri = "/api/v3/distributions/"
+        # TODO check and compare this to distribution locking on the distribution viewset.
         if settings.DOMAIN_ENABLED:
             uri = f"/{self.domain.name}{uri}"
         self.distros_uri = uri
@@ -133,49 +152,47 @@ def create_or_update_repository(self, remote):
             repository.save()
         return repository
 
+    def distribution_data(self, repository, upstream_distribution):
+        """
+        Return the fields that need to be updated/cleared on distributions for idempotence.
+        """
+        return {
+            "repository": get_url(repository),
+            "publication": None,
+            "base_path": upstream_distribution["base_path"],
+        }
+
     def create_or_update_distribution(self, repository, upstream_distribution):
+        distribution_data = self.distribution_data(repository, upstream_distribution)
         try:
             distro = self.distribution_model_cls.objects.get(
                 name=upstream_distribution["name"], pulp_domain=self.domain
             )
             # Check that the distribution has the right repository associated
-            needs_update = self.needs_update(
-                {
-                    "repository": get_url(repository),
-                    "base_path": upstream_distribution["base_path"],
-                },
-                distro,
-            )
+            needs_update = self.needs_update(distribution_data, distro)
             if needs_update:
                 # Update the distribution
                 dispatch(
                     general_update,
                     task_group=self.task_group,
+                    shared_resources=[repository],
                     exclusive_resources=[self.distros_uri],
                     args=(distro.pk, self.app_label, self.distribution_serializer_name),
                     kwargs={
-                        "data": {
-                            "name": upstream_distribution["name"],
-                            "base_path": upstream_distribution["base_path"],
-                            "repository": get_url(repository),
-                        },
+                        "data": distribution_data,
                         "partial": True,
                     },
                 )
         except self.distribution_model_cls.DoesNotExist:
             # Dispatch a task to create the distribution
+            distribution_data["name"] = upstream_distribution["name"]
             dispatch(
                 general_create,
                 task_group=self.task_group,
+                shared_resources=[repository],
                 exclusive_resources=[self.distros_uri],
                 args=(self.app_label, self.distribution_serializer_name),
-                kwargs={
-                    "data": {
-                        "name": upstream_distribution["name"],
-                        "base_path": upstream_distribution["base_path"],
-                        "repository": get_url(repository),
-                    }
-                },
+                kwargs={"data": distribution_data},
             )
 
     def sync_params(self, repository, remote):
@@ -193,35 +210,42 @@ def sync(self, repository, remote):
 
     def remove_missing(self, names):
         # Remove all distributions with names not present in the list of names
-        distros_to_delete = self.distribution_model_cls.objects.filter(
-            pulp_domain=self.domain
-        ).exclude(name__in=names)
-        for distro in distros_to_delete:
+        # Perform this in an extra task, because we hold a big lock here.
+        distribution_ids = [
+            (distribution.pk, self.app_label, self.distribution_serializer_name)
+            for distribution in self.distribution_model_cls.objects.filter(
+                pulp_domain=self.domain
+            ).exclude(name__in=names)
+        ]
+        if distribution_ids:
             dispatch(
-                general_delete,
+                general_multi_delete,
                 task_group=self.task_group,
                 exclusive_resources=[self.distros_uri],
-                args=(distro.pk, self.app_label, self.distribution_serializer_name),
+                args=(distribution_ids,),
             )
 
         # Remove all the repositories and remotes of the missing distributions
-        repos_to_delete = self.repository_model_cls.objects.filter(pulp_domain=self.domain).exclude(
-            name__in=names
-        )
-        for repo in repos_to_delete:
-            dispatch(
-                general_delete,
-                task_group=self.task_group,
-                exclusive_resources=[repo],
-                args=(repo.pk, self.app_label, self.repository_serializer_name),
+        repositories = list(
+            self.repository_model_cls.objects.filter(pulp_domain=self.domain).exclude(
+                name__in=names
             )
-        remotes_to_delete = self.remote_model_cls.objects.filter(pulp_domain=self.domain).exclude(
-            name__in=names
         )
-        for remote in remotes_to_delete:
+        repository_ids = [
+            (repo.pk, self.app_label, self.repository_serializer_name) for repo in repositories
+        ]
+
+        remotes = list(
+            self.remote_model_cls.objects.filter(pulp_domain=self.domain).exclude(name__in=names)
+        )
+        remote_ids = [
+            (remote.pk, self.app_label, self.remote_serializer_name) for remote in remotes
+        ]
+
+        if repository_ids or remote_ids:
             dispatch(
-                general_delete,
+                general_multi_delete,
                 task_group=self.task_group,
-                exclusive_resources=[remote],
-                args=(remote.pk, self.app_label, self.remote_serializer_name),
+                exclusive_resources=repositories + remotes,
+                args=(repository_ids + remote_ids,),
             )
diff --git a/pulpcore/app/tasks/replica.py b/pulpcore/app/tasks/replica.py
--- a/pulpcore/app/tasks/replica.py
+++ b/pulpcore/app/tasks/replica.py
@@ -7,7 +7,6 @@
 from pulpcore.app.apps import pulp_plugin_configs
 from pulpcore.app.models import UpstreamPulp, TaskGroup
 from pulpcore.app.replica import ReplicaContext
-from pulpcore.app.util import get_domain
 
 from pulp_glue.common import __version__ as pulp_glue_version
 
@@ -24,7 +23,6 @@ def user_agent():
 
 
 def replicate_distributions(server_pk):
-    domain = get_domain()
     server = UpstreamPulp.objects.get(pk=server_pk)
 
     # Write out temporary files related to SSL
@@ -80,18 +78,9 @@ def replicate_distributions(server_pk):
             # Create remote
             remote = replicator.create_or_update_remote(upstream_distribution=distro)
             if not remote:
-                # The upstream distribution is not serving any content, cleanup an existing local
-                # distribution
-                try:
-                    local_distro = replicator.distribution_model.objects.get(
-                        name=distro["name"], pulp_domain=domain
-                    )
-                    local_distro.repository = None
-                    local_distro.publication = None
-                    local_distro.save()
-                    continue
-                except replicator.distribution_model.DoesNotExist:
-                    continue
+                # The upstream distribution is not serving any content,
+                # let if fall throug the cracks and be cleanup below.
+                continue
             # Check if there is already a repository
             repository = replicator.create_or_update_repository(remote=remote)
 


Test patch:
diff --git a/pulpcore/tests/functional/__init__.py b/pulpcore/tests/functional/__init__.py
--- a/pulpcore/tests/functional/__init__.py
+++ b/pulpcore/tests/functional/__init__.py
@@ -955,6 +955,44 @@ def random_artifact(random_artifact_factory):
     return random_artifact_factory()
 
 
+@pytest.fixture()
+def domain_factory(pulpcore_bindings, pulp_settings, gen_object_with_cleanup):
+    def _domain_factory():
+        if not pulp_settings.DOMAIN_ENABLED:
+            pytest.skip("Domains not enabled")
+        keys = dict()
+        keys["pulpcore.app.models.storage.FileSystem"] = ["MEDIA_ROOT"]
+        keys["storages.backends.s3boto3.S3Boto3Storage"] = [
+            "AWS_ACCESS_KEY_ID",
+            "AWS_SECRET_ACCESS_KEY",
+            "AWS_S3_ENDPOINT_URL",
+            "AWS_S3_ADDRESSING_STYLE",
+            "AWS_S3_SIGNATURE_VERSION",
+            "AWS_S3_REGION_NAME",
+            "AWS_STORAGE_BUCKET_NAME",
+        ]
+        keys["storages.backends.azure_storage.AzureStorage"] = [
+            "AZURE_ACCOUNT_NAME",
+            "AZURE_CONTAINER",
+            "AZURE_ACCOUNT_KEY",
+            "AZURE_URL_EXPIRATION_SECS",
+            "AZURE_OVERWRITE_FILES",
+            "AZURE_LOCATION",
+            "AZURE_CONNECTION_STRING",
+        ]
+        settings = dict()
+        for key in keys[pulp_settings.DEFAULT_FILE_STORAGE]:
+            settings[key] = getattr(pulp_settings, key, None)
+        body = {
+            "name": str(uuid.uuid4()),
+            "storage_class": pulp_settings.DEFAULT_FILE_STORAGE,
+            "storage_settings": settings,
+        }
+        return gen_object_with_cleanup(pulpcore_bindings.DomainsApi, body)
+
+    return _domain_factory
+
+
 # Random other fixtures
 
 
@@ -1263,41 +1301,3 @@ def _wget_recursive_download_on_host(url, destination):
         )
 
     return _wget_recursive_download_on_host
-
-
-@pytest.fixture()
-def domain_factory(domains_api_client, pulp_settings, gen_object_with_cleanup):
-    def _domain_factory():
-        if not pulp_settings.DOMAIN_ENABLED:
-            pytest.skip("Domains not enabled")
-        keys = dict()
-        keys["pulpcore.app.models.storage.FileSystem"] = ["MEDIA_ROOT"]
-        keys["storages.backends.s3boto3.S3Boto3Storage"] = [
-            "AWS_ACCESS_KEY_ID",
-            "AWS_SECRET_ACCESS_KEY",
-            "AWS_S3_ENDPOINT_URL",
-            "AWS_S3_ADDRESSING_STYLE",
-            "AWS_S3_SIGNATURE_VERSION",
-            "AWS_S3_REGION_NAME",
-            "AWS_STORAGE_BUCKET_NAME",
-        ]
-        keys["storages.backends.azure_storage.AzureStorage"] = [
-            "AZURE_ACCOUNT_NAME",
-            "AZURE_CONTAINER",
-            "AZURE_ACCOUNT_KEY",
-            "AZURE_URL_EXPIRATION_SECS",
-            "AZURE_OVERWRITE_FILES",
-            "AZURE_LOCATION",
-            "AZURE_CONNECTION_STRING",
-        ]
-        settings = dict()
-        for key in keys[pulp_settings.DEFAULT_FILE_STORAGE]:
-            settings[key] = getattr(pulp_settings, key, None)
-        body = {
-            "name": str(uuid.uuid4()),
-            "storage_class": pulp_settings.DEFAULT_FILE_STORAGE,
-            "storage_settings": settings,
-        }
-        return gen_object_with_cleanup(domains_api_client, body)
-
-    return _domain_factory
diff --git a/pulpcore/tests/functional/api/test_replication.py b/pulpcore/tests/functional/api/test_replication.py
--- a/pulpcore/tests/functional/api/test_replication.py
+++ b/pulpcore/tests/functional/api/test_replication.py
@@ -44,6 +44,109 @@ def test_replication(
         assert task.state == "completed"
 
 
+@pytest.mark.parallel
+def test_replication_idempotence(
+    domain_factory,
+    bindings_cfg,
+    pulpcore_bindings,
+    file_bindings,
+    monitor_task,
+    monitor_task_group,
+    pulp_settings,
+    add_to_cleanup,
+    gen_object_with_cleanup,
+    file_distribution_factory,
+    file_publication_factory,
+    file_repository_factory,
+    tmp_path,
+):
+    # This test assures that an Upstream Pulp can be created in a non-default domain and that this
+    # Upstream Pulp configuration can be used to execute the replicate task.
+
+    # Create a domain to replicate from
+    source_domain = domain_factory()
+
+    # Add stuff to it
+    repository = file_repository_factory(pulp_domain=source_domain.name)
+    file_path = tmp_path / "file.txt"
+    file_path.write_text("DEADBEEF")
+    monitor_task(
+        file_bindings.ContentFilesApi.create(
+            file=file_path,
+            relative_path="file.txt",
+            repository=repository.pulp_href,
+            pulp_domain=source_domain.name,
+        ).task
+    )
+    publication = file_publication_factory(
+        pulp_domain=source_domain.name, repository=repository.pulp_href
+    )
+    file_distribution_factory(pulp_domain=source_domain.name, publication=publication.pulp_href)
+
+    # Create a domain as replica
+    replica_domain = domain_factory()
+
+    # Create an Upstream Pulp in the non-default domain
+    upstream_pulp_body = {
+        "name": str(uuid.uuid4()),
+        "base_url": bindings_cfg.host,
+        "api_root": pulp_settings.API_ROOT,
+        "domain": source_domain.name,
+        "username": bindings_cfg.username,
+        "password": bindings_cfg.password,
+    }
+    upstream_pulp = gen_object_with_cleanup(
+        pulpcore_bindings.UpstreamPulpsApi, upstream_pulp_body, pulp_domain=replica_domain.name
+    )
+    # Run the replicate task and assert that all tasks successfully complete.
+    response = pulpcore_bindings.UpstreamPulpsApi.replicate(upstream_pulp.pulp_href)
+    monitor_task_group(response.task_group)
+
+    for api_client in (
+        file_bindings.DistributionsFileApi,
+        file_bindings.RemotesFileApi,
+        file_bindings.RepositoriesFileApi,
+    ):
+        result = api_client.list(pulp_domain=replica_domain.name)
+        for item in result.results:
+            add_to_cleanup(api_client, item)
+
+    for api_client in (
+        file_bindings.DistributionsFileApi,
+        file_bindings.RemotesFileApi,
+        file_bindings.RepositoriesFileApi,
+        file_bindings.ContentFilesApi,
+    ):
+        result = api_client.list(pulp_domain=replica_domain.name)
+        assert result.count == 1
+
+    # Now replicate backwards
+
+    upstream_pulp_body = {
+        "name": str(uuid.uuid4()),
+        "base_url": bindings_cfg.host,
+        "api_root": pulp_settings.API_ROOT,
+        "domain": replica_domain.name,
+        "username": bindings_cfg.username,
+        "password": bindings_cfg.password,
+    }
+    upstream_pulp = gen_object_with_cleanup(
+        pulpcore_bindings.UpstreamPulpsApi, upstream_pulp_body, pulp_domain=source_domain.name
+    )
+    # Run the replicate task and assert that all tasks successfully complete.
+    response = pulpcore_bindings.UpstreamPulpsApi.replicate(upstream_pulp.pulp_href)
+    monitor_task_group(response.task_group)
+
+    for api_client in (
+        file_bindings.DistributionsFileApi,
+        file_bindings.RemotesFileApi,
+        file_bindings.RepositoriesFileApi,
+    ):
+        result = api_client.list(pulp_domain=replica_domain.name)
+        for item in result.results:
+            add_to_cleanup(api_client, item)
+
+
 @pytest.mark.parallel
 def test_replication_with_wrong_ca_cert(
     domain_factory,