internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: pulp/pulpcore

Issue:
[PR #4182/04738d77 backport][3.23] Made reclaim space task more tolerant in the face of artifact shared …
**This is a backport of PR #4182 as merged into main (04738d77a42d93f386294f0857230eed1983ba61).**

…between two content units.

closes #3610
[PR #4182/04738d77 backport][3.23] Made reclaim space task more tolerant in the face of artifact shared …
**This is a backport of PR #4182 as merged into main (04738d77a42d93f386294f0857230eed1983ba61).**

…between two content units.

closes #3610


Discussion:

Resolving patch:
diff --git a/containers/images/pulp-api/container-assets/wait_on_postgres.py b/containers/images/pulp-api/container-assets/wait_on_postgres.py
new file mode 100755
--- /dev/null
+++ b/containers/images/pulp-api/container-assets/wait_on_postgres.py
@@ -0,0 +1,29 @@
+#!/usr/bin/env python
+
+import sys
+import socket
+import time
+import os
+
+if __name__ == '__main__':
+
+    postgres_is_alive = False
+    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    tries = 0
+    print("Waiting on postgresql to start...")
+    while not postgres_is_alive and tries < 100:
+        tries += 1
+        try:
+            print("Checking postgres host %s" % os.environ['POSTGRES_SERVICE_HOST'])
+            s.connect((os.environ['POSTGRES_SERVICE_HOST'], 5432))
+        except socket.error:
+            time.sleep(3)
+        else:
+            postgres_is_alive = True
+
+    if postgres_is_alive:
+        print("Postgres started!")
+        sys.exit(0)
+    else:
+        print("Unable to reach postgres on port 5432")
+        sys.exit(1)


Test patch:
Repository: pulp/pulpcore

Issue:
Update CI files for branch 3.21
[noissue]


Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/fields.py b/pulpcore/app/serializers/fields.py
--- a/pulpcore/app/serializers/fields.py
+++ b/pulpcore/app/serializers/fields.py
@@ -200,8 +200,12 @@ class BaseURLField(serializers.CharField):
     """
 
     def to_representation(self, value):
-        if settings.CONTENT_HOST:
-            host = settings.CONTENT_HOST
-        else:
-            host = self.context['request'].get_host()
-        return ''.join([host, settings.CONTENT_PATH_PREFIX, value])
+        base_path = value
+        host = settings.CONTENT_HOST or ''
+        prefix = settings.CONTENT_PATH_PREFIX or ''
+        return '/'.join(
+            (
+                host.strip('/'),
+                prefix.strip('/'),
+                base_path.lstrip('/')
+            ))


Test patch:
Repository: pulp/pulpcore

Issue:
3.23 - Faster exports option



Discussion:

Resolving patch:
diff --git a/pulpcore/app/migrations/0002_distribution_remote.py b/pulpcore/app/migrations/0002_distribution_remote.py
--- a/pulpcore/app/migrations/0002_distribution_remote.py
+++ b/pulpcore/app/migrations/0002_distribution_remote.py
@@ -7,13 +7,13 @@
 class Migration(migrations.Migration):
 
     dependencies = [
-        ('pulp_app', '0001_initial'),
+        ('core', '0001_initial'),
     ]
 
     operations = [
         migrations.AddField(
             model_name='distribution',
             name='remote',
-            field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='distributions', to='pulp_app.Remote'),
+            field=models.ForeignKey(null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='distributions', to='core.Remote'),
         ),
     ]


Test patch:
Repository: pulp/pulpcore

Issue:
[PR #4609/4c4d6aa0 backport][3.28] Works around a sync-time performance regression on PG12
**This is a backport of PR #4609 as merged into main (4c4d6aa0745b4938e2aeee8ffc28a0b0392cc9ef).**

closes #4591

We still don't really "know":

* Why this only seems to happen on PG12
* Why the original query change (https://github.com/pulp/pulpcore/pull/4275/files) triggered this
* Why it only seems to happen on capsule syncs (this is most mysterious)


Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/repository.py b/pulpcore/app/serializers/repository.py
--- a/pulpcore/app/serializers/repository.py
+++ b/pulpcore/app/serializers/repository.py
@@ -267,10 +267,14 @@ def get_content_summary(self, obj):
         The summary of contained content.
 
         Returns:
-            dict:    {'added': {<_type>: {'count': <count>, 'href': <href>},
-                      'removed': {<_type>: {'count': <count>, 'href': <href>},
-                      'present': {<_type>: {'count': <count>, 'href': <href>},
-                     }
+            dict: The dictionary has the following format.::
+
+                {
+                    'added': {<_type>: {'count': <count>, 'href': <href>},
+                    'removed': {<_type>: {'count': <count>, 'href': <href>},
+                    'present': {<_type>: {'count': <count>, 'href': <href>},
+                }
+
         """
         to_return = {'added': {}, 'removed': {}, 'present': {}}
         for count_detail in obj.counts.all():


Test patch:
Repository: pulp/pulpcore

Issue:
Burst mode for pulpcore-worker
When upgrading, it is sometimes not safe to bring "old" task unfinished to the new version. Though we have a backwards compatibility claim, some admins may decide they want to drain the task queue anyway.
Providing a "--burst" option to the pulpcore worker that will tell it to exit instead of going to sleep will help there.
Also this may be useful in case you have a system stuck on a bunch of sync/import/export tasks to add a few workers that will disconnect themselves once the congestion is over.


Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/repository.py b/pulpcore/app/serializers/repository.py
--- a/pulpcore/app/serializers/repository.py
+++ b/pulpcore/app/serializers/repository.py
@@ -136,7 +136,8 @@ class RepositorySyncURLSerializer(serializers.Serializer):
     mirror = fields.BooleanField(
         required=False,
         default=False,
-        help_text=_('The synchronization mode, True for "mirror" and False for "additive" mode.')
+        help_text=_('If ``True``, synchronization will remove all content that is not present in '
+                    'the remote repository. If ``False``, sync will be additive only.')
     )
 
 


Test patch:
Repository: pulp/pulpcore

Issue:
Cleanup old publications - general cleanup
**Is your feature request related to a problem? Please describe.**
We do a lot of different new repos and publications - and also delete them after certain time.
Now i see, the publications resist.
Means, in normal handling we have hundreds of publications.
But, we don't know where they do belong to, becuase there is no name tag. 

**Describe the solution you'd like**
May be there is a possibility to remove old publications with no belonging or link to a repository or distribution. 

**Describe alternatives you've considered**
No idea, i just think about an clenaup job.




Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/publication.py b/pulpcore/app/serializers/publication.py
--- a/pulpcore/app/serializers/publication.py
+++ b/pulpcore/app/serializers/publication.py
@@ -92,6 +92,22 @@ class Meta:
 
 
 class BaseDistributionSerializer(MasterModelSerializer):
+    """
+    The Serializer for the BaseDistribution model.
+
+    The serializer deliberately omits the "remote" field, which is used for
+    pull-through caching only. Plugins implementing pull-through caching will
+    have to add the field in their derived serializer class like this::
+
+      remote = DetailRelatedField(
+          required=False,
+          help_text=_('Remote that can be used to fetch content when using pull-through caching.'),
+          queryset=models.Remote.objects.all(),
+          allow_null=True
+      )
+
+    """
+
     _href = DetailIdentityField()
     base_path = serializers.CharField(
         help_text=_('The base (relative) path component of the published url. Avoid paths that \
@@ -123,12 +139,6 @@ class BaseDistributionSerializer(MasterModelSerializer):
             )),
             UniqueValidator(queryset=models.BaseDistribution.objects.all())]
     )
-    remote = DetailRelatedField(
-        required=False,
-        help_text=_('Remote that can be used to fetch content when using pull-through caching.'),
-        queryset=models.Remote.objects.all(),
-        allow_null=True
-    )
 
     class Meta:
         abstract = True
@@ -138,7 +148,6 @@ class Meta:
             'base_url',
             'content_guard',
             'name',
-            'remote',
         )
 
     def _validate_path_overlap(self, path):


Test patch:
Repository: pulp/pulpcore

Issue:
Update CI files for branch 3.39
[noissue]


Discussion:

Resolving patch:
diff --git a/pulpcore/content/handler.py b/pulpcore/content/handler.py
--- a/pulpcore/content/handler.py
+++ b/pulpcore/content/handler.py
@@ -50,8 +50,8 @@ class Handler:
 
     3. If the Distribution has a `publication` serve that Publication's `PublishedArtifacts`,
        `PublishedMetadata` by the remaining `relative path`. If still unserved and if `pass_through`
-        is set, the associated `repository_version` will have its `ContentArtifacts` served by
-        `relative_path` also. This will serve the associated `Artifact`.
+       is set, the associated `repository_version` will have its `ContentArtifacts` served by
+       `relative_path` also. This will serve the associated `Artifact`.
 
     4. If still unmatched, and the Distribution has a `repository` attribute set, find it's latest
        `repository_version`. If the Distribution has a `repository_version` attribute set, use that.


Test patch:
Repository: pulp/pulpcore

Issue:
Fix division-by-zero on import
closes #4777

TODO: still working on the tests, it's a bit difficult to piece together how the fixtures interact...


Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/publication.py b/pulpcore/app/serializers/publication.py
--- a/pulpcore/app/serializers/publication.py
+++ b/pulpcore/app/serializers/publication.py
@@ -28,12 +28,13 @@ class PublicationSerializer(MasterModelSerializer):
         queryset=models.RepositoryVersion.objects.all(),
         required=False,
     )
-    repository = serializers.HyperlinkedRelatedField(
+    repository = RelatedField(
         help_text=_('A URI of the repository to be published.'),
         required=False,
         label=_('Repository'),
         queryset=models.Repository.objects.all(),
         view_name='repositories-detail',
+        write_only=True
     )
 
     def validate(self, data):


Test patch:
Repository: pulp/pulpcore

Issue:
Delete artifact files after sucessful transaction



Discussion:

Resolving patch:
diff --git a/pulpcore/app/middleware.py b/pulpcore/app/middleware.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/app/middleware.py
@@ -0,0 +1,7 @@
+from django.contrib.auth.middleware import RemoteUserMiddleware
+
+from pulpcore.app import settings
+
+
+class PulpRemoteUserMiddleware(RemoteUserMiddleware):
+    header = settings.REMOTE_USER_ENVIRON_NAME
diff --git a/pulpcore/app/settings.py b/pulpcore/app/settings.py
--- a/pulpcore/app/settings.py
+++ b/pulpcore/app/settings.py
@@ -84,10 +84,16 @@
     'django.middleware.common.CommonMiddleware',
     'django.middleware.csrf.CsrfViewMiddleware',
     'django.contrib.auth.middleware.AuthenticationMiddleware',
+    'pulpcore.app.middleware.PulpRemoteUserMiddleware',
     'django.contrib.messages.middleware.MessageMiddleware',
     'django.middleware.clickjacking.XFrameOptionsMiddleware',
 ]
 
+AUTHENTICATION_BACKENDS = [
+    'django.contrib.auth.backends.ModelBackend',
+    'django.contrib.auth.backends.RemoteUserBackend',
+]
+
 ROOT_URLCONF = 'pulpcore.app.urls'
 
 TEMPLATES = [
@@ -116,6 +122,7 @@
     'DEFAULT_PERMISSION_CLASSES': ('rest_framework.permissions.IsAuthenticated',),
     'DEFAULT_AUTHENTICATION_CLASSES': (
         'rest_framework.authentication.SessionAuthentication',
+        'rest_framework.authentication.RemoteUserAuthentication',
         'rest_framework.authentication.BasicAuthentication',
     ),
     'UPLOADED_FILES_USE_URL': False,
@@ -198,6 +205,8 @@
 CONTENT_HOST = None
 CONTENT_PATH_PREFIX = '/pulp/content/'
 
+REMOTE_USER_ENVIRON_NAME = "REMOTE_USER"
+
 PROFILE_STAGES_API = False
 
 SWAGGER_SETTINGS = {


Test patch:
Repository: pulp/pulpcore

Issue:
[PR #5015/8c3dcd88 backport][3.22] Backport the `--nightly` pytest arg
**This is a backport of PR #5015 as merged into 3.28 (8c3dcd887caa7ef6f940cb49cb627200fbc3bf6b).**

This pytest flag used to be provided by pulp-smash, but we no longer want to depend on that. So we need to provide this functionality in the pulpcore pytest plugin. A safeguard is in place to not totally confuse pytest if pulp-smash happens to be installed anyway.

[noissue]

(cherry picked from commit c6135e81ae8aacd9c6dde092856d1a12bfdac8ab)


Discussion:

Resolving patch:
diff --git a/pulpcore/app/viewsets/content.py b/pulpcore/app/viewsets/content.py
--- a/pulpcore/app/viewsets/content.py
+++ b/pulpcore/app/viewsets/content.py
@@ -2,7 +2,6 @@
 
 from django.db import models
 from rest_framework import mixins, status
-from rest_framework.parsers import FormParser, MultiPartParser
 from rest_framework.response import Response
 
 from pulpcore.app.models import Artifact, Content
@@ -50,7 +49,6 @@ class ArtifactViewSet(NamedModelViewSet,
     queryset = Artifact.objects.all()
     serializer_class = ArtifactSerializer
     filterset_class = ArtifactFilter
-    parser_classes = (MultiPartParser, FormParser)
 
     def destroy(self, request, pk):
         """


Test patch:
Repository: pulp/pulpcore

Issue:
Deps 3 22



Discussion:

Resolving patch:
diff --git a/pulpcore/app/serializers/repository.py b/pulpcore/app/serializers/repository.py
--- a/pulpcore/app/serializers/repository.py
+++ b/pulpcore/app/serializers/repository.py
@@ -111,9 +111,10 @@ class RemoteSerializer(MasterModelSerializer):
         min_value=1
     )
     policy = serializers.ChoiceField(
-        help_text="The policy to use when downloading content. The possible values include: "
-                  "'immediate', 'on_demand', and 'cache_only'. 'immediate' is the default.",
-        choices=models.Remote.POLICY_CHOICES,
+        help_text="The policy to use when downloading content.",
+        choices=(
+            (models.Remote.IMMEDIATE, 'When syncing, download all metadata and content now.')
+        ),
         default=models.Remote.IMMEDIATE
     )
 


Test patch:
Repository: pulp/pulpcore

Issue:
Verify if domain name has more than 50 chars
fixes: #4976


Discussion:

Resolving patch:
diff --git a/pulpcore/app/viewsets/upload.py b/pulpcore/app/viewsets/upload.py
--- a/pulpcore/app/viewsets/upload.py
+++ b/pulpcore/app/viewsets/upload.py
@@ -10,19 +10,33 @@
 
 from pulpcore.app.models import Upload
 from pulpcore.app.serializers import UploadChunkSerializer, UploadCommitSerializer, UploadSerializer
-from pulpcore.app.viewsets.base import NamedModelViewSet
+from pulpcore.app.viewsets import BaseFilterSet
+from pulpcore.app.viewsets.base import DATETIME_FILTER_OPTIONS, NamedModelViewSet
+from pulpcore.app.viewsets.custom_filters import IsoDateTimeFilter
+
+
+class UploadFilter(BaseFilterSet):
+    completed = IsoDateTimeFilter(field_name='completed')
+
+    class Meta:
+        model = Upload
+        fields = {
+            'completed': DATETIME_FILTER_OPTIONS + ['isnull']
+        }
 
 
 class UploadViewSet(NamedModelViewSet,
                     mixins.CreateModelMixin,
                     mixins.RetrieveModelMixin,
                     mixins.UpdateModelMixin,
+                    mixins.DestroyModelMixin,
                     mixins.ListModelMixin):
     """View for chunked uploads."""
     endpoint_name = 'uploads'
     queryset = Upload.objects.all()
     serializer_class = UploadSerializer
-    http_method_names = ['get', 'post', 'head', 'put']
+    filterset_class = UploadFilter
+    http_method_names = ['get', 'post', 'head', 'put', 'delete']  # remove PATCH
 
     content_range_pattern = r'^bytes (\d+)-(\d+)/(\d+|[*])$'
     content_range_parameter = \


Test patch:
Repository: pulp/pulpcore

Issue:
Update CI files for branch main
[noissue]


Discussion:

Resolving patch:
diff --git a/pulpcore/app/migrations/0003_remove_remote_validate.py b/pulpcore/app/migrations/0003_remove_remote_validate.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/app/migrations/0003_remove_remote_validate.py
@@ -0,0 +1,17 @@
+# Generated by Django 2.2.2 on 2019-06-27 20:20
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0002_contentappstatus'),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name='remote',
+            name='validate',
+        ),
+    ]
diff --git a/pulpcore/app/models/repository.py b/pulpcore/app/models/repository.py
--- a/pulpcore/app/models/repository.py
+++ b/pulpcore/app/models/repository.py
@@ -58,7 +58,6 @@ class Remote(MasterModel):
 
         name (models.CharField): The remote name.
         url (models.TextField): The URL of an external content source.
-        validate (models.BooleanField): If True, the plugin will validate imported files.
         ssl_ca_certificate (models.FileField): A PEM encoded CA certificate used to validate the
             server certificate presented by the external source.
         ssl_client_certificate (models.FileField): A PEM encoded client certificate used
@@ -97,7 +96,6 @@ class Remote(MasterModel):
     name = models.CharField(db_index=True, unique=True, max_length=255)
 
     url = models.TextField()
-    validate = models.BooleanField(default=True)
 
     ssl_ca_certificate = models.TextField(null=True)
     ssl_client_certificate = models.TextField(null=True)
diff --git a/pulpcore/app/serializers/repository.py b/pulpcore/app/serializers/repository.py
--- a/pulpcore/app/serializers/repository.py
+++ b/pulpcore/app/serializers/repository.py
@@ -55,10 +55,6 @@ class RemoteSerializer(MasterModelSerializer):
     url = serializers.CharField(
         help_text='The URL of an external content source.',
     )
-    validate = serializers.BooleanField(
-        help_text='If True, the plugin will validate imported artifacts.',
-        required=False,
-    )
     ssl_ca_certificate = SecretCharField(
         help_text='A string containing the PEM encoded CA certificate used to validate the server '
                   'certificate presented by the remote server. All new line characters must be '
@@ -122,9 +118,10 @@ class Meta:
         abstract = True
         model = models.Remote
         fields = MasterModelSerializer.Meta.fields + (
-            'name', 'url', 'validate', 'ssl_ca_certificate', 'ssl_client_certificate',
-            'ssl_client_key', 'ssl_validation', 'proxy_url', 'username', 'password',
-            '_last_updated', 'download_concurrency', 'policy')
+            'name', 'url', 'ssl_ca_certificate', 'ssl_client_certificate', 'ssl_client_key',
+            'ssl_validation', 'proxy_url', 'username', 'password', '_last_updated',
+            'download_concurrency', 'policy'
+        )
 
 
 class RepositorySyncURLSerializer(serializers.Serializer):


Test patch:
Repository: pulp/pulpcore

Issue:
[RFE?] Keep worker process running in case of db connection error
**Is your feature request related to a problem? Please describe.**
From time to time my Postgres instance restarts (not related to Pulp) which, in turn, causes a database connection error and pulpcore worker process exits (and pod restarts).


**Describe the solution you'd like**
Is there a way to handle these connection drops on pulpcore worker side? For example, the main process from pulpcore-api pods is a gunicorn app that forks children processes and whenever the connection to the database is lost, the api pods will throw exceptions but the main gunicorn process will keep running (only the forks will die), so this causes no pod restart. Is it possible to provide a similar behavior in pulpcore-worker pods so that the main process does not exit in case of database failure?  


**Additional context**
* We understand that the main problem is on the database side and the idea of this issue is just a "workaround".
* If we manage to find a way to keep the process running (and not restart the pod) will it handle/ensure the reconnection to the database?



Discussion:

Resolving patch:
diff --git a/pulpcore/app/migrations/0002_increase_artifact_size_field.py b/pulpcore/app/migrations/0002_increase_artifact_size_field.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/app/migrations/0002_increase_artifact_size_field.py
@@ -0,0 +1,18 @@
+# Generated by Django 2.2.2 on 2019-07-05 17:06
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='artifact',
+            name='size',
+            field=models.BigIntegerField(),
+        ),
+    ]
diff --git a/pulpcore/app/models/content.py b/pulpcore/app/models/content.py
--- a/pulpcore/app/models/content.py
+++ b/pulpcore/app/models/content.py
@@ -103,7 +103,7 @@ def storage_path(self, name):
         return storage.get_artifact_path(self.sha256)
 
     file = fields.ArtifactFileField(null=False, upload_to=storage_path, max_length=255)
-    size = models.IntegerField(null=False)
+    size = models.BigIntegerField(null=False)
     md5 = models.CharField(max_length=32, null=False, unique=False, db_index=True)
     sha1 = models.CharField(max_length=40, null=False, unique=False, db_index=True)
     sha224 = models.CharField(max_length=56, null=False, unique=False, db_index=True)


Test patch:
Repository: pulp/pulpcore

Issue:
Update CI files for branch 3.21



Discussion:

Resolving patch:
diff --git a/pulpcore/app/settings.py b/pulpcore/app/settings.py
--- a/pulpcore/app/settings.py
+++ b/pulpcore/app/settings.py
@@ -201,7 +201,7 @@
     }
 }
 
-CONTENT_HOST = None
+CONTENT_HOST = ''
 CONTENT_PATH_PREFIX = '/pulp/content/'
 CONTENT_APP_TTL = 30
 


Test patch: