Repository: pulp/pulpcore

Issue:
Migration of encrypted fields between 3.14 and 3.16 is broken for some remotes
**Version**

After upgrading from Katello 4.2 to 4.3, some users are encountering sync problems with some repositories.  A 500 error is triggered when the /repositories/.../sync/ endpoint is used.  The error seems to be related to the client_cert / client_key / ca_cert values.

```
Feb 24 11:28:57 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]: django.request:ERROR: Internal Server Error: /pulp/api/v3/remotes/rpm/rpm/89e5b587-2295-4f1d-a49d-c0e8ed518c67/
Feb 24 11:28:57 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 96, in _get_unverified_token_data
Feb 24 11:28:57 foreman pulpcore-api: data = base64.urlsafe_b64decode(token)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 133, in urlsafe_b64decode
Feb 24 11:28:57 foreman pulpcore-api: return b64decode(s)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 87, in b64decode
Feb 24 11:28:57 foreman pulpcore-api: return binascii.a2b_base64(s)
Feb 24 11:28:57 foreman pulpcore-api: binascii.Error: Incorrect padding
Feb 24 11:28:57 foreman pulpcore-api: During handling of the above exception, another exception occurred:
Feb 24 11:28:57 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
Feb 24 11:28:57 foreman pulpcore-api: response = get_response(request)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
Feb 24 11:28:57 foreman pulpcore-api: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Feb 24 11:28:57 foreman pulpcore-api: return view_func(*args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
Feb 24 11:28:57 foreman pulpcore-api: return self.dispatch(request, *args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
Feb 24 11:28:57 foreman pulpcore-api: response = self.handle_exception(exc)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
Feb 24 11:28:57 foreman pulpcore-api: self.raise_uncaught_exception(exc)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Feb 24 11:28:57 foreman pulpcore-api: raise exc
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
Feb 24 11:28:57 foreman pulpcore-api: response = handler(request, *args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/base.py", line 470, in partial_update
Feb 24 11:28:57 foreman pulpcore-api: return self.update(request, *args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/viewsets/base.py", line 452, in update
Feb 24 11:28:57 foreman pulpcore-api: instance = self.get_object()
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 96, in get_object
Feb 24 11:28:57 foreman pulpcore-api: obj = get_object_or_404(queryset, **filter_kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/generics.py", line 19, in get_object_or_404
Feb 24 11:28:57 foreman pulpcore-api: return _get_object_or_404(queryset, *filter_args, **filter_kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/shortcuts.py", line 76, in get_object_or_404
Feb 24 11:28:57 foreman pulpcore-api: return queryset.get(*args, **kwargs)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
Feb 24 11:28:57 foreman pulpcore-api: num = len(clone)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
Feb 24 11:28:57 foreman pulpcore-api: self._fetch_all()
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Feb 24 11:28:57 foreman pulpcore-api: self._result_cache = list(self._iterable_class(self))
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 68, in __iter__
Feb 24 11:28:57 foreman pulpcore-api: for row in compiler.results_iter(results):
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
Feb 24 11:28:57 foreman pulpcore-api: value = converter(value, expression, connection)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
Feb 24 11:28:57 foreman pulpcore-api: return force_str(self._fernet.decrypt(force_bytes(value)))
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
Feb 24 11:28:57 foreman pulpcore-api: timestamp, data = Fernet._get_unverified_token_data(token)
Feb 24 11:28:57 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 98, in _get_unverified_token_data
Feb 24 11:28:57 foreman pulpcore-api: raise InvalidToken
Feb 24 11:28:57 foreman pulpcore-api: cryptography.fernet.InvalidToken
Feb 24 11:28:57 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]:  - - [24/Feb/2022:10:28:57 +0000] "PATCH /pulp/api/v3/remotes/rpm/rpm/89e5b587-2295-4f1d-a49d-c0e8ed518c67/ HTTP/1.1" 500 145 "-" "OpenAPI-Generator/3.16.1/ruby"
Feb 24 11:28:58 foreman qdrouterd: SERVER (info) [C464018] Connection from 10.2.1.176:50898 (to :5647) failed: amqp:resource-limit-exceeded local-idle-timeout expired
Feb 24 11:28:58 foreman qdrouterd: ROUTER_CORE (info) [C464018][L1196864] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Feb 24 11:28:58 foreman qdrouterd: ROUTER_CORE (info) [C464018] Connection Closed
Feb 24 11:28:58 foreman qdrouterd: ROUTER_CORE (info) [C4][L1196865] Link detached: del=0 presett=0 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0 blocked=no
Feb 24 11:28:58 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]: django.request:ERROR: Internal Server Error: /pulp/api/v3/repositories/rpm/rpm/b4a448e8-896c-4fec-8ee2-fcdf4307f2fd/sync/
Feb 24 11:28:58 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 96, in _get_unverified_token_data
Feb 24 11:28:58 foreman pulpcore-api: data = base64.urlsafe_b64decode(token)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 133, in urlsafe_b64decode
Feb 24 11:28:58 foreman pulpcore-api: return b64decode(s)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/rh/rh-python38/root/usr/lib64/python3.8/base64.py", line 87, in b64decode
Feb 24 11:28:58 foreman pulpcore-api: return binascii.a2b_base64(s)
Feb 24 11:28:58 foreman pulpcore-api: binascii.Error: Incorrect padding
Feb 24 11:28:58 foreman pulpcore-api: During handling of the above exception, another exception occurred:
Feb 24 11:28:58 foreman pulpcore-api: Traceback (most recent call last):
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
Feb 24 11:28:58 foreman pulpcore-api: response = get_response(request)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
Feb 24 11:28:58 foreman pulpcore-api: response = wrapped_callback(request, *callback_args, **callback_kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
Feb 24 11:28:58 foreman pulpcore-api: return view_func(*args, **kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
Feb 24 11:28:58 foreman pulpcore-api: return self.dispatch(request, *args, **kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
Feb 24 11:28:58 foreman pulpcore-api: response = self.handle_exception(exc)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
Feb 24 11:28:58 foreman pulpcore-api: self.raise_uncaught_exception(exc)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
Feb 24 11:28:58 foreman pulpcore-api: raise exc
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
Feb 24 11:28:58 foreman pulpcore-api: response = handler(request, *args, **kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulp_rpm/app/viewsets.py", line 130, in sync
Feb 24 11:28:58 foreman pulpcore-api: serializer.is_valid(raise_exception=True)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/serializers.py", line 220, in is_valid
Feb 24 11:28:58 foreman pulpcore-api: self._validated_data = self.run_validation(self.initial_data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/serializers.py", line 419, in run_validation
Feb 24 11:28:58 foreman pulpcore-api: value = self.to_internal_value(data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/serializers.py", line 476, in to_internal_value
Feb 24 11:28:58 foreman pulpcore-api: validated_value = field.run_validation(primitive_value)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/relations.py", line 153, in run_validation
Feb 24 11:28:58 foreman pulpcore-api: return super().run_validation(data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/fields.py", line 568, in run_validation
Feb 24 11:28:58 foreman pulpcore-api: value = self.to_internal_value(data)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/relations.py", line 373, in to_internal_value
Feb 24 11:28:58 foreman pulpcore-api: return self.get_object(match.view_name, match.args, match.kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/serializers/base.py", line 322, in get_object
Feb 24 11:28:58 foreman pulpcore-api: return super().get_object(*args, **kwargs).cast()
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/rest_framework/relations.py", line 318, in get_object
Feb 24 11:28:58 foreman pulpcore-api: return queryset.get(**lookup_kwargs)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 431, in get
Feb 24 11:28:58 foreman pulpcore-api: num = len(clone)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 262, in __len__
Feb 24 11:28:58 foreman pulpcore-api: self._fetch_all()
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 1324, in _fetch_all
Feb 24 11:28:58 foreman pulpcore-api: self._result_cache = list(self._iterable_class(self))
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/query.py", line 68, in __iter__
Feb 24 11:28:58 foreman pulpcore-api: for row in compiler.results_iter(results):
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1122, in apply_converters
Feb 24 11:28:58 foreman pulpcore-api: value = converter(value, expression, connection)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib/python3.8/site-packages/pulpcore/app/models/fields.py", line 104, in from_db_value
Feb 24 11:28:58 foreman pulpcore-api: return force_str(self._fernet.decrypt(force_bytes(value)))
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
Feb 24 11:28:58 foreman pulpcore-api: timestamp, data = Fernet._get_unverified_token_data(token)
Feb 24 11:28:58 foreman pulpcore-api: File "/opt/theforeman/tfm-pulpcore/root/usr/lib64/python3.8/site-packages/cryptography/fernet.py", line 98, in _get_unverified_token_data
Feb 24 11:28:58 foreman pulpcore-api: raise InvalidToken
Feb 24 11:28:58 foreman pulpcore-api: cryptography.fernet.InvalidToken
Feb 24 11:28:58 foreman pulpcore-api: pulp [e2601bd6-70a5-4a87-ace3-a386280e3e30]:  - - [24/Feb/2022:10:28:58 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/b4a448e8-896c-4fec-8ee2-fcdf4307f2fd/sync/ HTTP/1.1" 500 145 
```

> 
> What did notice, though, is that on the old 4.2 server with pulpcore 3.14 all those remotes have an empty username and password set, i.e. an empty string “” and not NULL like many others. However, the list of remotes on the old server with empty string username is longer than the affected ones on my 4.3 server.
> 
> So it seems to be an issue with the database migration from 3.14 to 3.16 and encryption of some fields in the database. The migration/encryption seems to fail is some cases and the original content remains in the database which makes decryption failed. This is also true for my affected rhel repository: I can see in the database that the remote contains the RSA private key in client_key, unlike the other rhel repos which contain some base64 encoded string in the client_key fields.
> 
> Thus the problem is that for some rows in the database (at least the core_remote table) encryption of the fields client_key, username, password, (proxy_username, proxy_password, too, I guess) fails during migration from 3.14 to 3.16 leaving the original content in the database. This leads to decryption failure in 3.16.
> 
> For whatever reason only a few repos are affected, mostly those which have empty strings for username, password instead of NULL like many others. However, even with empty strings for some remotes the encryption works while it doesn’t for others.

**Describe the bug**

API calls to trigger a sync return a 500 error for some remotes

**To Reproduce**

Unclear

**Expected behavior**

A sync task is correctly created, no 500 error.

**Additional context**
Discourse discussion: https://community.theforeman.org/t/pulprpmclient-apierror-http-500-during-sync-repository/27480/13?u=dralley



Discussion:
Looking closer into the encrypted strings in the database after migration from 3.14 to 3.16 I have noticed that the empty strings are consecutive in an unordered select. That made me suspect that the problem is the chunking and the assumption the order of rows won't change between chunks. To verify I have decrypted all encrypted usernames in the migrated database and I found what I have suspected:

```
>>> f.decrypt(b'gAAAAABiKvJgZhyIdXy__3X2rh8QdXezaWj-Y-RelFEfYIWg2mrTREsKTB7ydPY2gn3ZhveMwE3ocN1KO8YV3h5iA-wMibo_aw==')
b''
>>> f.decrypt(b'gAAAAABiKvJzCDPxmuSFmmAawerxPi1AqUCP4H8NxWiO0ypnYwFraXPj35EWQ4ABupu_KIBbBPFhW2elE_4Ru6FQQWRggn1yeg==')
b''
>>> f.decrypt(b'gAAAAABiKvGrd6IVPXjJZuTUPYxXg_F3jXvaMmbH3l_O2x1hNnxG8vBKeTHav_0Bz2rjsjcUc6CH_K4eapwLpV0tNGF_cJZKbRbqsSB_JZTQyjW8jSovvTipMSsbWeQJJZ-B5yLWk6vBnNk9cQ81I6kQOnXZolXbRfIPFdPM9AhwCJro8vnDcN4AQ5NKe9dyOVM80hHDquUW2IavogypDl9XLbsnr6m9KQ==')
b'gAAAAABiKe5e7RWrDl9cNAUTLHZ9CjN30uvPAZ_KZZskG_pyCBDCJBJ5pY6pvKaidZltPjWa0iqLP8RsKGakC8fpTi5xMz-c6Q=='
...
```

Obviously the last one above has been encrypted twice. So basically, the migration does encrypt some rows twice and some not at all.

I guess the reason is in 0073_encrypt_remote_fields.py:
```
def encrypt_remote_fields(apps, schema_editor):
    offset = 0
    chunk_size = 100
    Remote = apps.get_model("core", "Remote")

    with connection.cursor() as cursor:
        while True:
            cursor.execute(
                f"SELECT pulp_id, {(',').join(fields)} FROM "
                f"core_remote LIMIT {chunk_size} OFFSET {offset}"
            )
            records = cursor.fetchall()
            offset += chunk_size

            if len(records) == 0:
                break

            for record in records:
                update = {
                    field: record[i] for i, field in enumerate(fields, 1) if record[i] is not None
                }
                if not update:
                    continue

                Remote.objects.filter(pk=record[0]).update(**update)
```

I have more than 100 rows in core_remote, i.e. it runs two selects:
```
pulpcore=# select pulp_id, "username", "password", "proxy_username", "proxy_password", "client_key" from core_remote limit 100 offset 0;
pulpcore=# select pulp_id, "username", "password", "proxy_username", "proxy_password", "client_key" from core_remote limit 100 offset 100;
```

This assumes that the order of the rows in the whole table does not change between the first and second select. I suspect that this isn't true. Comparing the select on the database before the migration and after returns rows definitively in different order.

So the migration should make sure to maintain order of returned rows, e.g. by adding a `ORDER by pulp_id`. That way the chunks should work unless some other operation in between would insert or delete rows or modify pulp_ids...

Of course this leaves people who have already migrated and cannot go back with broken content in those fields: some fields may still contain plaintext content which causes decryption to fail with this InvalidToken exception. Other fields are double encrypted and the decrypt will return the single encrypted string instead of the plaintext causing authentication to fail.

To make matters worse: at least on my Katello 4.2 server there are quite a few rows in core_remote containing empty strings "" in those fields instead of NULL, causing the migration to encrypt the empty string. Due to this, I have 41 rows in the 3.16 table with encrypted usernames even though I actually only have 4 repositories having a non-empty username...
Thinking a little bit more about it, I think LIMIT/chunking in the migration is simply wrong. It should just use the standard postgresql cursor/buffering to go through all rows and not trying to split it up manually. Using LIMIT/OFFSET for paging through an unsorted table is just conceptionally flawed.

In addition: technically, the SELECT and the following updates should be in a transaction block to make sure it's consistent and no other process can get in between the select and the updates.

Or otherwise the migration could temporarily add encrypted columns for those 5 fields to be encrypted and write the encrypted text there. This allows the migration to loop through the table (even with limit) and find rows which still need to be encrypted. Once there are not rows left to be encrypted, a single update can move the encrypted texts from the temporary columns into the normal columns and after that you can drop the temporary columns.
>So the migration should make sure to maintain order of returned rows, e.g. by adding a ORDER by pulp_id. That way the chunks should work unless some other operation in between would insert or delete rows or modify pulp_ids...

Yes, your theory is almost certainly the correct root cause.  That needs to be fixed as you've described, using ordered queries.

>In addition: technically, the SELECT and the following updates should be in a transaction block to make sure it's consistent and no other process can get in between the select and the updates.

This part should be fine though.  All Django migrations implicitly run inside a transaction unless you manually specify otherwise, and that isn't the case here.
> Yes, your theory is almost certainly the correct root cause. That needs to be fixed as you've described, using ordered queries.

Do you really think splitting the table into chunks is necessary? For the core_remote table, even if someone has a couple of thousand repositories, a single select of the whole table should easily fit into memory. And more importantly: you could easily optimize the select by search only for rows which actually have something set in on of the five columns to be encrypted. In my case, that would return 4 rows with username/password set and 5 redhat repositories with client keys (after setting the empty string "" columns to null)...

Chunks can be useful if it's a huge table with many (millions) of rows. But in that case, you actually wouldn't want to sort the table because it is much more efficient to deliver the rows in database order.

Isn't it possible with django to loop through the rows from the database directly, i.e. instead of using fetchall which I assume retrieves all rows from the select have something like a simple "fetch" to get the next row from the cursor until it hits the end?
>Do you really think splitting the table into chunks is necessary? For the core_remote table, even if someone has a couple of thousand repositories, a single select of the whole table should easily fit into memory.

I don't know why it was done this way in particular, the comments on that [PR](https://github.com/pulp/pulpcore/pull/1301) aren't illuminating, but I'll bring it up on Monday and see if that whole requirement can be dropped.

Resolving patch:
diff --git a/pulpcore/app/management/commands/datarepair-2327.py b/pulpcore/app/management/commands/datarepair-2327.py
new file mode 100644
--- /dev/null
+++ b/pulpcore/app/management/commands/datarepair-2327.py
@@ -0,0 +1,120 @@
+from gettext import gettext as _
+
+from django.db import connection
+from django.conf import settings
+from django.core.management import BaseCommand
+from django.db.models import Q
+from django.utils.encoding import force_bytes, force_str
+
+import cryptography
+
+from pulpcore.app.models import Remote
+
+
+class Command(BaseCommand):
+    """
+    Django management command for repairing incorrectly migrated remote data.
+    """
+
+    help = _(
+        "Repairs issue #2327. A small number of configuration settings may have been "
+        "corrupted during an upgrade from a previous version of Pulp to a Pulp version "
+        "between 3.15-3.18, resulting in trouble when syncing or viewing certain remotes. "
+        "This script repairs the data (which was not lost)."
+    )
+
+    def add_arguments(self, parser):
+        """Set up arguments."""
+        parser.add_argument(
+            "--dry-run",
+            action="store_true",
+            help=_("Don't modify anything, just collect results on how many Remotes are impacted."),
+        )
+
+    def handle(self, *args, **options):
+
+        dry_run = options["dry_run"]
+        fields = ("username", "password", "proxy_username", "proxy_password", "client_key")
+
+        with open(settings.DB_ENCRYPTION_KEY, "rb") as key_file:
+            fernet = cryptography.fernet.Fernet(key_file.read())
+
+        possibly_affected_remotes = (
+            Q(username__isnull=False)
+            | Q(password__isnull=False)
+            | Q(proxy_username__isnull=False)
+            | Q(proxy_password__isnull=False)
+            | Q(client_key__isnull=False)
+        )
+
+        number_unencrypted = 0
+        number_multi_encrypted = 0
+
+        for remote_pk in Remote.objects.filter(possibly_affected_remotes).values_list(
+            "pk", flat=True
+        ):
+            try:
+                remote = Remote.objects.get(pk=remote_pk)
+                # if we can get the remote successfully, it is either OK or the fields are
+                # encrypted more than once
+            except cryptography.fernet.InvalidToken:
+                # If decryption fails then it probably hasn't been encrypted yet
+                # get the raw column value, avoiding any Django field handling
+                with connection.cursor() as cursor:
+                    cursor.execute(
+                        "SELECT username, password, proxy_username, proxy_password, client_key "
+                        "FROM core_remote WHERE pulp_id = %s",
+                        [str(remote_pk)],
+                    )
+                    row = cursor.fetchone()
+
+                field_values = {}
+
+                for field, value in zip(fields, row):
+                    field_values[field] = value
+
+                if not dry_run:
+                    Remote.objects.filter(pk=remote_pk).update(**field_values)
+                number_unencrypted += 1
+            else:
+                times_decrypted = 0
+                keep_trying = True
+                needs_update = False
+
+                while keep_trying:
+                    for field in fields:
+                        field_value = getattr(remote, field)  # value gets decrypted once on access
+                        if not field_value:
+                            continue
+
+                        try:
+                            # try to decrypt it again
+                            field_value = force_str(fernet.decrypt(force_bytes(field_value)))
+                            # it was decrypted successfully again time, so it was probably
+                            # encrypted multiple times over. lets re-set the value with the
+                            # newly decrypted value
+                            setattr(remote, field, field_value)
+                            needs_update = True
+                        except cryptography.fernet.InvalidToken:
+                            # couldn't be decrypted again, stop here
+                            keep_trying = False
+
+                    times_decrypted += 1
+
+                if needs_update:
+                    if not dry_run:
+                        remote.save()
+                    number_multi_encrypted += 1
+
+        if dry_run:
+            print("Remotes with un-encrypted fields: {}".format(number_unencrypted))
+            print("Remotes encrypted multiple times: {}".format(number_multi_encrypted))
+        else:
+            if not number_unencrypted and not number_multi_encrypted:
+                print("Finished. (OK)")
+            else:
+                print(
+                    "Finished. ({} remotes fixed)".format(
+                        number_unencrypted + number_multi_encrypted
+                    )
+                )
diff --git a/pulpcore/app/migrations/0073_encrypt_remote_fields.py b/pulpcore/app/migrations/0073_encrypt_remote_fields.py
--- a/pulpcore/app/migrations/0073_encrypt_remote_fields.py
+++ b/pulpcore/app/migrations/0073_encrypt_remote_fields.py
@@ -1,59 +1,53 @@
 # Generated by Django 2.2.20 on 2021-04-29 14:33
 
-from django.db import connection, migrations
-from django.db.models import Q
-
+from django.db import migrations
 import pulpcore.app.models.fields
 
 fields = ("username", "password", "proxy_username", "proxy_password", "client_key")
+new_fields = ("_encrypted_username", "_encrypted_password", "_encrypted_proxy_username", "_encrypted_proxy_password", "_encrypted_client_key")
 
 
 def encrypt_remote_fields(apps, schema_editor):
-    offset = 0
-    chunk_size = 100
     Remote = apps.get_model("core", "Remote")
 
-    with connection.cursor() as cursor:
-        while True:
-            cursor.execute(
-                f"SELECT pulp_id, {(',').join(fields)} FROM "
-                f"core_remote LIMIT {chunk_size} OFFSET {offset}"
-            )
-            records = cursor.fetchall()
-            offset += chunk_size
+    remotes_needing_update = []
+    for remote in Remote.objects.all().iterator():
+        if not any([getattr(remote, field) for field in fields]):
+            continue
 
-            if len(records) == 0:
-                break
+        remote._encrypted_username = remote.username
+        remote._encrypted_password = remote.password
+        remote._encrypted_proxy_username = remote.proxy_username
+        remote._encrypted_proxy_password = remote.proxy_password
+        remote._encrypted_client_key = remote.client_key
+        remotes_needing_update.append(remote)
 
-            for record in records:
-                update = {
-                    field: record[i] for i, field in enumerate(fields, 1) if record[i] is not None
-                }
-                if not update:
-                    continue
+        if len(remotes_needing_update) > 100:
+            Remote.objects.bulk_update(remotes_needing_update, new_fields)
+            remotes_needing_update.clear()
 
-                Remote.objects.filter(pk=record[0]).update(**update)
+    Remote.objects.bulk_update(remotes_needing_update, new_fields)
 
 
 def unencrypt_remote_fields(apps, schema_editor):
     Remote = apps.get_model("core", "Remote")
 
-    q = Q()
-    for field in fields:
-        q &= Q(**{field: None}) | Q(**{field: ""})
+    remotes_needing_update = []
+    for remote in Remote.objects.all().iterator():
+        if not any([getattr(remote, field) for field in new_fields]):
+            continue
+        remote.username = remote._encrypted_username
+        remote.password = remote._encrypted_password
+        remote.proxy_username = remote._encrypted_proxy_username
+        remote.proxy_password = remote._encrypted_proxy_password
+        remote.client_key = remote._encrypted_client_key
+        remotes_needing_update.append(remote)
 
-    for remote in Remote.objects.exclude(q):
-        update = [
-            f"{field} = '{getattr(remote, field)}'"
-            for field in fields
-            if getattr(remote, field) is not None
-        ]
-        query = (
-            f"UPDATE core_remote cr SET {(', ').join(update)} WHERE pulp_id = '{remote.pulp_id}'"
-        )
+        if len(remotes_needing_update) > 100:
+            Remote.objects.bulk_update(remotes_needing_update, fields)
+            remotes_needing_update.clear()
 
-        with connection.cursor() as cursor:
-            cursor.execute(query)
+    Remote.objects.bulk_update(remotes_needing_update, fields)
 
 
 class Migration(migrations.Migration):
@@ -63,33 +57,82 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
-        migrations.AlterField(
+        # Add new fields to temporarily hold the encrypted values
+        migrations.AddField(
             model_name="remote",
-            name="client_key",
+            name="_encrypted_client_key",
             field=pulpcore.app.models.fields.EncryptedTextField(null=True),
         ),
-        migrations.AlterField(
+        migrations.AddField(
             model_name="remote",
-            name="password",
+            name="_encrypted_password",
             field=pulpcore.app.models.fields.EncryptedTextField(null=True),
         ),
-        migrations.AlterField(
+        migrations.AddField(
             model_name="remote",
-            name="proxy_password",
+            name="_encrypted_proxy_password",
             field=pulpcore.app.models.fields.EncryptedTextField(null=True),
         ),
-        migrations.AlterField(
+        migrations.AddField(
             model_name="remote",
-            name="proxy_username",
+            name="_encrypted_proxy_username",
             field=pulpcore.app.models.fields.EncryptedTextField(null=True),
         ),
-        migrations.AlterField(
+        migrations.AddField(
             model_name="remote",
-            name="username",
+            name="_encrypted_username",
             field=pulpcore.app.models.fields.EncryptedTextField(null=True),
         ),
+        # Populate the new fields with encrypted values computed from the unencrypted fields
         migrations.RunPython(
             code=encrypt_remote_fields,
             reverse_code=unencrypt_remote_fields,
         ),
+        # Remove the unencrypted columns
+        migrations.RemoveField(
+            model_name="remote",
+            name="client_key",
+        ),
+        migrations.RemoveField(
+            model_name="remote",
+            name="password",
+        ),
+        migrations.RemoveField(
+            model_name="remote",
+            name="proxy_password",
+        ),
+        migrations.RemoveField(
+            model_name="remote",
+            name="proxy_username",
+        ),
+        migrations.RemoveField(
+            model_name="remote",
+            name="username",
+        ),
+        # Replace the formerly-unencrypted columns with the new encrypted ones
+        migrations.RenameField(
+            model_name="remote",
+            old_name="_encrypted_client_key",
+            new_name="client_key",
+        ),
+        migrations.RenameField(
+            model_name="remote",
+            old_name="_encrypted_password",
+            new_name="password",
+        ),
+        migrations.RenameField(
+            model_name="remote",
+            old_name="_encrypted_proxy_password",
+            new_name="proxy_password",
+        ),
+        migrations.RenameField(
+            model_name="remote",
+            old_name="_encrypted_proxy_username",
+            new_name="proxy_username",
+        ),
+        migrations.RenameField(
+            model_name="remote",
+            old_name="_encrypted_username",
+            new_name="username",
+        ),
     ]


Test patch: