Repository: pulp/pulpcore

Issue:
After pulp-containerupgrade 2.9-->main push to new repos fails
**Version**
Please provide the versions of the pulpcore and pulp_container packages in use, and how they are installed.  If you are using Pulp via Katello, please provide the Katello version.

**Describe the bug**
**To Reproduce**
1. install 2.9
2. push repo ipanova/busybox
3. upgrade to main
4. push new content to ipanova/busybox. Succeed
5. push to new repo ipanova/test
6. fail with 500
 
**Push operatiosns fail  whether with ipanova user who has according perms or admin**


For some reason the traceback is swallowed, i could extract it only from the remote debugger:

`Out[2]: rest_framework.exceptions.APIException("Creation hook 'add_perms_to_distribution_group' was not registered for this view set.")
`

**Expected behavior**
A clear and concise description of what you expected to happen.

**Additional context**
```python
$ http https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/access_policies/?viewset_name=repositories/container/container-push  --auth admin:password 
HTTP/1.1 200 OK
Access-Control-Expose-Headers: Correlation-ID
Allow: GET, HEAD, OPTIONS
Connection: keep-alive
Content-Length: 2722
Content-Type: application/json
Correlation-ID: f71cf0881d7d47c4885d248d7fe3ef82
Date: Mon, 31 Oct 2022 18:00:46 GMT
Referrer-Policy: same-origin
Server: nginx
Vary: Accept, Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{
    "count": 1,
    "next": null,
    "previous": null,
    "results": [
        {
            "creation_hooks": [
                {
                    "function": "add_perms_to_distribution_group",
                    "parameters": {
                        "add_user_to_group": true,
                        "group_type": "owners"
                    },
                    "permissions": [
                        "container.view_containerpushrepository",
                        "container.modify_content_containerpushrepository",
                        "container.change_containerpushrepository"
                    ]
                },
                {
                    "function": "add_perms_to_distribution_group",
                    "parameters": {
                        "add_user_to_group": false,
                        "group_type": "collaborators"
                    },
                    "permissions": [
                        "container.view_containerpushrepository",
                        "container.modify_content_containerpushrepository",
                        "container.change_containerpushrepository"
                    ]
                },
                {
                    "function": "add_perms_to_distribution_group",
                    "parameters": {
                        "add_user_to_group": false,
                        "group_type": "consumers"
                    },
                    "permissions": [
                        "container.view_containerpushrepository"
                    ]
                }
            ],
            "customized": false,
            "permissions_assignment": [
                {
                    "function": "add_perms_to_distribution_group",
                    "parameters": {
                        "add_user_to_group": true,
                        "group_type": "owners"
                    },
                    "permissions": [
                        "container.view_containerpushrepository",
                        "container.modify_content_containerpushrepository",
                        "container.change_containerpushrepository"
                    ]
                },
                {
                    "function": "add_perms_to_distribution_group",
                    "parameters": {
                        "add_user_to_group": false,
                        "group_type": "collaborators"
                    },
                    "permissions": [
                        "container.view_containerpushrepository",
                        "container.modify_content_containerpushrepository",
                        "container.change_containerpushrepository"
                    ]
                },
                {
                    "function": "add_perms_to_distribution_group",
                    "parameters": {
                        "add_user_to_group": false,
                        "group_type": "consumers"
                    },
                    "permissions": [
                        "container.view_containerpushrepository"
                    ]
                }
            ],
            "pulp_created": "2022-10-31T17:21:58.412317Z",
            "pulp_href": "/pulp/api/v3/access_policies/219b492a-1334-4f68-b0a7-2244beb205ac/",
            "queryset_scoping": {
                "function": "get_push_repos_qs",
                "parameters": {
                    "dist_perm": "container.view_containerdistribution",
                    "ns_perm": "container.view_containernamespace"
                }
            },
            "statements": [
                {
                    "action": [
                        "list"
                    ],
                    "effect": "allow",
                    "principal": "authenticated"
                },
                {
                    "action": [
                        "retrieve"
                    ],
                    "condition_expression": [
                        "has_namespace_obj_perms:container.namespace_view_containerpush_repository or has_distribution_perms:container.view_containerdistribution"
                    ],
                    "effect": "allow",
                    "principal": "authenticated"
                },
                {
                    "action": [
                        "tag",
                        "untag",
                        "remove_image",
                        "sign",
                        "remove_signatures"
                    ],
                    "condition_expression": [
                        "has_namespace_obj_perms:container.namespace_modify_content_containerpushrepository or has_distribution_perms:container.modify_content_containerpushrepository"
                    ],
                    "effect": "allow",
                    "principal": "authenticated"
                },
                {
                    "action": [
                        "update",
                        "partial_update"
                    ],
                    "condition_expression": [
                        "has_namespace_obj_perms:container.namespace_change_containerpushrepository or has_distribution_perms:container.change_containerdistribution"
                    ],
                    "effect": "allow",
                    "principal": "authenticated"
                }
            ],
            "viewset_name": "repositories/container/container-push"
        }
    ]
}


(pulp) [vagrant@pulp3-source-fedora36 nginx]$ 

```



Discussion:
Here are the steps that can be taken via API ( or via CLI, it is also available) for the deployments that have uncustomized access policy.

1. Find the href for the uncustomized container push viewset access policy
```python
 http https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/access_policies/?viewset_name=repositories/container/container-push  --auth admin:password |  jq -r '.results[] | .customized'
false

 http https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/access_policies/?viewset_name=repositories/container/container-push  --auth admin:password |  jq -r '.results[] | .pulp_href'
/pulp/api/v3/access_policies/219b492a-1334-4f68-b0a7-2244beb205ac/
```
2. reset the access policy
```python
 http POST https://PULP3-SOURCE-FEDORA36.puffy.example.com/pulp/api/v3/access_policies/219b492a-1334-4f68-b0a7-2244beb205ac/reset/  --auth admin:password
```
3. restart services
4. re-try push operation. it should succeed.
There's a shortcut for resetting an access policy in the cli:
`pulp access-policy reset --help`.

Resolving patch:
diff --git a/pulpcore/app/apps.py b/pulpcore/app/apps.py
--- a/pulpcore/app/apps.py
+++ b/pulpcore/app/apps.py
@@ -250,9 +250,10 @@ def _populate_access_policies(sender, apps, verbosity, **kwargs):
                                 viewset_name=viewset_name
                             )
                         )
-                if not created and not db_access_policy.customized:
+                elif not db_access_policy.customized:
                     dirty = False
-                    for key, value in access_policy.items():
+                    for key in ["statements", "creation_hooks", "queryset_scoping"]:
+                        value = access_policy.get(key)
                         if getattr(db_access_policy, key, None) != value:
                             setattr(db_access_policy, key, value)
                             dirty = True


Test patch: