Repository: ckan/ckan

Issue:
package_update performance of repeated calls
**CKAN version**
all

**Describe the bug**
Repeated calls to `package_update` with a set of resources w/o consistent resource ids get slower over time

**Steps to reproduce**
For this test I used a dataset with 10 resources repeatedly submitted to `package_update`:
```json
{
  "name": "10res",
  "resources": [
    {"url": "1"}, {"url": "2"}, {"url": "3"}, {"url": "4"}, {"url": "5"},
    {"url": "6"}, {"url": "7"}, {"url": "8"}, {"url": "9"}, {"url": "10"}
  ]
}
```
![package_update](https://user-images.githubusercontent.com/153258/193670739-55a6aea5-a5b9-4fca-9b86-624a02e61a6e.png)

- orig: time to complete on original ckan source
- filter: time to complete with filter applied (see below)
- purge: time to complete with resources purged (see below)

**Expected behavior**
ckan shouldn't get slower over time as resources are updated.

**Additional details**
filter patch:
This targets only the source of the slowness (iterating over all the deleted resources in python and resubmitting them with `state='deleted'` to postgres) and maintains the ever-growing list of deleted resources in the db
```patch
diff --git a/ckan/lib/dictization/model_save.py b/ckan/lib/dictization/model_save.py
index f4eff4427..c9cba0cc2 100644
--- a/ckan/lib/dictization/model_save.py
+++ b/ckan/lib/dictization/model_save.py
@@ -66,8 +66,12 @@ def package_resource_list_save(
     if res_dicts is None and allow_partial_update:
         return
 
+    session = context['session']
+    model = context['model']
     resource_list = package.resources_all
-    old_list = package.resources_all[:]
+    old_list = session.query(model.Resource) \
+        .filter(model.Resource.package_id == package.id) \
+        .filter(model.Resource.state != 'deleted')[:]
 
     obj_list = []
     for res_dict in res_dicts or []:
```

purge patch:
This solves the issue by purging resources that are removed in a package_update. This seems like a better solution because I can't think of any value in keeping these deleted resources in the db
```patch
diff --git a/ckan/lib/dictization/model_save.py b/ckan/lib/dictization/model_save.py
index f4eff4427..080e13ecc 100644
--- a/ckan/lib/dictization/model_save.py
+++ b/ckan/lib/dictization/model_save.py
@@ -89,8 +89,7 @@ def package_resource_list_save(
 
     # Mark any left-over resources as deleted
     for resource in set(old_list) - set(obj_list):
-        resource.state = 'deleted'
-        resource_list.append(resource)
+        resource.purge()
 
 
 def package_extras_save(
```


Discussion:

Resolving patch:
diff --git a/ckan/lib/dictization/model_save.py b/ckan/lib/dictization/model_save.py
--- a/ckan/lib/dictization/model_save.py
+++ b/ckan/lib/dictization/model_save.py
@@ -66,8 +66,20 @@ def package_resource_list_save(
     if res_dicts is None and allow_partial_update:
         return
 
+    session = context['session']
+    model = context['model']
     resource_list = package.resources_all
-    old_list = package.resources_all[:]
+    # existing resources not marked as deleted - when removed these
+    # need to be kept in the db marked as deleted so that extensions like
+    # datastore have a chance to remove tables created for those resources
+    old_list = session.query(model.Resource) \
+        .filter(model.Resource.package_id == package.id) \
+        .filter(model.Resource.state != 'deleted')[:]
+    # resources previously deleted can be removed permanently as part
+    # of this update
+    deleted_list = session.query(model.Resource) \
+        .filter(model.Resource.package_id == package.id) \
+        .filter(model.Resource.state == 'deleted')[:]
 
     obj_list = []
     for res_dict in res_dicts or []:
@@ -87,6 +99,9 @@ def package_resource_list_save(
     # according to their ordering in the obj_list.
     resource_list[:] = obj_list
 
+    # Permanently remove old deleted resources
+    for resource in set(deleted_list) - set(obj_list):
+        resource.purge()
     # Mark any left-over resources as deleted
     for resource in set(old_list) - set(obj_list):
         resource.state = 'deleted'


Test patch: