DSpace/DSpace — pull request #661 by peterdietz (2014-09-26 13:30:39+00:00)
Title: DS-682 Optimize Select Collection Performance

If you have a site with greater than say 50 collections, there becomes a noticeable lag when waiting for the Select a Collection dropdown to load. We're talking like 10, 20, 30 seconds on large sites.

https://jira.duraspace.org/browse/DS-682
This would be an example slow page: http://localhost:8080/xmlui/submit

This work was sponsored by [The Ohio State University Libraries](http://library.osu.edu/)
![osu-universitylibraries](https://cloud.githubusercontent.com/assets/58014/4421119/ca6f5c9a-4580-11e4-92b9-88c48fe0550e.png)

The huge performance delay is due to Collection.findAuthorized() iterating through Collection.findAll(), and then using AuthorizationManager to ask if the current user has permission to view that Collection. So, the performance of this dropdown is O(n), where n is the number of collections. Also, it's making a SQL call per authorization lookup, so 1000 collections = 1000 queries. So, instead of refactoring the DSpace datamodel to use a graph database, we've come up with an approach of that queries for collections the user is authorized to ADD (submit) to from the opposite direction.
Slow: All collections, iterate each to find out which the user is authorized. O(n)
Optimized: One user, find their direct policy enabled collection, find their groups with policy, find group2group policies, find communities with policy and get child collections. About 5 queries. So: O(1)

Developers: If your laptop has an SSD, you might not realize accurate results, because your DB is so fast, if your production DB has slower IO you'll see much more significant gains.

I have two types of tests for this feature, one is the integration test, which I have about 5 scenarios that I'm checking, it passes. Feel free to suggest additional tests. I also have a standalone class that you call through dsrun (not a real test), but it checks all users in your DB, and compares the results of Collection.findAuthorized() with Collection.findAuthorizedOptimized to check that the new version has all the same collections, and gives you a caliper time benchmark that lets you know how much faster the optimized version is.

```
/dspace/bin/dspace dsrun org.dspace.app.util.OptimizeSelectCollection
OptimizeSelectCollection tool.
We want to verify that the optimized version of select collection logic produces the same values as the legacy select-collection logic.
User: personA@example.edu
...
...
...
User: personZ@example.edu
Legacy Find Authorized
====================================
This user is permitted to submit to the following collections.
- 123456789/35 -- SigPort Demonstration Collection
Total: 1
Find Authorized Optimized
====================================
This user is permitted to submit to the following collections.
- 123456789/35 -- SigPort Demonstration Collection
Total: 1
Number of collections matches - Good
StopWatch 'SelectCollectionStep Optimization (personZ@example.edu)': running time (millis) = 8569
-----------------------------------------
ms % Task name
-----------------------------------------
08242 096% findAuthorized
00000 000% ListingCollections
00327 004% findAuthorizedOptimized
00000 000% ListingCollectionsWithOptimizedCollections

All Good: 26 people have been checked, with same submission powers. TimeSaved(ms): 187027
```

So, if you check this on a clone of your production data, let me know that all corner cases have been covered. I've checked this against a number of instances already.

Full Test:
/dspace/bin/dspace dsrun org.dspace.app.util.OptimizeSelectCollection

Single User Test:
/dspace/bin/dspace dsrun org.dspace.app.util.OptimizeSelectCollection user@example.edu