Circulation Rules Interface and Backend Revamp RFC

== Rationale ==
The current circulation rules system in Koha has a few major issues:
* The variety of limits and policies that can be set for each situation means that the rules editor scrolls horizontally on the widest of screens.
* Often, a library wants to set a general policy for most situations and only override certain values (e.g., holds policy is the same for all item types, fines policy varies by branch).
* The specificity rules (by specific/generic itemtype, branch and categorycode) are difficult to remember, and are not clearly shown in the interface. This has led to frequent confusion; see 8137 or 8366.
== Proposed Solution ==
We propose solving these issues with a new, more flexible backend and a more intuitive interface on top.
=== Backend ===
To allow for more flexible specification of policy, a new table structure will be created. Instead of a single row per itemtype/branchcode/categorycode with columns for each policy value, we’ll use the following:
branchcode 	categorycode 	itemtype 	rule_name 	rule_value
Branch/itemtype/category, or NULL for all.	Branch/itemtype/category, or NULL for all.	Branch/itemtype/category, or NULL for all.	loanlength, fine, etc.	14, 0.05, etc.
This means that the set of rules for a given situation may be spread across many different rows with different specificity. For instance, the rules for MPL/ADULT/BOOK may be spread across MPL/*/*, */ADULT/BOOK, etc. To deal with this, we will add a new API:
Right now, Koha selects groups of rules by the combination of branchcode, categorycode and itemtype. So, given these three criteria, we currently return all rules (loan length, fines, etc). The subroutine that does this is named GetIssuingRule.
The first iteration of the new solution will recreate this style of rule selection by fetching each rule separately, then combining them together and returning the result such that the code that requests the rules for a given combination will not know the underlying schema has changed at all. 
The second iteration will identify all code calling the subroutine that fetches circulation rules, and instead of calling GetIssuingRule will fetch only the circulation rules it needs. Once this is complete, the subroutine GetIssuingRule will be removed altogether.
To migrate data to this new system, we will create a row for each column of each row of the current schema. That is, each row in issuingrules will become ~20 rows in the new table (for loanlength, fine, etc.). This is less than ideal, but we propose mitigation mechanisms below.
=== Frontend ===
Basic prototype:
This interface is intended to clearly show the specificity order and show only the rules that have been defined and are relevant.
Notes:
* While the rules are specified in a table, much like the current interface, only the columns that have defined rules are actually shown. Others can be added using the "Choose rule..." dropdown. (This may be unneeded for the per-category and per-itemtype rules.)
* Any cell whose value is inherited from a set of defaults is shown in italics. The user can specify a value by clicking on the cell.
Planned enhancements:
* A dropdown to jump to a specified library.
* A link next to each inherited value that takes you to the original definition.
* A button next to each set value allowing the user to reset it to the inherited value.
* Floating headers (like the system preferences editor), so the user knows which library they're currently editing rules for.
* Help text, similar to the current UI.
* A "Save" button.
Also, note the "Cleanup rules..." button. This would bring up an interface that allowed a librarian to deduplicate rules (if several itemtypes had the same loan length, for example, it would suggest copying that loan length to a default rule). As each existing row in the issuingrules table will create a large number of rules, this will be critical for making the new interface usable.
===== Overview =====
To give librarians a very quick way to ascertain what rule will be used for a given library / patron / item combination, a rules overview page will be added which will display a table of combinations, the resultant rule values, what rule the value came from, and a link to quickly get to that rule in the editor.
In this mockup we have only 3 libraries, 2 patron types, and 2 item types defined:
Mockup image
The simplify this a bit more, we could retain the library selector and show the rules per library rather than one encompassing table.
== Excerpted Notes from Koha General IRC Meeting 13 January 2016 ==
 15:35 < cait> I think i am slightly in favour of a more table like thing ot be honest
 15:36 < cait> but that's me
 15:36 < khall> cait: we already have a table like thing ; ) The primary goal is to move to a more flexible approach that allows for 
 simpler rule sets
 15:36 < cait> I'd like something where I can easily compare rules for different itemtypes or different categories
 15:36 < khall> this approach doesn't jive all that well with a table based approach
 15:36 < Joubu> I think we need oleonard :)
 15:36 < khall> cait: that's already on my list!
 15:37 < cait> i am not sure how that would work with the mock up
 15:37 < khall> I think a second screen that gives a matrix of types categories and libraries with what the rule will be ( and where 
 that rule is coming from ) would be a good companion
 15:37 < andreashm> I have read the RFC... I'm a bit concerned that having a lot of issuing rules will make this new appoach hard to 
 overview
 15:38 < andreashm> khall: that seems like a good approach
 15:38 < khall> andreashm: that is what the second screen I propose will take care of
 15:38 < khall> thanks!
 15:38 < barton> cait -- how would you feel about some sort of explicit 'compare circ rules' functionality?
 15:38 < cait> khall: could youmaybe mock up the second screen too?
 15:39 < cait> to get a better idea?
 15:39 < hbraum> my consortia is sponsoring this development, although i'm very much letting pianohacker and khall take the reins on 
 this. we've currently got 1200 issuingrule lines, so they have a good test case for lots of rules already.
 15:39 * andreashm would like that too
 15:39 < khall> cait: yes, I can do that
 15:39 < andreashm> hbraum: ouch
 15:39 < hbraum> and looking at the current mockup, a second screen overview would be helpful, I agree.
 15:39 < cait> #action khall asked nicely to add another mock-up for an overview kind of page
 15:39 <@bag> yeah 1200 lines is crazy 
 15:40 < khall> The first step is to change the internals without changing the externals. If anyone has an issue with the proposed
 schema please let us know asap
 15:40 < cait> it will be a bit harder to get a nice excel sheet out :)
 15:40 < cait> that's what we do now - because you can make it sortable
 15:40 < cait> if there is no longer a need for it... that might be not too bad
 15:40 < khall> this would be a table too, so very sortable
 15:41 <@rangi> first thing id do is create a script where you can pass a itemtype, categorycode and branch, and get back what the rule 
 applied would be
 15:41 <@rangi> then you can change the internals
 15:41 <@rangi> run that again
 15:41 <@rangi> and prove it hasn't suddenly changed
 15:41 < khall> rangi: agreed, unit tests ++
 15:42 < barton> khall: I've got most of that written.
 15:42 <@rangi> that script could then have a web frontend put on it later, so librarians could choose from a drop down, and see what 
 rule would be applied .. rather than having to read everything
 15:42 <@rangi> why was this book issued for 4 weeks .. oh it used this rule
 15:42 < khall> rangi: you're reading my mind ; ) We've discussed that idea
 15:42 < cait> #idea have a script that gives you the circ conditions for branch, itemtype and patron category entered
 15:42 <@rangi> 70% of support queuries gone :)
 15:42 < khall> I think that should definitely be built in
 15:42 <@bag> more time to signoff!!!
 15:43 < Joubu> bag: heh, we have found what you wanted
 15:43 < hbraum> rangi ha! no kidding.
 15:43 < barton> In design discussions, we talked about making the lookup available on the circ page too, just for reference.
 15:43 < cait> hbraum: nekls++ btw for sponsoring - much needed
 15:44 < cait> for libraries with only 150 and less rules too :)
 15:44 < cait> ok, any more ideas or can we move on?
 15:44 < cait> full agenda tonight, but i don't want to rush youtoo much :)
 15:44 < khall> going once
 15:44 < hbraum> yw! the overall development scope of the project will also help branch specific circ-related rules get applied through
 this, right khall?
 15:45 < khall> hbraum: yes!
 15:45 < khall> it will hopefully be a grand unification theory for Koha circ rules
Category:RFCs