Bug 7284: Authority matching improvements

Squashed patch incorporating all previous patches (there is no functional
change compared to the previous version of this patch, this patch merely
squashes the original patch and follow-up, and rebases on latest master).

=== TL;DR VERSION ===
*** Installation ***
1. Run installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt1
and installer/data/mysql/atomicupdate/bug_7284_authority_linking_pt2
2. Make sure you copy the following files from kohaclone to koha-dev:
etc/zeradb/authorities/etc/bib1.att,
etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml,
etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl,
etc/zebradb/marc_defs/marc21/authorities/koha-indexdefs-to-zebra.xsl, and
etc/zebradb/marc_defs/unimarc/authorities/record.abs
3. Run misc/migration_tools/rebuild_zebra.pl -a -r

*** New sysprefs ***
* AutoCreateAuthorities
* CatalogModuleRelink
* LinkerModule
* LinkerOptions
* LinkerRelink
* LinkerKeepStale

*** Important notes ***
You must have rebuild_zebra processing the zebraqueue for bibs when testing this
patch.

=== DESCRIPTION ===

*** Cataloging module ***
* Added an additional box to the authority finder plugin for "Heading match,"
  which consults not just the main entry but also See-from and See-also-from
  headings.

* With this patch, the automatic authority linking will actually work properly
  in the cataloging module. As Owen pointed out while testing the patch,
  though, longtime users of Koha will not be expecting that. In keeping with
  the principles of least surprise and maximum configurability, a new syspref,
  CatalogModuleRelink makes it possible to disable authority relinking in the
  cataloging module only (i.e. leaving it enabled for future runs of
  link_bibs_to_authorities.pl).  Note that though the default behavior matches
  the current behavior of Koha, it does not match the intended behavior.
  Libraries that want the intended behavior rather than the current behavior
  will need to adjust the CatalogModuleRelink syspref.

*** misc/link_bibs_to_authorities.pl ***
Added the following options to the misc/link_bibs_to_authorities.pl script:
--auth-limit        Only process those headings that match the authorities
                    matching the user-specified WHERE clause.
--bib-limit         Only process those bib records that match the
                    user-specified WHERE clause.
--commit            Commit the results to the database after every N records
                    are processed.
--link-report       Display a report of all the headings that were processed.

Converted misc/link_bibs_to_authorities.pl to use POD.

Added a detailed report of headings that linked, did not link, and linked
in a "fuzzy" fashion (the exact semantics of fuzzy are up to the individual
linker modules) during the run.

*** C4::Linker ***
Implemented new C4::Linker functionality to make it possible to easily add
custom authority linker algorithms. Currently available linker options are:
* Default: retains the current behavior of only creating links when there is
  an exact match to one and only one authority record; if the 'broader_headings'
  option is enabled, it will try to link to headings to authority records for
  broader headings by removing subfields from the end of the heading (NOTE:
  test the results before enabling broader_headings in a production system
  because its usefulness is very much dependent on individual sites' authority
  files)
* First Match: based on Default, creates a link to the *first* authority
  record that matches a given heading, even if there is more than one
  authority record that matches
* Last Match: based on Default, creates a link to the *last* authority
  record that matches a given heading, even if there is more than one record
  that matches

The API for linker modules is very simple. All modules should implement the
following two functions:
<get_link ($field)> - return the authid for the authority that should be
linked to the provided MARC::Field object, and a boolean to indicate whether
the match is "fuzzy" (the semantics of "fuzzy" are up to the individual plugin).
In order to handle authority limits, get_link should always end with:
    return $self->SUPER::_handle_auth_limit($authid), $fuzzy;

<flip_heading ($field)> - return a MARC::Field object with the heading flipped
to the preferred form. At present this routine is not used, and can be a stub.

Made the linking functionality use the SearchAuthorities in C4::AuthoritiesMarc
rather than SimpleSearch in C4::Search. Once C4::Search has been refactored,
SearchAuthorities should be rewritten to simply call into C4::Search. However,
at this time C4::Search cannot handle authority searching. Also fixed numerous
performance issues in SearchAuthorities and the Linker script:
* Correctly destroy ZOOM recordsets in SearchAuthorities when finished. If left
  undestroyed, efficiency appears to approach O(log n^n)
* Add an optional $skipmetadata flag to SearchAuthorities that can be used to
  avoid additional calls into Zebra when all that is wanted are authority
  records and not statistics about their use

*** New sysprefs ***
* AutoCreateAuthorities - When this and BiblioAddsAuthorities are both turned
  on, automatically create authority records for headings that don't have
  any authority link when cataloging. When BiblioAddsAuthorities is on and
  AutoCreateAuthorities is turned off, do not automatically generate authority
  records, but allow the user to enter headings that don't match an existing
  authority. When BiblioAddsAuthorities is off, this has no effect.
* CatalogModuleRelink - when turned on, the automatic linker will relink
  headings when a record is saved in the cataloging module when LinkerRelink
  is turned on, even if the headings were manually linked to a different
  authority by the cataloger. When turned off (the default), the automatic
  linker will not relink any headings that have already been linked when a
  record is saved.
* LinkerModule - Chooses which linker module to use for matching headings
  (current options are as described above in the section on linker options:
  "Default," "FirstMatch," and "LastMatch")
* LinkerOptions - A pipe-separated list of options to set for the authority
  linker (at the moment, the only option available is "broader_headings," which
  is described below)
* LinkerRelink - When turned on, the linker will confirm the links for headings
  that have previously been linked to an authority record when it runs. When
  turned off, any heading with an existing link will be ignored.
* LinkerKeepStale - When turned on, the linker will never *delete* a link to an
  authority record, though, depending on the value of LinkerRelink, it may
  change the link.

*** Other changes ***
* Cleaned up authorities code by removing unused functions and adding
  unimplemented functions and added some unit tests.

* This patch also modifies the authority indexing to remove trailing punctuation
  from Match indexes.

* Replace the old BiblioAddAuthorities subroutines with calls into the new
  C4::Linker routines.

* Add a simple implementation for C4::Heading::UNIMARC. (With thanks to F.
  Demians, 2011.01.09) Correct C4::Heading::UNIMARC class loading. Create
  biblio tag to authority types data structure at initialization rather than
  querying DB.

* Ran perltidy on all changed code.

*** Linker Options ***
Enter "broader_headings" in LinkerOptions. With this option, the linker will
try to match the following heading as follows:
=600  10$aCamins-Esakov, Jared$xCoin collections$vCatalogs$vEarly works to
1800.

First: Camins-Esakov, Jared--Coin collections--Catalogs--Early works to 1800
Next: Camins-Esakov, Jared--Coin collections--Catalogs
Next: Camins-Esakov, Jared--Coin collections
Next: Camins-Esakov, Jared (matches! if a previous attempt had matched, it
would not have tried this)

This is probably relevant only to MARC21 and LCSH, but could potentially be of
great use to libraries that make heavy use of floating subdivisions.

=== TESTING PLAN ===

Note: all of these tests require that you have some authority records,
preferably for headings that actually appear in your bibliographic data. At
least one authority record must contain a "see from" reference (remember which
one contains this, as you'll need it for some of the tests). The number shown
in the "Used in" column in the authority module is populated using Zebra
searches of the bibliographic database, so you *must* have
rebuild_zebra.pl -b -z [-x] running in cron, or manually run it after running
the linker.

*** Testing the Heading match in the cataloging plugin ***
1.  Create a new record, and open the cataloging plugin for an
    authority-controlled field.
2.  Search for an authority by entering the "see from" term in the Heading Match
    box
3.  Confirm that the appropriate heading shows up
4.  Search for an authority by entering the preferred heading into the Main
    entry or Main entry ($a only) box (i.e., repeat the procedure you usually
    use for cataloging, whatever that may be)
5.  Confirm that the appropriate heading shows up

*** Testing the cataloging interface ***
6.  Turn off BiblioAddsAuthorities
7.  Confirm that you cannot enter text directly in an authority-controlled field
8.  Confirm that if you search for a heading using the authority control plugin
    the heading is inserted (note, however, that this patch does not AND IS NOT
    INTENDED TO fix the bugs in the authority plugin with duplicate subfields;
    those are wholly out of scope- this check is for regressions)
9.  Turn on BiblioAddsAuthorities and AutoCreateAuthorities
10. Confirm that you can enter text directly into an authority-controlled field,
    and if you enter a heading that doesn't currently have an authority record,
    an authority record stub is automatically created, and the heading you
    entered linked
11. Confirm that if you enter a heading with only a subfield $a that fully
    *matches* an existing heading (i.e. the existing heading has only
    subfield $a populated), the authid for that heading is inserted into
    subfield $9
12. Confirm that if you enter a heading with multiple subfields that *matches*
    an existing heading, the authid for that heading is inserted into
    subfield $9
13. Turn on BiblioAddsAuthorities and turn off AutoCreateAuthorities
14. Confirm that you can enter text directly into an authority-controlled field,
    and if you enter a heading that doesn't currently have an authority record,
    an authority record stub is *not* created
15. Confirm that if you enter a heading with only a subfield $a that *matches*
    an existing heading, the authid for that heading is inserted into
    subfield $9
16. Confirm that if you enter a heading with multiple subfields that *matches*
    an existing heading, the authid for that heading is inserted into
    subfield $9
17. Create a record and link an authority record to an authorized field using
    the authority plugin.
18. Save the record. Ensure that the heading is linked to the appropriate
    authority.
19. Open the record. Change the heading manually to something else, leaving
    the link. Save the record.
20. Ensure that the heading remains linked to that same authority.
21. Change CatalogModuleRelink to "on."
22. Open the record. Use the authority plugin to link that heading to the
    same authority record you did earlier.
23. Save the record. Ensure that the heading is linked to the appropriate
    authority.
24. Open the record. Change the heading manually to something else, leaving
    the link. Save the record.
25. Ensure that the heading is no longer linked to the old authority record.

*** Testing link_bibs_to_authorities.pl ***
26. Set LinkerModule to "Default," turn on LinkerRelink and
    BiblioAddsAuthorities, and turn AutoCreateAuthorities and
    LinkerKeepStale off
27. Edit one bib record so that an authority controlled field that has already
    been linked (i.e. has data in $9) has a heading that does not match any
    authority record in your database
28. Run misc/link_bibs_to_authorities.pl --link-report --verbose --test (you may
    want to pipe the output into less or a file, as the result is quite a lot of
    information)
29. Look over the report to see if the headings that you have authority records
    for report being matched, that the heading you modified in step 2 is
    reported as "unlinked," and confirm that no changes were actually made to
    the database (to check this, look at the bib record you edited earlier, and
    check that the authid in the field you edited hasn't changed)
30. Run misc/link_bibs_to_authorities.pl --link-report --verbose (you may want
    to pipe the output into less or a file, as the result is quite a lot of
    information)
31. Check that the heading you modified has been unlinked
32. Change the modified heading back to whatever it was, but don't use the
    authority control plugin to populate $9
33. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
34. Confirm that the heading has been linked to the correct authority record
35. Turn LinkerKeepStale on
36. Change that heading to something else
37. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
38. Confirm that the $9 has not changed
39. Turn LinkerKeepStale off
40. Create two authorities with the same heading
41. Run misc/migration_tools/rebuild_zebra.pl -a -z
42. Enter that heading into the bibliographic record you are working with
43. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
44. Confirm that the heading has not been linked
45. Change LinkerModule to "FirstMatch"
46. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
47. Confirm that the heading has been linked to the first authority record it
    matches
48. Change LinkerModule to "LastMatch"
49. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --bib-limit="biblionumber=${BIB}" (replacing ${BIB} with the biblionumber
    of the record you've been editing)
50. Confirm that the heading has been linked to the second authority record it
    matches
51. Run misc/link_bibs_to_authorities.pl --link-report --verbose
    --auth-limit="authid=${AUTH}" (replacing ${AUTH} with an authid)
52. Confirm that only that heading is displayed in the report, and only those
    bibs with that heading have been changed

If all those things worked, good news! You're ready to sign off on the patch
for bug 7284.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master and squashed follow-up, 16 February 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on latest master, 21 February 2012

Signed-off-by: schuster <dschust1@gmail.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Tested with an UNIMARC setup that things work fine. They do

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Cherry-picked onto 3.6.x, 10 March 2012
Note for RMaint: this patch squashes all the bug 7284 patches from Master,
including the updatedatabase changes (but does not set the DB revision).

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
bug 7284: fix package build problem, move BEGIN into new

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7284 follow-up: Update DBRev

Version 3.06.04.001.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7493 [REVISED] Deleting a record with comments breaks "Recent Comments"

Adding foreign key constraint on biblio and borrowers:

- delete reviews when bibliographic records are deleted
- set set reviews.borrowernumber to NULL when patron records are deleted.

Before these constraints are set the database update script will
clean up existing instances of these problems by deleting reviews of
deleted biblios and setting to NULL reviews.borrowernumber for deleted
patrons.

In comments moderation, the list of comments will indicate "(deleted patron")
if the review was submitted by a patron who has since been deleted.

In the OPAC display of comments will omit the patron name altogether
if the patron has since been deleted.

To test:

1. CONFIRM THAT THE DATABASE UPDATE RUNS CORRECTLY

Before applying the patch:

 - delete the record for a title which has a review
 - delete a patron who has submitted a review (on a different title).

Viewing /cgi-bin/koha/opac-showreviews.pl should result in an error.

Apply the patch and make sure the database update runs. Viewing
the recent comments page in the OPAC should no longer result in an
error. The title with the comment from a deleted patron should
show no patron name on these pages:

 - Recent comments (opac-showreviews.pl)
 - Recent comments RSS (opac-showreviews.pl?format=rss)
 - Detail page (opac-detail.pl)

Comments from other patrons should show correctly.

2. CONFIRM THAT NEW DELETIONS PERFORM CORRECTLY

After applying the patch:

 - delete the record for a title which has a review
 - delete a patron who has submitted a review (on a different title).

Viewing /cgi-bin/koha/opac-showreviews.pl should NOT result in an error.
The review of the title which you deleted should no longer appear in the
list of recent comments.

The title with the comment from a deleted patron should
show no patron name on these pages:

 - Recent comments (opac-showreviews.pl)
 - Recent comments RSS (opac-showreviews.pl?format=rss)
 - Detail page (opac-detail.pl)

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7493 follow-up: Update DBRev

Version 3.06.04.002.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7271 - change sort in items.pm to sort by home library instead of holding branch

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7629 - Update sco.css to include CSS Gradients and prevent error upon missing background image file

Copying CSS gradients for input buttons and dialogs from opac.css. Also copied missing
CSS for language-picker footer. Markup corrected in a couple of places in order
for the right style to be applied.

To test corrections to dialog CSS, enter an invalid patron barcode.
Multiple languages will have to be enabled in order to see the footer
correction working.

Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Dialog CSS looks correct, footer also looks correct with multi languages.

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 6488: Display OPAC search result count conditional to OpacHiddenItems syspref

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Works as advertised. Amended this patch for a small spelling issue: supress->suppress. Could not suppress it ;)

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7631 - Self checkout renewal fails because of reference to non-existent subroutine in sco-main.pl

Changing "CanBookBeIssuedCheckout" to "CanBookBeIssued"

To test, try to renew an item which has no renewals left. Before the patch
you'll get an error:

Undefined subroutine &main::CanBookBeIssuedCheckout called at /opac/sco/sco-main.pl line 135.

After the patch you'll get the correct message about having no renewals left.
Other tests: checking out a barcode which doesn't exist, checking out an
item which is on hold for another patron.

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7645 - System preferences editor save button obscured by language chooser

This patch amends the preferences page's CSS file to add
bottom padding to the page container. This gives the bottom-
most submit button some breathing room.

Markup correction: Adding "row" and "col" attributes to
<textareas> to quiet validation errors. This doesn't
affect output at all (dimensions are specified in CSS).

To test, open the system preferences page to a tab like OPAC
which contains many preferences. At the bottom of the page
the last submit button should have about a line's worth of
white space below it.

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7636 - error when trying to email cart when opacuserlogin set to don't allow

This patch hides the "send cart" link if the opacuserlogin
preference is set to disallow. Other similar changes included:

- hide the "send list" link
- hide the "your lists" tab
- hide the "log in to create your own lists" link
- hide the "recent comments" link

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Tested with opacuserlogin set to both "Allow" and "Don't allow." With
opacuserlogin set to "Allow," the links showed up, and with opacuserlogin
set to "Don't allow," the links did not.

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7665 - Bibs with no ISBN's show broken images for covers when using Syndetics cover images

Patch fixes problem for syndetics + bibs with no ISBN. Also adds "no image found" capability to Syndetics results.

To test (first, contact me and I will let you use my syndetics credentials *for testing this bug only*):
* replicate bug - have a bib with an ISBN, and one without. The one with the ISBN should show a cover, the one without will show a broken image.
* apply patch
* search for the same bib - you should now see "no image available" for the one with no ISBN, and a cover image for your bib with an ISBN.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Ian Walls <koha.sekjal@gmail.com>
QA Comment:  normalized spacing

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 3216: UNIMARC author facets

Add 700$b to UNIMARC author facets.

Other facets subfields could be added now. For example, other subjects
subfields.

Following patches are required to handle better MARC21 subfields and choose
other subfields to deal with UNIMARC format.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Tested under both MARC21 and UNIMARC. Does not cause any regressions with
MARC21, and offers the possibility for better faceting there in the future.
Works as advertised with UNIMARC.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7609: Improving links to find analytics and volumes when using UseControlnumber (MARC21)

To test:
- switch on UseControlnumber
- add a serial record with 001
  - ldr pos 19 = a or
  - 008 (continuing resource), pos 21 = m
- add 2 analytics with with $w subfields containing the 001 of the serial
  - one with ldr pos 7 = a
  - one with ldr pos 7 = b
- add 1-2 monographical records linked to the serial
  - 8xx $w subfield containing the 001 of the serial
  - 490

The serial should show 2 links (in both OPAC and Intranet):
'Show volumes' should only bring up the monographical records.
'Show analytics' should only find the analytical records.

Without the patch both links would find all linked records, no difference
between both links.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7699: Restricted until datepicker broken

To test:
1. Edit a patron, and try to use the datepicker to set the date the patron is
   restricted until.
2. Note that the datepicker does not come up.
3. Apply patch.
4. Repeat step 1.
5. Note that the datepicker does come up.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 5749 Fix borrower address display in intranet

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7644 - Invalid markup in staff client language chooser

This patch removes invalid markup from the language chooser
in the intranet. It also copies the markup and style of
the staff client include to the OPAC so the two are
more consistent. I hope that will make future changes
and debugging a little easier.

I believe this patch also fixes Bug 7366, "Language chooser
display problems."

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Patch fixes bug 7366, but there is still a difference between staff
and OPAC.
Installed languages: en, en-NZ, de-DE, fr-FR
Activated langauges: en, en-NZ, fr-FR
OPC shows English, staff shows en-NZ as label for the list.

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
7368 Typo in cart_to_shelf

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Bug 7618 Escape HTML special characters for SQL displayed above results in Report module

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
welcome David Cook, you're 178th

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>