Bug 11096: add regression tests for indexing/retrieving large MARC records

This adds direct tests of indexing and retrieving
bib records that are larger than the ISO2709 limit
of 99,999 octets.

To test: verify that prove -v t/db_dependent/Search.t passes.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11096: Wrap each test set configuration in Search.t into a subtest

This makes it easier to read. Also fixes the test count, and is rebased
against 11499 which is needed to correctly test the patches for this bug
(i.e. indexing mode needs to be correctly set in koha-conf.xml)

To test:

[1] Run prove -v t/db_dependent/Search.t

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11096: support the retrieval of large MARCXML records

This patch makes Koha <-> Zebra use MARCXML for the serialization when
using DOM, and USMARC for GRS-1.

* The following functions are modified to set the Zebra record syntax
according to the current sysprefs and configuration:

- C4::Context->Zconn
- C4::Context-_new_Zconn

* A new function 'new_record_from_zebra' is introduced, which checks the
context we are in, and creates the MARC::Record object using the right
constructor.

The following packages get touched to make use of the new function:
- C4::Search
- C4::AuthoritiesMarc

and the same happens to the UI scripts that make use of them (both in
the OPAC and STAFF interfaces).

* Calls to the unsafe ZOOM::Record->render()[1] method are removed.

Due to this last change the code for building facets was rewritten. And
for performance on the facets creation I pushed higher version
dependencies for MARC::File::XML and MARC::Record (we rely on
MARC::Field->as_string).

* Calls to MARC::Record->new_from_xml and MARC::Record->new_from_usmarc
are wrapped with eval for catching problems [2].

* As of bug 3087, UNIMARC uses the 'unimarc' record syntax. this case is
  correctly handled.
* As of bug 7818 misc/migration_tools/rebuild_zebra.pl behaves like:

- bib_index_mode (defaults to 'grs1' if not specified)
- auth_index_mode (defaults to 'dom')

here we do exactly the same.

To test:
 - prove t/db_dependent/Search.t should pass.
 - Searching should remain functional.
 - Indexing and searching for a big record should work (that's what the
   unit tests do).
 - Test an index scan search (on the staff interface):
    Search > More options > Check "Scan indexes".
 - Enable 'itemBarcodeFallbackSearch' and try to circulate any word, it
   shouldn't break.
 - Searching for a biblio in a new subscription shouldn't break.
 - Running bulkmarcimport.pl shouldn't break.
 - And so on... for the rest of the .pl files.

[1] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#render()
[2] a record that cannot be parsed by MARC::Record is simply skipped (bug 10684)

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11078: Add locking to rebuild_zebra

This patch adds locking to rebuild_zebra.pl to ensure that simultaneous
changes are prevented (as one is likely to overwrite the other).
Incremental updates in daemon mode will skipped if the lock is busy
and they will be picked up on the next pass.  Non-daemon mode
invocations will also exit immediately if they cannot get the lock
unless the new flag -wait-for-lock is specified, in which case they
will wait until the get the lock and then proceed.

Supporting changes made to Makefile.PL and templates for the new
locking directory (paralleling the other zebra lock directories).
We stash the zebra_lockdir in koha-conf.xml so rebuild_zebra.pl
can find it.

To address earlier QA concerns we:
1. added code to check if flock is available and ignore locking if
it's missing (from M. de Rooy)

2. changed default for adhoc invocations to abort if they cannot
obtain the lock.  Added option -wait-for-lock if the user prefers
to wait until the lock is free, and then continue processing.

3. added missing entry to t/db_dependent/zebra_config.pl

4. added a fallback locking directory of /tmp

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Doug merged the original patch with the QA changes.
Just for the record, noting here that the original patch was tested
extensively too by Martin Renvoize.
I have added a followup for some exceptional cases.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11078: QA Follow-up for missing file permissions on lockfile

The original patch creates a lockfile in the ZEBRA_LOCKDIR.
It can fall back to /var/lock or even /tmp.
If the create fails, it dies. This can be considered as very
exceptional.

This followup adjusts the fallback location in /var/lock or /tmp
slightly.  It appends the database name to the folder in order to
prevent interfering between multiple Koha instances. Creation of the
lockfile has been moved to a subroutine extending directory and file
creation testing.

In the very unlikely case that we cannot create the lockfile (after
three separate tries), this follow-up allows you to continue instead
of die.  This is just as we did before we had file locking here. Every
time skipping a reindex could cause more harm than continuing and
having the race condition once in a while.

Test plan:
Test adding and removing lockdir from your koha-conf.xml. Check fallback.
Note that fallback in /var/lock or /tmp must contain database name.
Remove the lockdir config line and remove permissions from fallback. In
this case the reindex should continue but with a warning.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested with daemon and one-off invocation simultaneously.
Tested new wait parameter.
Tried all variations of lock directory (changing permissions etc.)

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11078: (follow-up) tidy code

- fix a couple typos in comments
- make replace a "$i" with a more descriptive variable name
- style some of the new code

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 7234: koha-common Debian package now includes koha-sites.conf

This patch adds a sample koha-sites.conf file, which will be
included by the koha-common package in /etc/koha (where it belongs).

It is fixed to include the current defaults (DOM indexing for both
biblios and authorities, disabled use of memcached, and good defaults
for a normal setup if enabled).

All options are commented for ease of use by average users.

To test
- verify that the defaults are sane (let me know if it needs to be
  fixed).
- build the package and verify that the file gets installed.

Sponsored-by: Universidad Nacional de Cordoba

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Robin Sheat <robin@catalyst.net.nz>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11866: do not require special permissions to view course reserves in staff interface

A librarian with no course reserves permissions cannot view course
reserves from the staff intranet, yet can from the OPAC. This doesn't
make much sense. Librarians should not require any course reserves
permissions to view courses and reserves from the staff intranet.

Test Plan:
1) Log into staff intranet as a librarian with no course reserves
   permissions
2) Note you cannot view course reserves
3) Apply this patch
4) Note you can now view course reserves
5) Verify you cannot modify courses or course reserves

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Bug 11866 [Follow-up] Staff side course reserves too restrictive

This follow-up patch adds a check for the "UseCourseReserves" system
preference to the display of the Course reserves menu item in the
header.

To test, view the "More" menu with the "UseCourseReserves" system
preference on and off. The menu item should appear and disappear
accordingly.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11866: (follow-up) ensure that empty toolbar is not displayed

If a user does not have course_reserve permission, a an empty toolbar is
displayed.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11866: (follow-up) repair security issue introduced by main patch

The main patch allows /any/ user with an account on the Koha system
to view the staff-side course-reserves home page -- including ordinary
patrons.  This patch repairs the oversight.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11802 - corrects for the patron purchase suggestions page

This patch corrects some errors with the staff client's patron purchase
suggestions page (members/purchase-suggestions.pl).

To test, apply the patch and make sure the patronimages system
preference is enabled.

- View the purchase suggestions page for a patron whose record has a
  patron image attached. The image should appear in the left-hand
  sidebar.

- View the page for a patron who has submitted no purchase suggestions.
  The message should be styled correctly.

- The toolbar on the page should look correct and work correctly.

Signed-off-by: Jesse Maseto <jesse@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11328 - Show local cover images in staff client search results

Local cover images do not appear in staff client search results. This
patch adds them.

To test, enable the LocalCoverImages system preference and add at least
one cover image to a title. Perform a search which will return results
that include your title. Confirm that the local cover image is
displayed.

Search results should also display correctly with and without
AmazonCoverImages enabled as well as with LocalCoverImages disabled.

Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11860: Display subscription callnumber in OPAC detail page

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Bug 11860: (follow-up) standardize spelling of "call number"

It is "call number", not "callnumber".

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11794 - restore functioning of opacSerialDefaultTab

The opacSerialDefaultTab system preference should control whether to
show the holdings or subscriptions tab by default on the detail page for
a serial. This behavior was broken when we last updated jQueryUI because
of a change in the jQueryUI active tab class name.

This patch corrects the class on the detail page and removes the old
class from the authorities detail page (where it was unnecessary
anyway).

To test, view a serial record in the OPAC under various settings of the
opacSerialDefaultTab preference. The default tab should correctly follow
the preference.

Test in prog and Bootstrap themes.

Edit: Rebased on current master. Note that the change to
opac-auth-detail.tt is because the page never has more than one tab,
hence doesn't need the "active" designation.

Signed-off-by: Jesse Maseto <jesse@bywatersolutions.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tested in prog and bootstrap.
All tests and QA script pass.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 10558 - Convert records table in manage-marc-import.pl to Ajax DataTable

Some libraries would like to sort by columns for the records of an
import batch. This seems like a good use of Ajax DataTables.

Test plan:
1) Apply this patch
2) Import a record batch into Koha
   a) Use some form of matching
   b) Have some records that will match and some that won't
   c) Have at least 30 records so you can test the pager
3) Verify the new table is functionally equivalent to the old static one

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Tests fine and looks good with the exception of the corrections I put in
a follow-up.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 10558: (follow-up) styling improvements

This patch makes improvements to the default DataTables CSS and corrects
a couple of errors in the original patch. CSS for the "full numbers"
type pagination, used by this and the quotes editor, has been cleaned up
and made more visually consistent with the brief pagination style.

Also changed:

- removed some hard-coded paths containing "http://staff.kohadev..."
- Added a column header to the column showing record match details. This
  provides both information and a clearer target for clicking to resort
- Expanding the default table pager controls for this page and the
  quotes editor (for consistency)
- Correction to quotes.css to fix pager display problem on wide screens

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 10558 [QA Follow-up]

This patch addresses a number of issues with the main patch:

- regression on bug 2060 (i.e., displaying authority import batches
  correctly)
- regression on bug 10170 (translation of import record states)A
- use of datatables.inc
- lack of clarity as to the licensing of tools/batch_records_ajax.pl
- insufficent sanitizing of input used to generate an SQL statement

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 10558: (follow-up) choose label for matching records correctly

This patch ensures that the record type of the import batch (i.e.,
biblio or authority) is consulted in order to choose the "Matches
biblio" or "Matches authority" labels.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11587 - IsSuperLibrarian generates warnings (UT)

Simply viewing OPAC detail triggers a modulus warning entry.
This first patch adds two test cases to t/Context.t to test for
this situation.

TEST PLAN
---------
1) Apply this patch (to upgrade t/Context.t)
2) prove -v t/Context.t
-- Tests 7 and 8 will fail
3) Apply main patch (to amend C4/Context.pm)
4) prove -v t/Context.t
-- All tests will succeed

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>