Bug 11717 - Use new DataTables include in member templates

Bug 10649 introduced a new include file for adding DataTables-related
JavaScript assets. This patch adds use of this include file to all
member pages which use DataTables.

Apply the patch and test the following pages to confirm that table
sorting works correctly:

- Patron details (members/moremember.pl)
  - The checkouts, relatives' checkouts, and holds tables have been
    modified to exclude articles when sorting of titles.

- Notices (members/notices.pl)
- Circulation history (members/readingrec.pl)
- Statistics (members/statistics.pl)
- Update child to adult patron (members/update-child.pl) - This is the
  page which appears in a pop-up when you update a child patron to an
  adult (from the "More" menu in the toolbar when viewing patron
  details).

Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11710 - Use new DataTables include in cataloging Z39.50 search template

This patch modifies the template for the cataloging Z39.50 search
template to use the new DataTables include. This simplifies the
inclusion of assets and updates to the latest DataTables version.

To test, go to cataloging and choose "New from Z39.50." Perform a search
and confirm that table sorting still works correctly.

Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11716 - Use new DataTables include in label manage template

This patch modifies the template for the label manage pages to use
the new DataTables include. This simplifies the inclusion of assets and
updates to the latest DataTables version.

To test, view the following pages and confirm that table sorting is
working correctly:

Manage layouts (/cgi-bin/koha/labels/label-manage.pl?label_element=layout)
Manage templates (/cgi-bin/koha/labels/label-manage.pl?label_element=template)
Manage profiles (/cgi-bin/koha/labels/label-manage.pl?label_element=profile)
Manage batches (/cgi-bin/koha/labels/label-manage.pl?label_element=batch)

Signed-off-by: wajasu <matted-34813@mypacks.net>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11736 - Use new DataTables include in Koha news templates

Bug 10649 introduced a new include file for adding DataTables-related
JavaScript assets. This patch adds use of this include file to the Koha
news page.

To test you should have existing news items with varying creation and
expiration dates. Apply the patch and confirm that table sorting works
correctly for all settings of the dateformat system preference.

C4::NewsChannels.pm has been modified so that it now passes an
unformatted date to the template, where the KohaDates plugin is used to
apply the correct formatting. Sorting is based on the unformatted date.

Also corrected: Capitalization errors.

Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described, no problems found.
Also passes tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11805 - Use validation plugin when creating new basket in Acquisitions

The page for adding a new basket in Acquisitions includes some custom
form validation JavaScript which can be removed in favor of HTML5
validation attributes and Koha's built-in validation plugin. This patch
does so.

To test, apply the patch and go to Acquisitions -> Choose a vendor ->
New basket. Try submitting the form without entering a basket name. This
should trigger a validation warning.

Submission of the form with valid data should work correctly. Editing an
existing basket should also work correctly.

Patch works as expected.
Signed-off-by: Marc Véron <veron@veron.ch>

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as expected, passes all tests and QA script.
Tried adding a new basket with/without basket name and editing
an existing basket editing/emptying the basket name.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11829: fix JavaScript error on staff client biblio details pages if template variables are undefined

The JavaScript included in cat-toolbar.inc expects several variables to
be passed to the template for use in calculations. Several scripts
do not pass the required variables to the template, leading to a JS
error.

This patch re-adds (See Bug 9823) hold count calculation to the required
scripts and copies the acquisitions-related count calculation from
detail.pl into related bibliographic detail scripts so that the
commonly-included cat-toolbar.inc can receive correct values.

To test, locate a bibliographic record which has no holds and is not
used in a current or deleted order, but which has a local cover image
attached. Check the following views to confirm there is no JavaScript
error:

- Normal
- MARC
- Labeled MARC
- ISBD
- Items
- Images (from the Images tab on the "Normal" detail page)

Next, locate a bibliographic record with no items attached which is used
in an Acquisitions order. On each of the pages listed above, choose
"Delete record" from the Edit menu. This should trigger a warning that
the record is used in an order. Cancel the deletion.

Finally, delete the order which references the bibliographic record. On
each of the pages listed above, choose "Delete record" again. This
should trigger a warning that the record is used in a deleted order.
Cancel the deletion.

I went through the test plan, everything works OK.
It resolves Bug 11831 as well.

Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
- Printing from MARC and ISBD view works again
- No more Javascript errors found
- Deleting a record with attached order gives the correct warning,
  both for ordered and cancelled.

Passes all tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11798: fix problems with column sorting and row visibility in invoices table

1) Sorting on "Billing date" (functionality introduced by bug 10794)
doesn't work properly any longer after bug 10401 was pushed (another
side-effect being that "Vendor" column is currently not sortable
at all).
2) Only "top ten" invoices are always displayed in the table, and it's
not possible to change that without missing top/bottom DataTables
pagination bars.

This patch fixes sorting for "Billing date" &  "Vendor" columns
in invoices table and restores the ability to see all table
rows/search results (not just "top ten").

Test plan (you'll need to have 11+ invoices with various billing
dates entered in the system):
1) apply patch,
2) view the Acquisitions invoices page (acqui/invoices.pl), do some
search (= as broad search as possible) and confirm that "Billing
date" and "Vendor" columns now do sort correctly,
3) in situations when there are more than 10 search results, they
should all be visible in the invoices table.

NOTE: Generating invoices is a painfully convoluted mess.
      All my "Billing date" entries were blank, so was not
      able to confirm Billing date issues. However, Vendor
      sorting issues, seeing more than 10, and in my bizarre
      initial case the non-visible checkboxes were all fixed.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11096: add regression tests for indexing/retrieving large MARC records

This adds direct tests of indexing and retrieving
bib records that are larger than the ISO2709 limit
of 99,999 octets.

To test: verify that prove -v t/db_dependent/Search.t passes.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11096: Wrap each test set configuration in Search.t into a subtest

This makes it easier to read. Also fixes the test count, and is rebased
against 11499 which is needed to correctly test the patches for this bug
(i.e. indexing mode needs to be correctly set in koha-conf.xml)

To test:

[1] Run prove -v t/db_dependent/Search.t

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11096: support the retrieval of large MARCXML records

This patch makes Koha <-> Zebra use MARCXML for the serialization when
using DOM, and USMARC for GRS-1.

* The following functions are modified to set the Zebra record syntax
according to the current sysprefs and configuration:

- C4::Context->Zconn
- C4::Context-_new_Zconn

* A new function 'new_record_from_zebra' is introduced, which checks the
context we are in, and creates the MARC::Record object using the right
constructor.

The following packages get touched to make use of the new function:
- C4::Search
- C4::AuthoritiesMarc

and the same happens to the UI scripts that make use of them (both in
the OPAC and STAFF interfaces).

* Calls to the unsafe ZOOM::Record->render()[1] method are removed.

Due to this last change the code for building facets was rewritten. And
for performance on the facets creation I pushed higher version
dependencies for MARC::File::XML and MARC::Record (we rely on
MARC::Field->as_string).

* Calls to MARC::Record->new_from_xml and MARC::Record->new_from_usmarc
are wrapped with eval for catching problems [2].

* As of bug 3087, UNIMARC uses the 'unimarc' record syntax. this case is
  correctly handled.
* As of bug 7818 misc/migration_tools/rebuild_zebra.pl behaves like:

- bib_index_mode (defaults to 'grs1' if not specified)
- auth_index_mode (defaults to 'dom')

here we do exactly the same.

To test:
 - prove t/db_dependent/Search.t should pass.
 - Searching should remain functional.
 - Indexing and searching for a big record should work (that's what the
   unit tests do).
 - Test an index scan search (on the staff interface):
    Search > More options > Check "Scan indexes".
 - Enable 'itemBarcodeFallbackSearch' and try to circulate any word, it
   shouldn't break.
 - Searching for a biblio in a new subscription shouldn't break.
 - Running bulkmarcimport.pl shouldn't break.
 - And so on... for the rest of the .pl files.

[1] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#render()
[2] a record that cannot be parsed by MARC::Record is simply skipped (bug 10684)

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11078: Add locking to rebuild_zebra

This patch adds locking to rebuild_zebra.pl to ensure that simultaneous
changes are prevented (as one is likely to overwrite the other).
Incremental updates in daemon mode will skipped if the lock is busy
and they will be picked up on the next pass.  Non-daemon mode
invocations will also exit immediately if they cannot get the lock
unless the new flag -wait-for-lock is specified, in which case they
will wait until the get the lock and then proceed.

Supporting changes made to Makefile.PL and templates for the new
locking directory (paralleling the other zebra lock directories).
We stash the zebra_lockdir in koha-conf.xml so rebuild_zebra.pl
can find it.

To address earlier QA concerns we:
1. added code to check if flock is available and ignore locking if
it's missing (from M. de Rooy)

2. changed default for adhoc invocations to abort if they cannot
obtain the lock.  Added option -wait-for-lock if the user prefers
to wait until the lock is free, and then continue processing.

3. added missing entry to t/db_dependent/zebra_config.pl

4. added a fallback locking directory of /tmp

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Doug merged the original patch with the QA changes.
Just for the record, noting here that the original patch was tested
extensively too by Martin Renvoize.
I have added a followup for some exceptional cases.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11078: QA Follow-up for missing file permissions on lockfile

The original patch creates a lockfile in the ZEBRA_LOCKDIR.
It can fall back to /var/lock or even /tmp.
If the create fails, it dies. This can be considered as very
exceptional.

This followup adjusts the fallback location in /var/lock or /tmp
slightly.  It appends the database name to the folder in order to
prevent interfering between multiple Koha instances. Creation of the
lockfile has been moved to a subroutine extending directory and file
creation testing.

In the very unlikely case that we cannot create the lockfile (after
three separate tries), this follow-up allows you to continue instead
of die.  This is just as we did before we had file locking here. Every
time skipping a reindex could cause more harm than continuing and
having the race condition once in a while.

Test plan:
Test adding and removing lockdir from your koha-conf.xml. Check fallback.
Note that fallback in /var/lock or /tmp must contain database name.
Remove the lockdir config line and remove permissions from fallback. In
this case the reindex should continue but with a warning.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested with daemon and one-off invocation simultaneously.
Tested new wait parameter.
Tried all variations of lock directory (changing permissions etc.)

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11078: (follow-up) tidy code

- fix a couple typos in comments
- make replace a "$i" with a more descriptive variable name
- style some of the new code

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 7234: koha-common Debian package now includes koha-sites.conf

This patch adds a sample koha-sites.conf file, which will be
included by the koha-common package in /etc/koha (where it belongs).

It is fixed to include the current defaults (DOM indexing for both
biblios and authorities, disabled use of memcached, and good defaults
for a normal setup if enabled).

All options are commented for ease of use by average users.

To test
- verify that the defaults are sane (let me know if it needs to be
  fixed).
- build the package and verify that the file gets installed.

Sponsored-by: Universidad Nacional de Cordoba

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Robin Sheat <robin@catalyst.net.nz>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11866: do not require special permissions to view course reserves in staff interface

A librarian with no course reserves permissions cannot view course
reserves from the staff intranet, yet can from the OPAC. This doesn't
make much sense. Librarians should not require any course reserves
permissions to view courses and reserves from the staff intranet.

Test Plan:
1) Log into staff intranet as a librarian with no course reserves
   permissions
2) Note you cannot view course reserves
3) Apply this patch
4) Note you can now view course reserves
5) Verify you cannot modify courses or course reserves

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Bug 11866 [Follow-up] Staff side course reserves too restrictive

This follow-up patch adds a check for the "UseCourseReserves" system
preference to the display of the Course reserves menu item in the
header.

To test, view the "More" menu with the "UseCourseReserves" system
preference on and off. The menu item should appear and disappear
accordingly.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11866: (follow-up) ensure that empty toolbar is not displayed

If a user does not have course_reserve permission, a an empty toolbar is
displayed.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11866: (follow-up) repair security issue introduced by main patch

The main patch allows /any/ user with an account on the Koha system
to view the staff-side course-reserves home page -- including ordinary
patrons.  This patch repairs the oversight.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11802 - corrects for the patron purchase suggestions page

This patch corrects some errors with the staff client's patron purchase
suggestions page (members/purchase-suggestions.pl).

To test, apply the patch and make sure the patronimages system
preference is enabled.

- View the purchase suggestions page for a patron whose record has a
  patron image attached. The image should appear in the left-hand
  sidebar.

- View the page for a patron who has submitted no purchase suggestions.
  The message should be styled correctly.

- The toolbar on the page should look correct and work correctly.

Signed-off-by: Jesse Maseto <jesse@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11328 - Show local cover images in staff client search results

Local cover images do not appear in staff client search results. This
patch adds them.

To test, enable the LocalCoverImages system preference and add at least
one cover image to a title. Perform a search which will return results
that include your title. Confirm that the local cover image is
displayed.

Search results should also display correctly with and without
AmazonCoverImages enabled as well as with LocalCoverImages disabled.

Signed-off-by: wajasu <matted-34813@mypacks.net>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11860: Display subscription callnumber in OPAC detail page

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>