Bug 9114: Make frameworks import/export routines correctly use UTF-8

Currently both the import_export_framework.pl script outputs data with
Perl's default encoding, ISO-8859. This patch properly sets
the binmode to UTF-8 when exporting SQL and CSV files using the PerlIO
layer (":encoding(UTF-8)") for STDOUT.

To test:

Export step test
- Use some ASCII character(s) with DIACRITICS in some field description
  in a chosen framework.
- Export the framework at Administration > MARC frameworks
- Run this to check the file is ISO-8859 encoded:
 $ file export_XXX.csv
 export_XXX.csv: ISO-8859 text, with very long lines
 (Note: try SQL and other output formats too. But not ODS)
- Apply the patch
- Export the framework again (change the name), and test encoding:
 $ file export_XXX_2.csv
 export_XXX_2.csv: UTF-8 Unicode text

Import step test
I assume you have two files, export_XXX.csv (ISO-8859 encoded) and
export_XXX_2.csv (XXX will depend on your framework's code)
- Reset your testing branch to master
- Import export_XXX.csv
- The string with non-ASCII chars is truncated at the first non-ASCII
  char's position (Note: this is the current behaviour).
- Import export_XXX_2.csv
- The non-ASCII chars are broken, the logs show errors on non-UNICODE
  chars.  (Note: even thou UTF-8 is the expected encoding it is
  treated as ISO-8859).
- Apply the patch
- Import the good (UTF-8 as expected) file and check everything worked
  as expected.

No double encoding should occur with either combination of formats.

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Magnus Enger <digitalutvikling@gmail.com>
I put some Norwegian and accented letters in a fremawork to test.
Before the patch, the exported CSV came out as ISO-8859, after the
patch it came out as UTF-8. ODS and XML (viewed in LibreOffice)
both looked good, before and after the patch.

Importing the ISO-8859 CSV cut off the strings at the first non-ASCII
char. Importing the UTF-8 CSV worked as epected.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as expected, passes tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11403: (follow-up) Fix typo in renew help

This patch adds a missing letter to the Renew help file

To test:

Go to Circulation > Renew
Click Help
Confirm that the text now reads 'scan' instead of 'can'

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11762 - intranet & prog: Re-order 245 $a $b $h in results.

In the results view for OPAC and Intranet search results, titles are
currently displayed in 245 $a $b $h order. In detailed view the titles
are displayed in the better 245 $a $h $b order.

This patch corrects the behaviour for NORMARC and MARC21: all now
display 245 $a $h $b.

To test:
- Find records containing 245 $a $b and $h fields.
- Compare the way they look in detail and results view in OPAC
  (non-bootstrap) and intranet (you should see a discrepancies).
- Apply the patch.
- Re-check detail and results view: all should now look the same in
  both OPAC (non-bootstrap) and intranet.

Signed-off-by: Jesse Weaver <pianohacker@gmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes all tests and QA script, works as described.
Checked with Magnus Enger about the changes for the NORMARC
stylesheets and the changes are ok.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11762 - bootstrap: Re-order 245 $a $b $h in results.

In the results view for OPAC and Intranet search results, titles are
currently displayed in 245 $a $b $h order. In detailed view the titles
are displayed in the better 245 $a $h $b order.

This patch corrects the behaviour for NORMARC and MARC21 in
BOOTSTRAP: all now display 245 $a $h $b.

To test:
- Find records containing 245 $a $b and $h fields.
- Compare the way they look in detail and results view on the OPAC
  (bootstrap).
- Apply the patch.
- Re-check detail and results view: all should now look the same
  on the OPAC (bootstrap).

Signed-off-by: Jesse Weaver <pianohacker@gmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 10647 - Make OPAC MARC plain view work for all flavours of MARC

On "MARC view" in the OPAC, clicking on "Plain view" does not work
for UNIMARC and NORMARC.

To test:
- Make sure you have a UNIMARC or NORMARC setup
- Go to the "MARC view" of a record in the OPAC
- Click on "view plain" and observe the "Sorry, plain view is
  temporarily unavailable." error message
- Apply the patch
- Click on "view plain" and observe that a plain view of the MARC
  record is now displayed
- Sign off

I have only tested this on NORMARC, it might be good if someone
can test on UNIMARC.

Updated 2014-03-13: Incoroprates changes suggested by Marcel.
Test plan is the same as before.

Updated 2014-03-13: Tested in my UNIMARC system.
toggled opactheme to all three values, with OPACXSLTDetailsDisplay
and OPACXSLTResultsDisplay both set to default.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Tested with MARC21 and UNIMARC, passes all tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 9366: add widget to bib editor to control display of the MARC tags

If the syspref hide_marc is set to "don't display", tags were not shown
by default.

This patch adds a checkbox to hide/show tags directly on the
cataloguing page. The value is stored in a cookie.

Test plan:
Enable/disable the sysprefs hide_marc and advancedMARCeditor in order to
check if the display is according with what the user want.
Reload the page and verify the value is kept.
The first value of the cookie is retrieved from the hide_marc pref.

Bonus:
This patch removes the link "Show MARCtag documentation links" if the
syspref advancedMARCeditor is enabled.

Signed-off-by: Joel Aloi <aloi54@live.fr>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
This works as described, the editor will remember the choices
made by the cataloguer.

Passes all tests and QA script.

Notes:
- The error messages for mandatory fields upon saving only
give the tag number and not the description, so this might require
to turn back on the tag display.

- The advancedMARCeditor doesn't display the documentation links before
and after the patch was applied. I think it could still be useful for
an advanced cataloger to be able to look up information on a field quickly.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11637: don't try to guess publication dates for irregular subscriptions

Test plan:
Create a subscription with an irregular numbering pattern.

Verify the prediction pattern. With this patch, it should show "unknown"
instead of the date of the first issue publication date.

This patch does not fix the issue on generating the next serial.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
This is a partial fix for a small regression in the serials module.
Now the prediction pattern doesn't predict dates when you test it,
and shows 'Unknown' instead.

We still need to fix the 'planned date' set for the expected serial.
There should be no date, it should also be 'unknown'.

Passes tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11938: fix capitalization: Serial prediction pattern preview - Publication Date

During add or edit of a subscription when you test the
prediction pattern, "Publication Date" in the displayed table
has incorrect capitalization. This patch corrects.

To test, apply the patch and create or edit a subscription. In
step two click the "test prediction pattern" button to display
the table.  "Publication date" should be correctly capitalized.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
String change, all good.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11935: fix capitalization: OPAC prog theme, checkouts tab

opac-user.tt in the prog theme contains 'Renew Selected' and 'Renew
All'. These are incorrectly capitalized. This patch corrects.

To test you must have OpacRenewalAllowed enabled. Apply the patch,
switch to the prog theme, and log in to the OPAC as a patron who has
checkouts.

At the bottom of the table of checkouts you should see the renew buttons
with correct capitalization.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
String change, all good.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 8168: Fixing header of non CSV files for overdue notices

This determines if the CSV header should be included or not and
then generates it as needed using the delimiter specified in the
delimiter system preference.

TEST PLAN
---------
1. make some overdues books
2. run the overdue notices script without the -csv
3. check emails notice csv header is in the email
4. apply the patch
5. run the overdue notice again
6. check email notice CSV header is absent

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: wajasu <matted-34813@mypacks.net>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 8168: (follow-up) Use semicolon as default CSV delimiter

installer/data/mysql/sysprefs.sql has semicolon as default.
This fixes both instances to use the same fallback value.

It also prevents CSV header info from being included in non-CSV messages.

Signed-off-by: wajasu <matted-34813@mypacks.net>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11957: reomve disused sample_only_param_tables.sql

This sql file tries to insert data into nonexistent tables.
It is useless and can be removed.

Test plan:
  git grep sample_only_param_tables
should not return any result.

NOTE: This is basically a git rm. Simple patch.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
File is not used by the installers and it wouldn't work
with today's Koha for trying to insert data into non-existant
tables and tables with different structure.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11973 - Fix Armenian language description

This patch fix Armenian language description

To test:
1) Do a fake translation to Armenian,
   mkdir koha-tmpl/intranet-tmpl/prog/hy-Armn
2) Go to Administration > System preferences > I18N/L10N
3) Armenian description looks like the second string in Bug description
4) Apply the patch
5) Run updatedatabase.pl
6) Reload page, now description look as first string in
Bug description (it's the sixth letter, similar to 't')

Ammended patch, wrong column.

Signed-off-by: Holger Meißner <h.meissner.82@web.de>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Patch fixes the English sample file for translations and includes
a database update to correct the string in existing installations.
Passes all tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11973: DBRev 3.15.00.026

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11352: fix unexpected data loss issues with batch patron deletion/anonymization

Batch patron deletion/anonymization tool has some issues:

1) If 'dateformat' setting in I18N/L10N system preferences is set
to anything other then 'iso' (eg. 'metric'), bulk deletion when using
'expiration date is before' criterion is not working properly. Date
entered in this field will be efectivelly ignored (or possibly
sometimes wrongly interpreted as different date, in other format)
on the final patron deletion stage. This may result in deleting
(or moving to trash) more borrower records then intended.
2) Bulk/batch patron deletion should skip borrowers with
nonzero account balance (ones with oustanding fines or credits)
3) This tool shouldn't offer to choose as deletion criterion
those patron categories which have category_type set to 'S'
(= staff patron categories)

This patch fixes above mentioned problems. It also adds an option
to "test run" patron batch deletion, and makes this option
the default choice in "warning" stage.

Test plan:
- prepare test database with some patron records (at least 2,
the more the better) set up in such a way that they will be vulnerable
to issues 1 & 2
- confirm issues 1,2
- restore test database
- apply patch
- ensure issues 1 & 2 are no longer present - first by using new "test
run" option: for #1, record counts in "warning" stage and "final" stage
should be now the same; for #2, observe that patron records with nonzero
balance are now excluded from deletion
- redo the tests, this time choosing "delete permanently" and "move
to trash" instead of "test run"
- test #3 by changing "Category type" to "S" in some
test patron categories - after that, those categories should no
longer be choosable as deletion criteria.

Signed-off-by: Magnus Enger <digitalutvikling@gmail.com>
Tested with dateformat = dd/mm/yyyy. I tested with two expired patrons,
one with fines and one without. Before the patch a lot of unexpected
patrons were deleted along with the expected ones. After
applying the patch only the expired patron was deleted, not the
one with fines. The test run and the "real" run reported correct numbers.

The patch also makes sure no patron categories with category_type = S
are suggested for batch deletion.

Note: The ergonomics of the "Batch delete/anonymize" tool is hardly
optimal, but this patch fixes a real, data-loosing bug, so let's
deal with the ergonomics later.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11701: Unit tests for C4::Context::Zconn

TEST PLAN
---------
1) apply patch
2) run koha qa test tool
3) prove -v t/Context.t

Unit tests ran as expected.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11701: (follow-up) move unit tests to t/db_dependent

The new tests will not function if there isn't a valid
configuration (or mocking of preference()) so this patch
moves them to t/db_dependent/Context.t.

To test:

[1] After applying the main patch for this bug, verify that
    prove -v t/db_dependent/Context.t passes.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11701: Fix C4::Context::Zconn

When calling C4::Context::Zconn twice with different parameters,
the same ZOOM::Connection object is returned twice (parameters
of 2nd call are not used) This patch fixes that.

This is in part because the connection cache is keyed on server
name only. This patch corrects this by keying on all parameters.

TEST PLAN
---------
1) apply patch
2) run koha qa test tools
3) prove -v t/Context.t

The unit tests properly triggers the modified routine for
testing. Additionally, in hunting for ways it could break,
no nested synchronous or asynchronous Zconn's were found.
And even if they were, the keying on all parameters should allow
it to function properly without messing up the other connection.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11939: set active currency when using English sample data

This patch offers a better default for currencies,
with USA dollar as default active currency.

To test the patch:
1)Install master without the patch, during web installation
select 'parameters.sql' file
2)Finish install and go to Administration -Currencies & Exchange rates
3)No active currency

4)Install master with this patch, during web installation
 select 'parameters.sql' file
5)Finish install and go to Administration -Currencies & Exchange rates
6)Now USA dollar is the active currency

NOTE: This does solve the problem of everything being NULL for active,
      and lacking an active value. It also corrects the description
      of the optional item to check related to currencies.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Works as described, passes all tests and QA script.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Bug 11939: set active currency for non-English sample data

Revised Test Plan
-----------------
The following SQL commands were used to test this.

installer/data/mysql/en/optional/parameters.sql is part of the
first patch and didn't really need testing again.

installer/data/mysql/de-DE/optional/parameters.sql is not part
of the patch, but since I was playing with the script, which
is also attached here, it was included for good measure.

use kohadata;
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/pl-PL/optional/parameters.sql
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/es-ES/optional/parameters.sql
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/ru-RU/optional/sample_currency.sql
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/uk-UA/optional/sample_currency.sql
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/it-IT/necessari/parameters.sql
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/de-DE/optional/parameters.sql
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/nb-NO/2-Valgfritt/parameters.sql
SET FOREIGN_KEY_CHECKS=0;
TRUNCATE currency;
source /home/mtompset/kohaclone/installer/data/mysql/en/optional/parameters.sql

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Signed-off-by: Galen Charlton <gmc@esilibrary.com>