Understanding Zebra indexing

Warning for those who follow: this page is largely outdated, as all examples refer to indexing using GRS-1. The file record.abs is obsolete, and all examples below need to be re-written to use biblio-koha-indexdefs.xml. 
----
 
== Adding additional fields to Zebra search ==
You will have to modify following files to add previously unsupported field (856$u in our example) to Zebra: 
 
=== zebradb/biblios/etc/bib1.att ===
List of search indexes and their corresponding Z39.50 use attributes 
Documentation for bib1 is available at http://www.loc.gov/z3950/agency/bib1.html and according to it, our 856$u field should map to 1032 
 dpavlin@koha-dev:/etc/koha/zebradb$ grep 1032 biblios/etc/bib1.att
 att 1032 Doc-id
=== zebradb/marc_defs/marc21/biblios/record.abs ===
mapping of MARC fields to indexes 
 dpavlin@koha-dev:/etc/koha/zebradb$ grep Doc-id marc_defs/marc21/biblios/record.abs
 melm 856$u Doc-id
=== zebradb/ccl.properties ===
for searching purposes 
 dpavlin@koha-dev:/etc/koha/zebradb$ grep Doc-id ccl.properties
 Doc-id 1=1032
=== /usr/share/koha/lib/C4/Search.pm ===
(The path to your Search.pm may be different.) 
add to my @indexes under # biblio indexes. 
 'Doc-id'
After that, you will have to reindex Zebra so that new index gets populated. 
Alternative example of modifying Koha to support additional fields is available in http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=4506 
== Examine Zebra index ==
Example is provided by Adam Dickmeiss <adam@indexdata.dk> on koha-devel mailing list. 
You should be able to view indexing for any record by doing a 'present' of the record with element set zebra::index and syntax being XML or SUTRS, see https://www.indexdata.com/zebra/doc/special-retrieval.html 
Most Z39.50 client should be able to do that. 
yaz-client example: 
 yaz-client z3950.indexdata.com/marc
 Connecting...OK.
 Sent initrequest.
 Connection accepted by v3 target.
 ID : 81
 Name : Zebra Information Server/GFS/YAZ
 Version: 4.1.3 3ec3ad2bd1e248022bbb7c9d762844bef7f8a461
 Options: search present delSet triggerResourceCtrl scan sort
 extendedServices namedResultSets
 Elapsed: 0.055151
 Z> f computer
 Sent searchRequest.
 Received SearchResponse.
 Search was a success.
 Number of hits: 10, setno 1
 SearchResult-1: term=computer cnt=10
 records returned: 0
 Elapsed: 0.029065
 Z> s
 Sent presentRequest (1+1).
 Records: 1
 [marc]Record type: USmarc
 00366nam 22001698a 4504
 001 11224466
 003 DLC
 005 00000000000000.0
 008 910710c19910701nju 00010 eng
 010 $a 11224466
 040 $a DLC $c DLC
 050 00 $a 123-xyz
 100 10 $a Jack Collins
 245 10 $a How to program a computer
 260 1 $a Penguin
 263 $a 8710
 300 $a p. cm.
 
 nextResultSetPosition = 2
 Elapsed: 0.026880
 Z> form xml
 Z> elem zebra::index
 Z> s 1+1
 Sent presentRequest (1+1).
 Records: 1
 Record type: XML
 <record xmlns="http://www.indexdata.com/zebra/" sysno="2" set="zebra::index">
 <index name="Author-name-personal" type="w" seq="1">@^</index>
 <index name="Author-name-personal" type="w" seq="1"></index>
 <index name="Author-name-personal" type="w" seq="2">jack</index>
 <index name="Author-name-personal" type="w" seq="3">collins</index>
 <index name="Author-name-personal" type="p" seq="4">jack collins</index>
 <index name="Author" type="w" seq="4">@^</index>
 <index name="Author" type="w" seq="1"></index>
 <index name="Author" type="w" seq="5">jack</index>
 <index name="Author" type="w" seq="6">collins</index>
 <index name="Author" type="p" seq="7">jack collins</index>
 <index name="any" type="w" seq="7">@^</index>
 <index name="any" type="w" seq="1"></index>
 <index name="any" type="w" seq="8">jack</index>
 <index name="any" type="w" seq="9">collins</index>
 <index name="title" type="w" seq="10">@^</index>
 <index name="title" type="w" seq="1"></index>
 <index name="title" type="w" seq="11">how</index>
 <index name="title" type="w" seq="12">to</index>
 <index name="title" type="w" seq="13">program</index>
 <index name="title" type="w" seq="14">a</index>
 <index name="title" type="w" seq="15">computer</index>
 <index name="title" type="p" seq="16">how to program a computer</index>
 <index name="any" type="w" seq="16">@^</index>
 <index name="any" type="w" seq="17">how</index>
 <index name="any" type="w" seq="18">to</index>
 <index name="any" type="w" seq="19">program</index>
 <index name="any" type="w" seq="20">a</index>
 <index name="any" type="w" seq="21">computer</index>
 <index name="Place-publication" type="w" seq="22">@^</index>
 <index name="Place-publication" type="w" seq="1"></index>
 <index name="Place-publication" type="w" seq="23">penguin</index>
 <index name="any" type="w" seq="24">@^</index>
 <index name="any" type="w" seq="25">penguin</index>
 <index name="_ALLRECORDS" type="w" seq="2"></index>
 </record>
 nextResultSetPosition = 2
 Elapsed: 0.027786
In XML output "seq" is sequence number, position of "token". 
== Recover MARC records from Zebra index ==
Sometimes, you drop the database, but you would really want to keep biblio records which are still in Zebra. 
-m option will dump results into marc file which you can later import back into Koha 
 dpavlin@koha-dev:~$ yaz-client -m ~/backup.marc unix:/var/run/koha/gimpoz/bibliosocket
 Connecting...OK.
 Sent initrequest.
 Connection accepted by v3 target.
 ID  : 81
 Name  : Zebra Information Server/GFS/YAZ
 Version: 4.2.29 94b8ca343cc7084394526bd26c80c208aa16c9f9
 Options: search present delSet triggerResourceCtrl scan sort extendedServices namedResultSets
 Elapsed: 0.000950
 Z> base biblios
I'm assuming that every record in database has gimpoz string in it to select all records: 
 Z> find gimpoz
 Sent searchRequest.
 Received SearchResponse.
 Search was a success.
 Number of hits: 6022, setno 1
 SearchResult-1: term=gimpoz cnt=6022
 records returned: 0
 Elapsed: 0.006662
Number of hits seems OK, so we start retrieving records. You will not be able to retrieve all records using single show command, so we do it in batches of 1000 records: 
 Z> show 1+1000
 # record output...
 nextResultSetPosition = 1001
 Elapsed: 0.112413
 Z> show 1001+1000
 ...
 nextResultSetPosition = 2001
 Elapsed: 0.107331
 Z> show 2001+1000
 ...
 nextResultSetPosition = 3001
 Elapsed: 0.105962
 Z> show 3001+1000
 ...
 nextResultSetPosition = 4001
 Elapsed: 0.106484
 Z> show 4001+1000
 ...
 nextResultSetPosition = 5001
 Elapsed: 0.105443
 Z> show 5001+1000
 ...
 nextResultSetPosition = 6001
 Elapsed: 0.105816
 Z> show 6001+22
 ...
 nextResultSetPosition = 6023
 Elapsed: 0.002511
 Z> exit
 See you later, alligator.
 $ ls -al ~/backup.marc 
 -rw-r--r-- 1 dpavlin dpavlin 1395577 2012-04-24 23:50 /home/dpavlin/backup.marc
Rember to use koha-dump to create backup before you start destructive operations! 
== "Abandon hope, all ye who enter" ==
(jcamins originally pasted this in the pastebin, copying it here for more people to see it.) 
To give a concrete example of what I'm talking about, consider the following data: 
 =610 20$aGoddard College$vFaculty publications.
And this configuration: 
 melm 610$a Name-and-title
 melm 610$t Name-and-title,Title
 melm 610$9 Koha-Auth-Number
 melm 610 Name,Subject,Subject:p,Corporate-name
From that, you get these indexes: 
 Name-and-title = {"Goddard College"}
 Name = {"Faculty publications"}
 Subject = {"Faculty publications"}
 Subject:p = {"Faculty publications"}
 Corporate-name = {"Faculty publications"}
What we wanted, obviously, was for the Subject, Name, and Corporate-name indexes to include the entire 610 field. But that doesn't happen, because the moment the "melm 610$a" line is executed, the parser stops, and the "melm 610" line is never reached). In order to get the behavior we want, we need the following configuration (actually, I'm not including subfield $t or subfield $9 in those indexes): 
 melm 610$a Name-and-title,Subject,Subject:p,Name,Corporate-name
 melm 610$t Name-and-title,Title
 melm 610$9 Koha-Auth-Number
 melm 610 Name,Subject,Subject:p,Corporate-name
From this correct configuration, we get these indexes: 
 Name-and-title = {"Goddard College"}
 Name = {Goddard, College, Faculty, publications}
 Subject = {Goddard, College, Faculty, publications}
 Subject:p = {"Goddard College", "Faculty publications"}
 Corporate-name = {Goddard, College, Faculty, publications}
Hopefully someone will be able to do something useful with this information. 
== Following discussion ==
*<hdl> oh that... Yes. Indeeed. We had that kind of problem. 
*<cait> looks like something that should go on the wiki 
*<hdl> we made a patch on record.abs in order to index items. 
*<hdl> And yes record.abs Is a nightmare... But changing DOM indexing is also quite promising in terms of fun... There is no "any" index by default. you have to add that. 
*<jcamins> hdl: the "any index doesn't work the way everyone thinks it does with GRS-1, either. 
*<jcamins> And if we use xsltproc to compile our DOM configuration, it's easy enough to add a "match='*'" template. 
*<hdl> Well at least, it searches the indexed data 
*<jcamins> hdl: yes, but I am only certain that two people understand that. 
*<hdl> jcamins: I much more believe in Solr than doing overcomplex XSLT just to index stuff. 
*<jcamins> (probably more do, but the only two people who I *know* understand what "all any" does are me, and you, since you just told me you did;) 
*<hdl> count paul and gmcharlt and chris and chris_n too in I guess fredericd is also a good guess for that... That makes quite a lot ;) 
*<hdl> (a kind of community of geeks working on the same stuff...) 
*<hdl> any is gathering all the indexed elements. Not all the elements of the record. 
*<jcamins> Right. 
*<hdl> nor all the elements that have not been indexed. 
*<hdl> can be compared to all_fields in solr
== Another real life example ==
Here are the changes that I avoided to explain in the Kohacon2012 presentation Koha in language centres, maybe it will help someone. This example is for MARC21. The idea was to index four fields, map them to strings for the search query and make them searchable in the OPAC. The fields are 
 MARC strings
 653$a ln-class
 521$a lex
 658$a curriculum
 655$a index-term-genre
We had to edit four files: 
 /usr/share/koha/lib/C4/Search.pm
 /etc/koha/zebradb/biblios/etc/bib1.att
 /etc/koha/zebradb/ccl.properties
 /etc/koha/zebradb/marc_defs/marc21/biblios/record.abs
+ means we added a line 
- means we deleted a file 
= means we kept the line, it's in the example so you have and idea what section of a file is edited 
 1. /usr/share/koha/lib/C4/Search.pm 
 
:We added the strings that we wanted to map to the MARC fields to @indexes here
 
 = sub getIndexes{
 = my @indexes = (
 = # biblio indexes
 
[…] 
 + 'Index-term-genre',
 + 'lex',
 + 'ln-class',
 + 'curriculum',
 2. /etc/koha/zebradb/biblios/etc/bib1.att 
 
:We added two lines here, the other two were in there already.
 
 = ## Fixed Fields and other special indexes
 
[…] 
 = att 9903 lex
 + att 9653 ln-class
 + att 9655 Index-term-genre
 = att 9658 curriculum
 3. /etc/koha/zebradb/ccl.properties 
 
:again, two lines were already in here, two were added
 
 = lex 1=9903 r=r
 = curriculum 1=9658
 + ln-class 1=9653
 + Index-term-genre 1=9655
 4. /etc/koha/zebradb/marc_defs/marc21/biblios/record.abs 
 
:and here we changed one and added two more lines
 
 - melm 521$a lex:n
 + melm 521$a lex:w, lex:p
[…] 
 + melm 653$a ln-class:w,ln-class:p,Subject,Subject:p
 = melm 653$9 Koha-Auth-Number
 = melm 653 Subject,Subject:p
[…] 
 + melm 655$a Index-term-genre:w,Index-term-genre:p,Subject,Subject:p
 = melm 655$9 Koha-Auth-Number
 = melm 655 Subject,Subject:p
[…] 
 = melm 658$a curriculum:w,curriculum:p,Subject,Subject:p
 You have to reindex your records after the changes. An example search query using some of these changes is 
 opac-search.pl?idx=ln-class&q=English&limit=curriculum:TOEFL
== Related pages ==
*Troubleshooting Koha as a Z39.50 server - test Zebra server with yaz-client and more 
*Developing Zebra - Develop Zebra using Docker and your C skills
Category:Zebra