From: max@ccrl.nj.nec.com (Maximilian Ott)
Subject: adding "dynamic types" to waisindex
Organization: C&C Research Labs, NEC USA, Princeton, N.J.
Date: Tue, 5 Jan 93 03:47:41 GMT

I use Wais to index a directory tree where I through all kind of
information into. This includes text, images, postscript, ... It
always annoyed me that waisindex seems to assume that all files have
the same type. 

Not anymore. I wanted to start the New Year with a bit of hacking and
added a new data type called "magic". Similar to file(1) and magic(5)
the indexer now looks at the first few bytes of a file and tries to
determine its type. The heuristic is very simple and may fail
miserably, but it was quick to implement and it seems to work for my
files.

The different types are described in the file MAGIC_FILE (to be
defined in Makefile). Each line describes one type with 

	numberOfBytes   cookie    waisType

The cookie can be either a string or a hex (need to start with 0x). A
short sample:

4	From		rmail
3	GIF		gif
2	%!		ps
2	0x4d4d		tiff

The enclosed shar file includes five new files (magic.[hc] irtypes.[ch]
magic) and the diffs for irtfile.c, irbuild.c and Makefile.

Apply the diffs (I'm not sure if I did it right)and change MAGIC_FILE
to wherever you want to store that file. "make" should build a new
indexer and a new server (only because of a new inv.a). Now running
waisindex with "-t magic" will try to determine the type of every file
before it gets indexed.

WHAT DID I DO:

irbuild.c:

The major change to irbuild.c was to isolate the endless if-else chain
to parse the "-t" flag into a separate function. This is necessary as
the different separator functions may need to be set for every file.
I moved this function out of irbuild.c and into irtypes.c to include it
into inv.a. I also added a few more image types to it (PPM, JFIF, ..)

irtfiles.c:

"-t magic" will simply set the type variable to "CHECK_OUT". This will
be checked at the beginning of index_text_file. If true the type of
the current file will be determined and the approbate variables will be
set. 

magic.c:

The public function is char* get_magic_type(char* filename) which will
return the typename. The first time around the magic file will be read
to build an array with an entry (cookie, mask, typename) for each
type. This is very simple; fixed array size, unsigned long better be
at least 32 bit long. If you want to check more than just the first
four bytes you have to do some work.

Enjoy,

max

Sat Jan 2 1993

p.s. Working for a company which "owns" whatever I think, even on a
holiday I found it necessary to put some legal stuff into all the
files. If that causes any problems drop me a line and I'll clear it
with our legal guys. 
-----
Max Ott                 max@ccrl.nj.nec.com         phone: (609)951-2469
NEC C&C Research        Princeton, NJ, 08540        fax:   (609)951-2499

#--------------------------------CUT HERE-------------------------------------
#! /bin/sh
#
# This is a shell archive.  Save this into a file, edit it
# and delete all lines above this comment.  Then give this
# file to sh by executing the command "sh file".  The files
# will be extracted into the current directory owned by
# you with default permissions.
#
# The files contained herein are:
#
# -rw-r--r--  1 max         10164 Jan  3 17:34 irbuild.c.diff
# -rw-r--r--  1 max           635 Jan  3 17:34 irtfiles.c.diff
# -rw-r--r--  1 max           741 Jan  3 17:35 Makefile.diff
# -rw-r--r--  1 max           997 Jan  3 17:22 irtypes.h
# -rw-r--r--  1 max         10506 Jan  3 17:22 irtypes.c
# -rw-r--r--  1 max          5253 Jan  3 16:58 magic.c
# -rw-r--r--  1 max           622 Jan  3 16:57 magic.h
# -rw-r--r--  1 max           326 Jan  3 18:15 magic
# -rw-r--r--  1 max          3471 Jan  4 22:41 README.magic
#
echo 'x - irbuild.c.diff'
if test -f irbuild.c.diff; then echo 'shar: not overwriting irbuild.c.diff'; else
sed 's/^X//' << '________This_Is_The_END________' > irbuild.c.diff
X6,10d5
X< 
X<    See the readme.magic for the disclaimer regarding the changes made
X<    by NEC USA, Inc.
X< 
X<    max@ccrl.nj.nec.com
X105d99
X< #include "irtypes.h"
X175d168
X< 
X352,365c345,598
X< 
X<       /* the magic type requires the index routine to check the type
X<          of each file seperately. */
X<       typename = next_argument;
X<       if(0 == strcmp("magic", next_argument)) {
X<         type = "CHECK_OUT";
X<         waislog(WLOG_MEDIUM, WLOG_INDEX, 
X<                 "Using magic to determine the type of each file");
X< 
X<       } else
X<         set_file_characteristics(next_argument, &type,
X<                                  &separator_function, &header_function,
X<                                  &date_function, &finish_header_function,
X<                                  &check_for_text_file);
X---
X>       if(0 == strcmp("groliers", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = groliers_separator_function;
X> 	header_function = groliers_header_function;
X> 	finish_header_function = groliers_finish_header_function;
X>       }
X> #ifdef NeXT
X>       else if(0 == strcmp("objc", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = wobjc_separator_function;
X> 	header_function = wobjc_header_function;
X> 	finish_header_function = wobjc_finish_header_function;
X>       }
X> #endif /* def NeXT */
X>       else if(0 == strcmp("mail", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = mail_separator_function;
X> 	header_function = mail_header_function;
X> 	date_function = mail_date_function;
X> 	finish_header_function = mail_finish_header_function;
X>       }
X>       else if(0 == strcmp("mail_or_rmail", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = mail_or_rmail_separator;
X> 	header_function = mail_header_function;
X> 	date_function = mail_date_function;
X> 	finish_header_function = mail_finish_header_function;
X>       }
X>       else if(0 == strcmp("mail_digest", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = mail_digest_separator_function;
X>  	header_function = mail_header_function;
X>  	date_function = mail_date_function;
X>  	finish_header_function = mail_finish_header_function;
X>       }
X>       else if(0 == strcmp("mh_bboard", next_argument)){
X>  	typename = next_argument;
X>  	type = "TEXT";
X>  	separator_function = mh_bboard_separator_function;
X> 	header_function = mail_header_function;
X> 	date_function = mail_date_function;
X> 	finish_header_function = mail_finish_header_function;
X>       }
X>       else if(0 == strcmp("rmail", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = rmail_separator_function;
X> 	header_function = mail_header_function;
X> 	date_function = mail_date_function;
X> 	finish_header_function = mail_finish_header_function;
X>       }
X>       else if(0 == strcmp("netnews", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = NULL;
X> 	header_function = mail_header_function;
X> 	date_function = mail_date_function;
X> 	finish_header_function = mail_finish_header_function;
X>       }
X>       else if(0 == strcmp("rn", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = rn_separator_function;
X> 	header_function = mail_header_function;
X> 	date_function = mail_date_function;
X> 	finish_header_function = mail_finish_header_function;
X>       }
X>       else if(0 == strcmp("emacsinfo", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = emacs_info_separator_function;
X> 	header_function = emacs_info_header_function;
X> 	finish_header_function = emacs_info_finish_header_function;
X>       }
X>       else if(0 == strcmp("catalog", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = catalog_separator_function;
X> 	header_function = catalog_header_function;
X> 	finish_header_function = catalog_finish_header_function;
X>       }
X>       else if(0 == strcmp("bio", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = bio_separator_function;
X> 	header_function = bio_header_function;
X> 	finish_header_function = bio_finish_header_function;
X>       }
X>       else if(0 == strcmp("cmapp", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";	
X> 	separator_function = cmapp_separator_function;
X> 	header_function = cmapp_header_function;
X> 	finish_header_function = cmapp_finish_header_function;
X>       }
X>       else if(0 == strcmp("ftp", next_argument)){
X> 	type = "TEXT-FTP";
X> 	typename = next_argument;
X> 	separator_function = first_line_separator_function;
X> 	header_function = first_line_header_function;
X> 	finish_header_function = first_line_finish_header_function;
X>       }
X>       else if(0 == strcmp("jargon", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = jargon_separator_function;
X> 	header_function = jargon_header_function;
X> 	finish_header_function = jargon_finish_header_function;
X>       }
X>       else if(0 == strcmp("server", next_argument)){
X> 	typename = next_argument;
X> 	type = "WSRC";
X> 	finish_header_function = filename_finish_header_function;
X>       }
X>       else if(0 == strcmp("text", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	check_for_text_file = true;
X>       }
X>       else if(0 == strcmp("filename", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	finish_header_function = filename_finish_header_function;
X>       }
X>       else if(0 == strcmp("irg", next_argument)){
X> 	typename = next_argument;
X> 	type = "TEXT";
X> 	separator_function = irg_separator_function;
X> 	header_function = irg_header_function;
X> 	finish_header_function = irg_finish_header_function;
X>       }
X>       /* dash-separated items , Intro to Algorithms buglist, etc */
X>       else if(0 == strcmp("dash", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = dash_separator_function;
X> 	header_function = dash_header_function;
X> 	finish_header_function = dash_finish_header_function;
X>       }
X>       /* one_line-separated items */
X>       else if(0 == strcmp("one_line", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = one_line_separator_function;
X> 	header_function = one_line_header_function;
X> 	finish_header_function = one_line_finish_header_function;
X>       }
X>       /* blank line-separated items (paragraphs) */
X>       else if(0 == strcmp("para", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = para_separator_function;
X> 	header_function = para_header_function;
X> 	finish_header_function = para_finish_header_function;
X>       }
X>       /* seeker items */
X>       else if(0 == strcmp("seeker", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = seeker_separator_function;
X> 	header_function = seeker_header_function;
X> 	finish_header_function = seeker_finish_header_function;
X>       }
X>       /* medline format */
X>       else if(0 == strcmp("medline", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = medline_separator_function;
X> 	header_function = medline_header_function;
X> 	finish_header_function = medline_finish_header_function;
X>       }
X>       /* refer format */
X>       else if(0 == strcmp("refer", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = refer_separator_function;
X> 	header_function = refer_header_function;
X> 	finish_header_function = refer_finish_header_function;
X>       }
X>       /* first_line format */
X>       else if(0 == strcmp("first_line", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = first_line_separator_function;
X> 	header_function = first_line_header_function;
X> 	finish_header_function = first_line_finish_header_function;
X>       }
X>       /* rlin items */
X>       else if(0 == strcmp("rlin", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = rlin_separator_function;
X> 	header_function = rlin_header_function;
X> 	finish_header_function = rlin_finish_header_function;
X>       }
X>       else if(0 == strcmp("dvi", next_argument)){
X> 	typename = next_argument;
X> 	type = "DVI";
X> 	finish_header_function = filename_finish_header_function;
X>       }
X>       else if(0 == strcmp("ps", next_argument)){
X> 	typename = next_argument;
X> 	type = "PS";
X> 	finish_header_function = filename_finish_header_function;
X>       }
X>       else if(0 == strcmp("pict", next_argument)){
X> 	typename = next_argument;
X> 	type = "PICT";	
X> 	finish_header_function = filename_finish_header_function;
X> 	index_contents = false;
X>       }
X>       else if(0 == strcmp("gif", next_argument)){
X> 	typename = next_argument;
X> 	type = "GIF";	
X> 	finish_header_function = filename_finish_header_function;
X> 	index_contents = false;
X>       }
X>       else if(0 == strcmp("tiff", next_argument)){
X> 	typename = next_argument;
X> 	type = "TIFF";	
X> 	finish_header_function = filename_finish_header_function;
X> 	index_contents = false;
X>       }
X>       /* BibTeX items */
X>       else if(0 == strcmp("bibtex", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = bibtex_separator_function;
X> 	header_function = bibtex_header_function;
X> 	finish_header_function = bibtex_finish_header_function;
X>       }
X>       /* ?:? seperated hypertext items */
X>       else if(0 == strcmp("nhyp", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = nhyp_separator_function;
X> 	header_function = nhyp_header_function;
X> 	finish_header_function = nhyp_finish_header_function;
X>       }
X>       else if(0 == strcmp("ziff", next_argument)){
X> 	type = "TEXT";
X> 	typename = next_argument;
X> 	separator_function = ziff_separator_function;
X> 	header_function = ziff_header_function;
X> 	finish_header_function = ziff_finish_header_function;
X>       }
X>       else{
X> 	panic("Don't recognize the '%s' type", next_argument);
X>       }
________This_Is_The_END________
if test `wc -c < irbuild.c.diff` -ne 10164; then
	echo 'shar: irbuild.c.diff was damaged during transit (should have been 10164 bytes)'
fi
fi		; : end of overwriting check
echo 'x - irtfiles.c.diff'
if test -f irtfiles.c.diff; then echo 'shar: not overwriting irtfiles.c.diff'; else
sed 's/^X//' << '________This_Is_The_END________' > irtfiles.c.diff
X6,10d5
X< 
X<    See the readme.magic for the disclaimer regarding the changes made
X<    by NEC USA, Inc.
X< 
X<    max@ccrl.nj.nec.com
X65,66d59
X< #include "irtypes.h"
X< #include "magic.h"
X82d74
X< 
X684,695d675
X< 
X<   if (0 == strcmp(type, "CHECK_OUT")) {
X<     char *typename;
X< 
X<     typename = get_magic_type(filename);
X<     waislog(WLOG_LOW, WLOG_INDEX, "This is a <%s> type file", typename);
X<     set_file_characteristics(typename, &type, &separator_function,
X<                          &header_function, &date_function,
X<                          &finish_header_function,
X<                          &check_for_text_file);
X<   }
X< 
________This_Is_The_END________
if test `wc -c < irtfiles.c.diff` -ne 635; then
	echo 'shar: irtfiles.c.diff was damaged during transit (should have been 635 bytes)'
fi
fi		; : end of overwriting check
echo 'x - Makefile.diff'
if test -f Makefile.diff; then echo 'shar: not overwriting Makefile.diff'; else
sed 's/^X//' << '________This_Is_The_END________' > Makefile.diff
X52,53d51
X< MAGIC_FILE      = /usr/local/pub/lib/wais/magic # cookie database
X< 
X85,87c83
X< 		lock.c \
X< 		irtypes.c \
X< 		magic.c
X---
X> 		lock.c
X113c109
X< INDEX_SRC=	irbuild.c 
X---
X> INDEX_SRC=	irbuild.c
X139,141c135
X< 		lock.o \
X< 		irtypes.o \
X< 		magic.o
X---
X> 		lock.o
X237,249d230
X< magic.o:	magic.c
X< 		$(CC) $(CFLAGS) -c -DMAGIC_FILE="\"$(MAGIC_FILE)\"" magic.c
X< 
X< max_shar:;
X< 		#rm -f irbuild.c.diff
X< 		#diff irbuild.c irbuild.c-orig > irbuild.c.diff
X< 		#rm -f irtfiles.c.diff
X< 		#diff irtfiles.c irtfiles.c-orig > irtfiles.c.diff
X< 		#rm -f Makefile.diff
X< 		#diff Makefile Makefile-orig > Makefile.diff
X< 		mkshar irbuild.c.diff irtfiles.c.diff Makefile.diff\
X< 			irtypes.h irtypes.c \
X< 			magic.c magic.h magic README.magic
________This_Is_The_END________
if test `wc -c < Makefile.diff` -ne 741; then
	echo 'shar: Makefile.diff was damaged during transit (should have been 741 bytes)'
fi
fi		; : end of overwriting check
echo 'x - irtypes.h'
if test -f irtypes.h; then echo 'shar: not overwriting irtypes.h'; else
sed 's/^X//' << '________This_Is_The_END________' > irtypes.h
X/*
X * irtypes.h --setting the characteristics of indexed files
X * Copyright (C) 1993   Maximilian Ott, NEC USA, Inc.
X
X   No guarantees or restrictions.  See the readme.magic file for the
X   full standard disclaimer.
X
X   max@ccrl.nj.nec.com
X
X * Author          : Maximilian Ott
X * Created On      : Sun Jan  3 16:51:11 1993
X * Last Modified On: Sun Jan  3 17:22:56 1993
X * Status          : Unknown, Use with caution!
X * Version         : $Revision$
X ****************************************************************/
X
X#ifndef _irtypes_h_
X#define _irtypes_h_
X
Xextern void 
Xset_file_characteristics _AP((char     *typename,
X                              char     **type,
X                              boolfunc **separator_function,
X                              voidfunc **header_function,
X                              longfunc **date_function,
X                              voidfunc **finish_header_function,
X                              boolean  *check_for_text_file));
X
X#endif /* _irtypes_h_ */
________This_Is_The_END________
if test `wc -c < irtypes.h` -ne 997; then
	echo 'shar: irtypes.h was damaged during transit (should have been 997 bytes)'
fi
fi		; : end of overwriting check
echo 'x - irtypes.c'
if test -f irtypes.c; then echo 'shar: not overwriting irtypes.c'; else
sed 's/^X//' << '________This_Is_The_END________' > irtypes.c
X/* WIDE AREA INFORMATION SERVER SOFTWARE:
X   No guarantees or restrictions.  See the readme file for the full standard
X   disclaimer.
X
X   Brewster@think.com
X
X   See the readme.magic for the disclaimer regarding the changes made
X   by NEC USA, Inc.
X
X*/
X
X/********************************
X * The following code was extracted from <irbuild.c>.
X *
X * If we use the above routines to dynamically determine the type of a
X * file we have to set the various seperator functions from within
X * <irtfile.c>. The latter is part of <inv.a>. In the static case
X * these functions were assigned at the beginning of running waisindex
X * in <irbuild.c>. For clean linking this has to be moved into
X * <inv.a>. As we have to add this file as well we better do it right
X * now. 
X */
X
X/*
X * Also added new types for PPM files and zilla's image format
X */
X
X#include <string.h>
X#include "panic.h"  
X#include "cutil.h"
X#include "irtfiles.h"
X#include "ircfiles.h"
X
Xextern boolean index_contents;
X
Xvoid
Xset_file_characteristics(typename,
X                         type,
X                         separator_function,
X                         header_function,
X                         date_function,
X                         finish_header_function,
X                         check_for_text_file)
X
X     char     *typename;
X     char     **type;
X     boolfunc **separator_function;
X     voidfunc **header_function;
X     longfunc **date_function;
X     voidfunc **finish_header_function;
X     boolean  *check_for_text_file;
X
X{
X  /* default settings */
X  *separator_function = NULL; /* initailize to nil */
X  *header_function = NULL;
X  *date_function = NULL;
X  *finish_header_function = NULL;
X  *type = "TEXT"; /* default to text */
X  index_contents = true;
X
X  if(0 == strcmp("groliers", typename)) {
X    *type = "TEXT";
X    *separator_function = groliers_separator_function;
X    *header_function = groliers_header_function;
X    *finish_header_function = groliers_finish_header_function;
X  }
X#ifdef NeXT
X  else if(0 == strcmp("objc", typename)) {
X    *type = "TEXT";
X    *separator_function = wobjc_separator_function;
X    *header_function = wobjc_header_function;
X    *finish_header_function = wobjc_finish_header_function;
X  }
X#endif /* def NeXT */
X  else if(0 == strcmp("mail", typename)) {
X    *type = "TEXT";
X    *separator_function = mail_separator_function;
X    *header_function = mail_header_function;
X    *date_function = mail_date_function;
X    *finish_header_function = mail_finish_header_function;
X  }
X  else if(0 == strcmp("mail_or_rmail", typename)) {
X    *type = "TEXT";
X    *separator_function = mail_or_rmail_separator;
X    *header_function = mail_header_function;
X    *date_function = mail_date_function;
X    *finish_header_function = mail_finish_header_function;
X  }
X  else if(0 == strcmp("mail_digest", typename)) {
X    *type = "TEXT";
X    *separator_function = mail_digest_separator_function;
X    *header_function = mail_header_function;
X    *date_function = mail_date_function;
X    *finish_header_function = mail_finish_header_function;
X  }
X  else if(0 == strcmp("mh_bboard", typename)) {
X    *type = "TEXT";
X    *separator_function = mh_bboard_separator_function;
X    *header_function = mail_header_function;
X    *date_function = mail_date_function;
X    *finish_header_function = mail_finish_header_function;
X  }
X  else if(0 == strcmp("rmail", typename)) {
X    *type = "TEXT";
X    *separator_function = rmail_separator_function;
X    *header_function = mail_header_function;
X    *date_function = mail_date_function;
X    *finish_header_function = mail_finish_header_function;
X  }
X  else if(0 == strcmp("netnews", typename)) {
X    *type = "TEXT";
X    *separator_function = NULL;
X    *header_function = mail_header_function;
X    *date_function = mail_date_function;
X    *finish_header_function = mail_finish_header_function;
X  }
X  else if(0 == strcmp("rn", typename)) {
X    *type = "TEXT";
X    *separator_function = rn_separator_function;
X    *header_function = mail_header_function;
X    *date_function = mail_date_function;
X    *finish_header_function = mail_finish_header_function;
X  }
X  else if(0 == strcmp("emacsinfo", typename)) {
X    *type = "TEXT";
X    *separator_function = emacs_info_separator_function;
X    *header_function = emacs_info_header_function;
X    *finish_header_function = emacs_info_finish_header_function;
X  }
X  else if(0 == strcmp("catalog", typename)) {
X    *type = "TEXT";
X    *separator_function = catalog_separator_function;
X    *header_function = catalog_header_function;
X    *finish_header_function = catalog_finish_header_function;
X  }
X  else if(0 == strcmp("bio", typename)) {
X    *type = "TEXT";
X    *separator_function = bio_separator_function;
X    *header_function = bio_header_function;
X    *finish_header_function = bio_finish_header_function;
X  }
X  else if(0 == strcmp("cmapp", typename)) {
X    *type = "TEXT";	
X    *separator_function = cmapp_separator_function;
X    *header_function = cmapp_header_function;
X    *finish_header_function = cmapp_finish_header_function;
X  }
X  else if(0 == strcmp("ftp", typename)) {
X    *type = "TEXT-FTP";
X    *separator_function = first_line_separator_function;
X    *header_function = first_line_header_function;
X    *finish_header_function = first_line_finish_header_function;
X  }
X  else if(0 == strcmp("jargon", typename)) {
X    *type = "TEXT";
X    *separator_function = jargon_separator_function;
X    *header_function = jargon_header_function;
X    *finish_header_function = jargon_finish_header_function;
X  }
X  else if(0 == strcmp("server", typename)) {
X    *type = "WSRC";
X    *finish_header_function = filename_finish_header_function;
X  }
X  else if(0 == strcmp("text", typename)) {
X    *type = "TEXT";
X    *check_for_text_file = true;
X  }
X  else if(0 == strcmp("filename", typename)) {
X    *type = "TEXT";
X    *finish_header_function = filename_finish_header_function;
X  }
X  else if(0 == strcmp("irg", typename)) {
X    *type = "TEXT";
X    *separator_function = irg_separator_function;
X    *header_function = irg_header_function;
X    *finish_header_function = irg_finish_header_function;
X  }
X  /* dash-separated items , Intro to Algorithms buglist, etc */
X  else if(0 == strcmp("dash", typename)) {
X    *type = "TEXT";
X    *separator_function = dash_separator_function;
X    *header_function = dash_header_function;
X    *finish_header_function = dash_finish_header_function;
X  }
X  /* one_line-separated items */
X  else if(0 == strcmp("one", typename)) {
X    *type = "TEXT";
X    *separator_function = one_line_separator_function;
X    *header_function = one_line_header_function;
X    *finish_header_function = one_line_finish_header_function;
X  }
X  /* blank line-separated items (paragraphs) */
X  else if(0 == strcmp("para", typename)) {
X    *type = "TEXT";
X    *separator_function = para_separator_function;
X    *header_function = para_header_function;
X    *finish_header_function = para_finish_header_function;
X  }
X  /* seeker items */
X  else if(0 == strcmp("seeker", typename)) {
X    *type = "TEXT";
X    *separator_function = seeker_separator_function;
X    *header_function = seeker_header_function;
X    *finish_header_function = seeker_finish_header_function;
X  }
X  /* medline format */
X  else if(0 == strcmp("medline", typename)) {
X    *type = "TEXT";
X    *separator_function = medline_separator_function;
X    *header_function = medline_header_function;
X    *finish_header_function = medline_finish_header_function;
X  }
X  /* refer format */
X  else if(0 == strcmp("refer", typename)) {
X    *type = "TEXT";
X    *separator_function = refer_separator_function;
X    *header_function = refer_header_function;
X    *finish_header_function = refer_finish_header_function;
X  }
X  /* first_line format */
X  else if(0 == strcmp("first", typename)) {
X    *type = "TEXT";
X    *separator_function = first_line_separator_function;
X    *header_function = first_line_header_function;
X    *finish_header_function = first_line_finish_header_function;
X  }
X    /* rlin items */
X  else if(0 == strcmp("rlin", typename)) {
X    *type = "TEXT";
X    *separator_function = rlin_separator_function;
X    *header_function = rlin_header_function;
X    *finish_header_function = rlin_finish_header_function;
X  }
X  else if(0 == strcmp("dvi", typename)) {
X    *type = "DVI";
X    *finish_header_function = filename_finish_header_function;
X  }
X  else if(0 == strcmp("ps", typename)) {
X    *type = "PS";
X    *finish_header_function = filename_finish_header_function;
X  }
X  else if(0 == strcmp("pict", typename)) {
X    *type = "PICT";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  else if(0 == strcmp("gif", typename)) {
X    *type = "GIF";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  else if(0 == strcmp("ppm", typename)) {
X    *type = "PPM";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  else if(0 == strcmp("jfif", typename)) {
X    *type = "JFIF";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  else if(0 == strcmp("sunraster", typename)) {
X    *type = "SUNRASTER";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  else if(0 == strcmp("sgi", typename)) {
X    *type = "SGI";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  else if(0 == strcmp("zilla", typename)) {
X    *type = "ZILLA";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  else if(0 == strcmp("tiff", typename)) {
X    *type = "TIFF";	
X    *finish_header_function = filename_finish_header_function;
X    index_contents = false;
X  }
X  /* BibTeX items */
X  else if(0 == strcmp("bibtex", typename)) {
X    *type = "TEXT";
X    *separator_function = bibtex_separator_function;
X    *header_function = bibtex_header_function;
X    *finish_header_function = bibtex_finish_header_function;
X  }
X  /* ?:? seperated hypertext items */
X  else if(0 == strcmp("nhyp", typename)) {
X    *type = "TEXT";
X    *separator_function = nhyp_separator_function;
X    *header_function = nhyp_header_function;
X    *finish_header_function = nhyp_finish_header_function;
X  }
X  else if(0 == strcmp("ziff", typename)) {
X    *type = "TEXT";
X    *separator_function = ziff_separator_function;
X    *header_function = ziff_header_function;
X    *finish_header_function = ziff_finish_header_function;
X  } else 
X    panic("Don't recognize the '%s' type", typename);
X} /* end of set_file_characteristics */
________This_Is_The_END________
if test `wc -c < irtypes.c` -ne 10506; then
	echo 'shar: irtypes.c was damaged during transit (should have been 10506 bytes)'
fi
fi		; : end of overwriting check
echo 'x - magic.c'
if test -f magic.c; then echo 'shar: not overwriting magic.c'; else
sed 's/^X//' << '________This_Is_The_END________' > magic.c
X/*@  make -k 
X * magic.c -- tries to determine the type of a file 
X * Copyright (C) 1992   Maximilian Ott, NEC USA, Inc.
X
X   No guarantees or restrictions.  See the readme.magic file for the
X   full standard disclaimer.
X
X   max@ccrl.nj.nec.com
X
X * Author          : Maximilian Ott
X * Created On      : Sat Jan  2 13:26:41 1993
X * Last Modified On: Sun Jan  3 16:58:05 1993
X * Status          : Unknown, Use with caution!
X * Version         : $Revision$
X *+********************************************************************
X * Description:
X
X *   char* get_magic_type(char* filename) will return the typename of
X *   filename. The first time around the magic file will be read to
X *   build an array with an entry (cookie, mask, typename) for each
X *   type. This is very simple; fixed array size, unsigned long better
X *   be at least 32 bit long. If you want to check more than just the
X *   first four bytes you have to do some work.
X
X *   The different types are described in the file MAGIC_FILE (to be
X *   defined in Makefile). Each line describes one type with  
X
X        numberOfBytes   cookie    waisType
X
X *   The cookie can be either a string or a hex (need to start with
X *   0x). A short sample:
X
X        4	From		rmail
X        3	GIF		gif
X        2	%!		ps
X        2	0x4d4d		tiff
X
X *
X * HISTORY:
X *
X * Known Bugs:
X *
X * Functions Defined:
X *
X *    char* get_magic_type(char* filename)
X *
X **-******************************************************************/
X#include <stdio.h>
X#include <string.h>
X#include "cutil.h"
X#include "futil.h"
X#include "panic.h"
X#include "version.h"
X
X#ifndef LINT
Xstatic char rcsid[] = 
X  "$Id$";
X#endif
X
X/******* DEFINITIONS *************************************************/
X
X#ifndef MAGIC_FILE
X#define MAGIC_FILE   "/usr/local/pub/lib/wais/magic"
X#endif  MAGIC_FILE
X
X/* if you increase MAXCOOKIELENGTH you better check the code as this
X   is used to create a mask and a long might not be enough to hold it.
X   I agree, this is is pretty ugly. */ 
X#define MAXCOOKIELENGTH    4    /* max number of bytes for cookie  */
X
X#define MAXMAGICENTRIES   40    /* max number of entries in the magic
X                                   file  */
X#define MAXBUFLEN        256    /* max length of a line in the magic
X                                   file. */
X#define MAXTYPESTRING     30    /* max length of a type string. */
X
X
X/******* USEFUL MACROS ***********************************************/
X
X/******* EXTERNAL VARIABLES ******************************************/
X
X/******* LOCAL VARIABLES *********************************************/
X
Xstatic boolean magic_file_read = FALSE; /* set to TRUE after magic
X                                           file has been read */
X
Xtypedef struct {
X  long    cookie;
X  u_long  mask;
X  char    type[MAXTYPESTRING];
X} Magic;
X
XMagic magic_list[MAXMAGICENTRIES]; /* make an entry for every line in the
X                                      magic file. */
Xint   magic_list_length = 0;    /* the number of entries in the
X                                   magic_list. */
X
X/******* EXTERNAL FUNCTIONS ******************************************/
X
X/******* LOCAL FUNCTIONS *********************************************/
X
Xstatic void
Xread_magic_file()
X
X{
X  FILE*	  fp;
X  char	  buf[MAXBUFLEN];
X  int     length;
X  int     i;
X  char    cookie[30];
X  Magic*  m;
X
X  if ((fp = fopen(MAGIC_FILE, "r")) == NULL)
X    panic("Can't open magic file <%s>", MAGIC_FILE);
X
X  waislog(WLOG_LOW, WLOG_INDEX, "Reading magic file <%s>",
X          MAGIC_FILE); 
X  while(fgets(buf, sizeof(buf), fp) != NULL) {
X    if (buf[0] == '#')          /* comment */
X      continue;
X    m = &magic_list[magic_list_length];
X    if (3 != sscanf(buf, "%d%s%s", &length, cookie, m->type))
X      panic("Wrong format in magic file <%s>", buf);
X    /* convert cookie to hex */
X    m->cookie = 0;
X    if (0 == strncmp(cookie, "0x", 2)) {
X      if (1 != sscanf(cookie, "%i", &m->cookie))
X        panic("Can't convert cookie to hex <%s>", cookie);
X    } else {
X      for (i = 0; i < length; i++)
X        m->cookie = m->cookie * 256 + cookie[i];
X    }
X    for (i = 0; i < MAXCOOKIELENGTH - length; i++)
X      m->cookie *= 256;
X    m->mask = 0xffffffff;
X    for (i = 0; i < MAXCOOKIELENGTH - length; i++)
X      m->mask = m->mask ^ (0xff << 8 * i);
X    waislog(WLOG_LOW, WLOG_INDEX, 
X            "length: %d\tcookie: (%x) %s\tmask: %x\ttype: %s", 
X            length, m->cookie, cookie, m->mask, m->type);
X    magic_list_length++;
X  }
X  fclose(fp);
X  magic_file_read = TRUE;
X} /* end of read_magic_file */
X
X
Xchar*
Xget_magic_type(filename)
X
X     char* filename;
X
X{
X  char*   type = "text";        /* default */
X  FILE*   fp;
X  char    buf[MAXCOOKIELENGTH];
X  long    cookie;
X  Magic   *m;
X  int     i;
X
X  if (!magic_file_read)
X    read_magic_file();
X
X  if ((fp = fopen(filename, "r")) == NULL)
X    panic("Can't open file <%s>", filename);
X  if (NULL == fgets(buf, sizeof(buf), fp))
X    panic("Can't read cookie from file <%s>", filename);
X  fclose(fp);
X
X  cookie = 0;
X  for (i = 0; i < MAXCOOKIELENGTH; i++)
X    cookie = cookie * 256 + buf[i];
X
X  m = magic_list;
X  for (i = 0; i < magic_list_length; i++, m++) {
X    if (0 == ((cookie ^ m->cookie) & m->mask))
X      return (m->type);
X  }
X  return(type);
X} /* end of get_magic_type */
________This_Is_The_END________
if test `wc -c < magic.c` -ne 5253; then
	echo 'shar: magic.c was damaged during transit (should have been 5253 bytes)'
fi
fi		; : end of overwriting check
echo 'x - magic.h'
if test -f magic.h; then echo 'shar: not overwriting magic.h'; else
sed 's/^X//' << '________This_Is_The_END________' > magic.h
X/*
X * magic.h -- tries to determine the type of a file
X * Copyright (C) 1992   Maximilian Ott, NEC USA, Inc.
X
X   No guarantees or restrictions.  See the readme.magic file for the
X   full standard disclaimer.
X
X   max@ccrl.nj.nec.com
X
X * Author          : Maximilian Ott
X * Created On      : Sat Jan  2 13:28:09 1993
X * Last Modified On: Sun Jan  3 16:58:04 1993
X * Status          : Unknown, Use with caution!
X * Version         : $Revision$
X ****************************************************************/
X
X#ifndef _magic_h_
X#define _magic_h_
X
Xextern char*
Xget_magic_type  _AP((char* filename));
X
X#endif /* _magic_h_ */
________This_Is_The_END________
if test `wc -c < magic.h` -ne 622; then
	echo 'shar: magic.h was damaged during transit (should have been 622 bytes)'
fi
fi		; : end of overwriting check
echo 'x - magic'
if test -f magic; then echo 'shar: not overwriting magic'; else
sed 's/^X//' << '________This_Is_The_END________' > magic
X#
X#  Similiar to file(1) and magic(5) we look at the first few bytes in
X#  a file to determine its type.
X#
X#bytes	cookie	wais type
X#
X4	From		rmail
X3	GIF		gif
X2	%!		ps
X2	0x4d4d		tiff
X2	0xffd8		jfif
X4	0x59a66a95	sunraster
X2	0x01da		sgi
X2	0x5069		zilla
X2	0x5369		zilla
X2	P1		ppm
X2	P2		ppm
X2	P3		ppm
X2	P4		ppm
X2	P5		ppm
X2	P6		ppm
________This_Is_The_END________
if test `wc -c < magic` -ne 326; then
	echo 'shar: magic was damaged during transit (should have been 326 bytes)'
fi
fi		; : end of overwriting check
echo 'x - README.magic'
if test -f README.magic; then echo 'shar: not overwriting README.magic'; else
sed 's/^X//' << '________This_Is_The_END________' > README.magic
X
X
XWARRANTY DISCLAIMER
X 
XThis software was modified by NEC USA, Inc. and is distributed free of
Xcharge.  It is placed in the public domain and permission is granted
Xto anyone to use, duplicate, modify and redistribute it provided that
Xthis notice is attached.
X 
XNEC USA, Inc. provides absolutely NO WARRANTY OF ANY KIND with respect
Xto this software.  The entire risk as to the quality and performance
Xof this software is with the user.  IN NO EVENT WILL THINKING MACHINES
XCORPORATION BE LIABLE TO ANYONE FOR ANY DAMAGES ARISING OUT THE USE OF
XTHIS SOFTWARE, INCLUDING, WITHOUT LIMITATION, DAMAGES RESULTING FROM
XLOST DATA OR LOST PROFITS, OR FOR ANY SPECIAL, INCIDENTAL OR
XCONSEQUENTIAL DAMAGES.
X----------------------------------------------------------------------
X
XI use Wais to index a directory tree where I through all kind of
Xinformation into. This includes text, images, postscript, ... It
Xalways annoyed me that waisindex seems to assume that all files have
Xthe same type. 
X
XNot anymore. I wanted to start the New Year with a bit of hacking and
Xadded a new data type called "magic". Similar to file(1) and magic(5)
Xthe indexer now looks at the first few bytes of a file and tries to
Xdetermine its type. The heuristic is very simple and may fail
Xmiserably, but it was quick to implement and it seems to work for my
Xfiles.
X
XThe different types are described in the file MAGIC_FILE (to be
Xdefined in Makefile). Each line describes one type with 
X
X	numberOfBytes   cookie    waisType
X
XThe cookie can be either a string or a hex (need to start with 0x). A
Xshort sample:
X
X4	From		rmail
X3	GIF		gif
X2	%!		ps
X2	0x4d4d		tiff
X
XThe enclosed shar file includes five new files (magic.[hc] irtypes.[ch]
Xmagic) and the diffs for irtfile.c, irbuild.c and Makefile.
X
XApply the diffs (I'm not sure if I did it right)and change MAGIC_FILE
Xto wherever you want to store that file. "make" should build a new
Xindexer and a new server (only because of a new inv.a). Now running
Xwaisindex with "-t magic" will try to determine the type of every file
Xbefore it gets indexed.
X
XWHAT DID I DO:
X
Xirbuild.c:
X
XThe major change to irbuild.c was to isolate the endless if-else chain
Xto parse the "-t" flag into a separate function. This is necessary as
Xthe different separator functions may need to be set for every file.
XI moved this function out of irbuild.c and into irtypes.c to include it
Xinto inv.a. I also added a few more image types to it (PPM, JFIF, ..)
X
Xirtfiles.c:
X
X"-t magic" will simply set the type variable to "CHECK_OUT". This will
Xbe checked at the beginning of index_text_file. If true the type of
Xthe current file will be determined and the approbate variables will be
Xset. 
X
Xmagic.c:
X
XThe public function is char* get_magic_type(char* filename) which will
Xreturn the typename. The first time around the magic file will be read
Xto build an array with an entry (cookie, mask, typename) for each
Xtype. This is very simple; fixed array size, unsigned long better be
Xat least 32 bit long. If you want to check more than just the first
Xfour bytes you have to do some work.
X
XEnjoy,
X
Xmax
X
XSat Jan 2 1993
X
Xp.s. Working for a company which "owns" whatever I think, even on a
Xholiday I found it necessary to put some legal stuff into all the
Xfiles. If that causes any problems drop me a line and I'll clear it
Xwith our legal guys. 
X-----
XMax Ott                 max@ccrl.nj.nec.com         phone: (609)951-2469
XNEC C&C Research        Princeton, NJ, 08540        fax:   (609)951-2499
X
X
________This_Is_The_END________
if test `wc -c < README.magic` -ne 3471; then
	echo 'shar: README.magic was damaged during transit (should have been 3471 bytes)'
fi
fi		; : end of overwriting check
exit 0



-- 
Max Ott                 max@ccrl.nj.nec.com         phone: (609)951-2469
NEC C&C Research        Princeton, NJ, 08540        fax:   (609)951-2499
Newsgroups: comp.infosystems.wais