From: umueller@iiic.ethz.ch (Urban Dominik Mueller)
Subject: XPK compression standard (long)
Organization: Dept. Informatik, Swiss Federal Institute of Technology (ETH), Zurich, CH
Date: Sat, 25 Jul 1992 23:20:49 GMT

Since there is comp.compression.research now, I can dare to talk about
something as trivial as a new standard for compression programs. I have
implemented an interface between applications and compression programs
that makes both sides exchangeable. The result is called XPK as for 
'eXternal PacKer'. The only implementation yet exists on the Amiga, and
I'm very interested in hearing from you in case you are interested in
porting it to any other platforms. Note that one key element of the XPK
standard are late-binding libraries (like .DLL on PC's) which should
be provided by the respective OS.

Here comes the description:
--------------------------------------------------------------------


XPK - A NEW STANDARD FOR DATA COMPRESSION
=========================================
 
MOTIVATION
----------
* Many programs that should offer data compression (e.g. HD backup utilities)
  don't.
* Many programs that offer data compression use an old, slow, inefficient or
  inappropriate algorithm.
* All programs that offer data compression offer just one algorithm, you're
  stuck with that one.
* Many good packers are not used by any application program and have no
  good user interface.
* The installation of most packers requires AmigaShell knowledge (putting
  LHarc in the path so that DiskMaster can find it)
* The decompression of all files packed with existing packers requires
  knowledge about the packer used for compression.
* Many compression programs can not deal with files that are larger than
  available memory.
* The existing compression programs are either slow or have a low 
  compression factor.
* There is no way to support upcoming hardware compression cards in already
  existing applications yet.  
* For none of the current compression programs exists a real decompressing 
  file handler that uses no dirty tricks to decompress files on the fly.
 
The solution to all these problems is xpk.
 
 
OVERVIEW
--------
The xpk standard is to data compression what xpr is to file transmission.
It consists of three layers:
Level 2: The application/xpk interface for archives (planned)
Level 1: The application/xpk interface for files
Level 0: The xpk/packer interface
In addition, there is an optional standard xpk file format.
 
All parts of the xpk standard are implemented in shared libraries. There
is one master library for archive level access, one master library for file
level access, and one library for each compression algorithm.
 
level 2            xpkarchive.library
                           |
                           V
level 1             xpkmaster.library
                     /     |     \
                    /      |      \
                   V       V       V
level 0          xpkNUKE  xpkZOO  xpkLHARC
 
 
 
LEVEL 0 LIBRARIES
-----------------
All level 0 libraries offer the same functions. They're very small. Typical 
calls are: "Tell me what you can", "Compress this chunk of memory to another
chunk of memory", and "Decompress this chunk of memory to another chunk of
memory". These libs are very limited, their functionality is expanded by
xpkmaster.library. No one would want to talk to a sub library directly.
 
 
THE LEVEL 1 LIBRARY
-------------------
Offers functions like "Compress this file to that chunk of memory using that
algorithm". All combinations permitted: Mem to mem, file to file, mem to file,
decompression and compression. Asynchronous packing possible. Very convenient
tag based caller interface. Automatically finds out which sub library to use
for decompression. Returns detailed error messages.
 
 
THE LEVEL 2 LIBRARY
-------------------
Offers archiving functions like "add this file to that archive" or "show
me the contents of that archive". Not implemented yet.
 
 
OVERRIDING
----------
Libraries of a lower level can offer higher level functions. They then 
override the automatic functinality expansion by the higher level library.
xpkmaster.library, for example, enforces the use of the xpk standard file
format unless overridden by a sub library.
 
 
THE XPK FILE FORMAT
-------------------
Offers checksums, chunks (important when Seek()s on a compressed file become
necessary) and automatic handling by the xpkmaster.library. This means that
any new packer that can only pack mem to mem has its own file format 
immediately. And most important: The name of the packer library is contained
in the file. Therefore, copying a new sub library to LIBS: is all you
have to do to install a new packer (easily done in installation scripts);
xpkmaster.library recognizes the new file type immediately. No changes to 
xpkmaster.library or the application programs necessary. In case the xpk
file format is not used, the introduction of a new packer requires a change
the xpkmaster.library.
 
 
THE EXISTING PACKERS
--------------------
Two excellent unpublished compression programs already use the xpk standard.
 
* BLZW (Bryan's LZW)
  Combines unseen compression speed with fast decompression speed and a
  resonable compression factor. Timings on the 25 MHz A3000:
 
  Compression speed  : 100-120K/sec
  Decompression speed: 200-250K/sec
  Compression factor : Like ZOO
 
* NUKE
  Designed for lightning fast decompression while retaining a good compresion
  factor.
 
  Compression speed  :  30- 40K/sec
  Decompression speed: 650-700K/sec
  Compression factor : Like LHarc, 1-2% worse
 
* PowerPacker
  Is supported for decompression only. This is a hack in the xpkmaster.library.
 
For comparison: The fastest packer so far (Zoo) packs 35K/sec, the fastest
unpacker so far (Imploder) decompresses 240K/sec. Detailed statistics are
available.
 
 
TYPICAL APPLICATIONS
--------------------
A few examples for applications that could use xpk:
* A GadTools based archiver interface that can deal with all archivers
* A CLI based file compressor/decompressor
* A hard disk backup utility that stores compressed data
* A 'more' program with automatic decompression
* A DTP program that stores its fonts in compressed format
* A network protocol with built in data compression for slow connections
* A hypertext utility that allows all data to be compressed
* A file handler that overlays an existing filesystem and uncompresses any 
  file while loading
...plus many more we don't even need to think about yet.
 
 
OTHER USES FOR XPK
------------------
Xpk could become an all purpose file conversion facility. E.g. it could
be used to convert JPEG images to IFF images while loading them into
DPaint, or encrypt files, or add redundancy to files for error correction.
 
 
CONCLUSION
----------
Xpk would increase the usefulness and flexibility of both application and 
compression programs while improving their user friendliness at the same
time.
 
 U. Dominik Mueller              umueller@amiga.physik.unizh.ch