Newsgroups: comp.os.msdos.misc,alt.bbs
Subject: Compressing Usenet message base:  A test
From: mike@batpad.lgb.CA.US (Mike Batchelor)
Date: Mon, 13 Apr 92 22:16:15 PDT
Followup-To: comp.os.msdos.misc
Organization: The Batchelor Pad


I am in the process of putting together a Usenet uucp BBS, using
uuPCB and PCBoard software, and a dedicated 230Mb Quantum hard disk
for the message base.  I intend to install the Stacker disk
compression driver to get maximum space out of the Quantum, since I
want to carry as full a newsfeed as I can get.  I didn't know,
however, if I would have enough space to carry the binaries groups,
since I had no idea how well uuencoded files would compress.  So I
performed some tests, using several MSDOS compression utilities,
figuring that Stacker will get compression ratios somewhere in the
range between compress 4.3d and ARJ 2.30.

Compression utilities used for the test:

compress 4.3d
PKZip 1.10 (option -ex: maximum compression)
LHA 2.13
ARJ 2.30 (option -m1: maximum compression)

Test files:

testtar - a .tar file made by tarring together all the pieces of the
          uuencoded DJGPP.ZIP distribution, a total of 59 x 60Kb
          uuencoded files.

uuexetar - I created this file by tarring together the .exe and .com
           files in several directories, then uuencoding the resulting
           .tar file into a single uuencoded file.  I have not used any
           executable compression utilities on these files, although
           some of them may have been compressed by the vendor.

newstar - a .tar file made by tarring up my entire Waffle 1.64 message
          base (small newsfeed).  This should compare well to the results
          I would get compressing PCBoard's message base on the Stacker
          volume, since PCBoard uses one file per newsgroup (conference),
          instead of one file per article, as with Waffle.

                            =======
                            RESULTS
                            =======

testtar       3389440 - .tar file of uuencoded DJGPP.ZIP pieces
testtar.arj   2436524 - 28.11%, 1:1.39 compression ratio
testtar.lzh   2459566 - 27.43%, 1:1.38 compression ratio
testtar.z     2971527 - 12.33%, 1:1.14 compression ratio
testtar.zip   3030787 - 10.58%, 1:1.12 compression ratio

Clearly, Stacker is going to have some problems saving any significant
disk space with uuencoded, compressed binaries.  I expect similar results
when compressing uuencoded GIF images, since they are also compressed.

--------------------------------------------------------------------------
uuexetar      9046209 - uuencoded .tar file of uncompressed binaries
uuexetar.arj  4910345 - 45.72%, 1:1.84 compression ratio
uuexetar.lzh  4969230 - 45.07%, 1:1.82 compression ratio
uuexetar.z    6061383 - 32.99%, 1:1.49 compression ratio
uuexetar.zip  5743593 - 36.51%, 1:1.58 compression ratio

Little bit better compression all-round with uncompressed, uuencoded
binaries, but still not a whole lot.  PKZip did a lot better on this
test.  I was kind of surprised that it did so poorly on testtar,
compared to compress 4.3d.

BTW, PKZip managed to lock up the computer trying to compress
uuexetar, inside a DESQview BigDOS window.  After resetting, it had no
problem from plain DOS.  None of the other compressors had this
problem, but they all took quite some time to finish!

--------------------------------------------------------------------------
newstar       9656320 - .tar file of Waffle news directory
newstar.arj   2737356 - 71.65%, 1:3.54 compression ratio
newstar.lzh   3227667 - 66.57%, 1:2.99 compression ratio
newstar.z     3741481 - 61.25%, 1:2.58 compression ratio
newstar.zip   3332335 - 65.49%, 1:2.90 compression ratio

I'm very encouraged by these results.  Since Stacker uses basically
the same LZW-derivative algorithm as all of the other compressors, it
looks like I can expect at least a 1:2.5 compression ratio for the
plain text newsgroups, or an effective message base size of 575 Mb;
maybe as much as 805 Mb (yeah, yeah, wishful thinking!).


Bottom line:  I won't be carrying any binaries newsgroups.  I might see
if I can squeeze in a pictures group or two.  The sources newsgroups
should compress rather well, I would think.

Now to clean up the 60-odd megabytes of test files I made...


[] ---
[] Mike Batchelor -- mike@batpad.lgb.CA.US -- cerritos.edu!batpad!mike
[] Long Beach, California
[] TNG Quote: "You were travelling in close formation at over 80,000 kph, only
[] 10 meters from Cadet Albert, and you didn't even know his orientation?!?"