From: nevries@accucx.cc.ruu.nl (Nico de Vries.)
Newsgroups: comp.compression
Subject: More on PKZIP 2.00 DMA support.
Date: 10 Apr 92 16:39:16 GMT
Organization: Academic Computer Centre Utrecht

Here a conversation on compuserve about PKZIP 2.0. I personally DON'T agree
with the conclusion that PKWARE does DMA in parralel with compression/
decompression based on this conversation. But maybe am I wrong and is PKWARE
going to surprize us with such an extremely fast implementation. PK has 
surprized us before (and it were always pleasant surprizes.

Thanks to Veldhuizen for sending me his CISTAP gathered information.


Nico de Vries
-----------------------------------------------------------------------------
USENET: nevries@cc.ruu.nl                 FIDO      : 2:281/708.1  (very new) 
VOICE : +31-3404-24931 (+31-30-316931)    FAX       : +31-30-312033 
BBS   : +31-30-341401 (Nico De_vries)     COMPUSERVE: (comming)
MAIL  : Hortensialaan 45, 3702VE, Zeist, The Netherlands 

IMPORT StdDisclaimer, MyPersonalOpinion;          - O   Place reserved for
Complaints about my spelling to DEV/NUL please.    |    funny joke. Sugestions 
I work for BITECH, I don't SPEAK for BITECH.      \_/   apreciated.

===============================================
#: 38771 S11/PKWare  [PCVENC]
    18-Mar-92  16:28:35
Sb: #38656-#FAILURE, BAD CRC VALUE
Fm: PKWARE - Douglas Hay 75300,730
To: Scott Derrick 76330,1640 (X)

PK is looking into the problem.  He now has a file that fails the CRC on 2 of
our 4 computers.  We all have the same computer, a Gateway 486/33!

He wrote a 50 line program that calculates the CRC, and it fails, so the bug
is not really a ZIP bug, or a CRC bug, it is some kind of memory copy bug
specific to 386/486's.

I'll post a message when I know more.

Doug




There is 1 Reply.

#: 38891 S11/PKWare  [PCVENC]
    19-Mar-92  12:41:42
Sb: #38771-#FAILURE, BAD CRC VALUE
Fm: Scott Derrick 76330,1640
To: PKWARE - Douglas Hay 75300,730 (X)

Douglas,

        Thanks...  I don't mean to be harsh or sound like I'm running down
PKWARE, I just feel like I'm up against a wall with not many options.
        Thanks for the assurance that you folks are trying to solve this
problem. It can only get bigger as more and more people buy up into 386/486
machines.
        If you need somebody to test a possible fix I have a couple of
machines that it fails on consistantly. I'd be happy to beta any solution you
come up with. You can address me directly  or send me mail
on the BPROGB forum. I'm there every day.

Scott

There is 1 Reply.

#: 38920 S11/PKWare  [PCVENC]
    19-Mar-92  16:34:10
Sb: #38891-#FAILURE, BAD CRC VALUE
Fm: PKWARE - Douglas Hay 75300,730
To: Scott Derrick 76330,1640 (X)

PK has created a "C" file that just does a "memcpy" and then compares the
source buffer with the destination buffer.  The two do not always match.  If
either the Internal 486 cache or the software cache are disabled, it works.

I will ask PK if I can upload the source code here.

Doug




There is 1 Reply.

#: 39313 S11/PKWare  [PCVENC]
    23-Mar-92  11:57:41
Sb: #PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: john pearson 70020,222

We have to work around this memory problem, and we don't know how long that
will take.  It will be at least a month.

Doug




There is 1 Reply.

#: 39471 S11/PKWare  [PCVENC]
    24-Mar-92  12:34:24
Sb: #39440-#PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Steve Bertz 73220,3140 (X)

Perhaps you guys could track down the memory nug we are having that is
hardware related <g>.  Then we could ship a whole lot sooner!

Doug




There is 1 Reply.

#: 39519 S11/PKWare  [PCVENC]
    24-Mar-92  19:47:36
Sb: #39471-#PKZip 2.0
Fm: Steven Katz 70304,2602
To: PKWARE - Douglas Hay 75300,730 (X)

What is this memory problem that you are having? Maybe some of us up here can
help you out.

There is 1 Reply.

#: 39593 S11/PKWare  [PCVENC]
    25-Mar-92  12:25:39
Sb: #39519-#PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Steven Katz 70304,2602 (X)

It is real easy to duplicate.

1. Open a file
2. Read 4K
3. Do a memcpy to another buffer
4. Compare the 2 buffers
5. Goto step 2

Sometimes the test fails, sometimes it doesn't.  This is a random problem,
only on 386/486's.  The 486 that it fails on here is a Gateway 486/33.  If
we boot with no autoexec.bat and no config.sys, it will still fail.  If we
disable the 486 internal cache, we encounter no problems.

Doug



There are 6 Replies.

#: 39672 S11/PKWare  [PCVENC]
    25-Mar-92  21:59:37
Sb: #39593-#PKZip 2.0
Fm: Sebouh H. Karakashian 74007,1007
To: PKWARE - Douglas Hay 75300,730 (X)

Some of the Gateway 80486/33 motherboards were known to have a problem... Did
you attempt to get a replacement, or did you try several other 80486's? I have
one (EISA) that I'd be willing to use to test it for you?

                                               Sebouh




There is 1 Reply.

Files busy, code 5-3(61) ...

Files busy, code 5-3(61) ...

Files busy, code 5-3(61) ...
 have files (38 sec)

#: 39739 S11/PKWare  [PCVENC]
    26-Mar-92  15:56:09
Sb: #39672-#PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Sebouh H. Karakashian 74007,1007 (X)

I read that the Gateways had motherboard problems, but this is reproducable on
other machines as well.

I'll try to post a test program soon.

Doug




There is 1 Reply.

#: 39782 S11/PKWare  [PCVENC]
    26-Mar-92  20:48:45
Sb: #39739-#PKZip 2.0
Fm: Sebouh H. Karakashian 74007,1007
To: PKWARE - Douglas Hay 75300,730 (X)

Good... we can give it a try...  BTW, does it only NOT work on a particular
Gateway 2000-486 or is it a problem on all Gateway 486's. or 80486's in
general.
                                               Sebouh



There is 1 Reply.

#: 39841 S11/PKWare  [PCVENC]
    27-Mar-92  12:50:00
Sb: #39782-PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Sebouh H. Karakashian 74007,1007 (X)

It happens on 2 or our 4 Gatways.  It has happened on other machine brands as
well.

Doug

#: 39792 S11/PKWare  [PCVENC]
    26-Mar-92  21:36:56
Sb: #39593-#PKZip 2.0
Fm: Steven Katz 70304,2602
To: PKWARE - Douglas Hay 75300,730 (X)

I tried the following program and I could not duplicate your problem.  Does
this program represent your problem. If it does,  I will try it on several
different types of machines at work.

1. Open a file 2. Read 4K 3. Do a memcpy to another buffer 4. Compare the 2
buffers 5. Goto step 2


------Start of Program-----#include <malloc.h> #include <stdio.h>


main() {
   FILE *fp;
   char *buffer1, *buffer2;
   int loop;

   buffer1 = (char *) malloc(4000);
   buffer2 = (char *) malloc(4000);
   fp = fopen("test.fil","r");

   for (loop=1;loop<=10;loop++) {
      fread(buffer1,4000,1,fp);
      memcpy(buffer2,buffer1,4000);
      memcmp(buffer2,buffer1,4000);
      printf("Test - %d\n",loop);
   }

   fclose(fp);
   free(buffer1);
   free(buffer2); } ------End of Program-----
There is 1 Reply.

#: 39842 S11/PKWare  [PCVENC]
    27-Mar-92  12:50:09
Sb: #39792-#PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Steven Katz 70304,2602 (X)

That program is very similar to ours.  Our program compares the buffer a byte
at a time though (don't know why).  I think PK wanted to only use a memcpy and
no other memory functions.

Like I have said before, it only happens on certain computers.  And it is not
reliable.  We run it from a batch file that calls itself.  We run it 100 times
and get 10 failures.  You must run several hundred tests.

Doug




There are 2 Replies.

#: 39910 S11/PKWare  [PCVENC]
    27-Mar-92  18:44:01
Sb: #39842-#PKZip 2.0
Fm: Gary Weinfurther 71521,2352
To: PKWARE - Douglas Hay 75300,730 (X)

  >> Our program compares the buffer a byte at a time though (don't     >>
know why).  I think PK wanted to only use a memcpy and no other     >> memory
functions.

Why not create your own memcmp() then, using some high-speed assembler (e.g.
REPNE SCASB)?  I don't know if it would fix the (apparent) cache problem, but
it certainly would be fast and small.

                                                        ...Gary


There is 1 Reply.

Files busy, code 5-3(61) ...
 have files (17 sec)

#: 40109 S11/PKWare  [PCVENC]
    30-Mar-92  11:17:32
Sb: #39910-PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Gary Weinfurther 71521,2352 (X)

Writing a new memcpy will probably have the same result.  We thought the cache
wasn't keeping up with the CPU, so we added a delay loop.  It had no effect.
I sent the code to intel.

Doug

#: 39930 S11/PKWare  [PCVENC]
    27-Mar-92  21:31:28
Sb: #39842-#PKZip 2.0
Fm: Steven Katz 70304,2602
To: PKWARE - Douglas Hay 75300,730 (X)

What happens when it does not work properly.

There is 1 Reply.

#: 40110 S11/PKWare  [PCVENC]
    30-Mar-92  11:17:39
Sb: #39930-PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Steven Katz 70304,2602 (X)

With PKUNZIP, the files report CRC errors.

Doug

#: 39820 S11/PKWare  [PCVENC]
    27-Mar-92  05:57:26
Sb: #39593-#PKZip 2.0
Fm: Paul  Jackson 70402,2417
To: PKWARE - Douglas Hay 75300,730 (X)

Your memory problem sounds alot like the cache coherency problems that we run
into developing workstations (Silicon Graphics, MIPS CPU and external cache).
I'd bet a Coke that the disk I/O doesn't update the cache in time to be seen
by the memcpy.  If so, this would only be seen on systems with external
caches, not for example on my cheapo 16 MHz 386SX.  And if that's so, I'd be
talking to whoever designed the external cache (Gateway, ...?), not Intel.

I'd also bet there's a variation of this test which will fail about as often:

        1. Open a file
        2. Read 4K
        3. Sum the 4K: for(i=0;i<4096/sizeof(int);i++)sum+=((int*)buf)[i];
        4. Except on first loop, compare sum to lastsum
        5. Goto step 2

I hope I haven't wasted too much net, speaking out of my area.

There is 1 Reply.

#: 39838 S11/PKWare  [PCVENC]
    27-Mar-92  12:20:49
Sb: #39593-PKZip 2.0
Fm: Paul  Jackson 70402,2417
To: PKWARE - Douglas Hay 75300,730 (X)

Walking further out on my limb ...

Let me both alter and extend my analogy with workstations.

The cache coherency problem happens when using DMA.  When I/O from the disk is
done with a CPU copy loop, the cache is kept current.  But some combination of
either cache flushing or non-cached CPU loads is needed for the CPU to safely
load data that has just been DMA'd into main memory perhaps bypassing the
cache.  This could happen with either external cache or onboard x86 cache (to
alter my story).

(I'm hearing a cracking sound, like that of a tree branch about to break ...)

#: 40301 S11/PKWare  [PCVENC]
    31-Mar-92  15:19:51
Sb: #40242-PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: Sam Baxter 76317,3274 (X)

We think the problem is in the cache on the motherboard, not the Intel cache.
It fails on 2 of the 17 machines we have here, both Gateway 486/33s.

This problem was reported to us, not something we discovered.

We uncovered MAJOR bugs in PCKWIK, Novell Netware and Above Disk.  Everyone
thought we were crazy when we said Novell had bugs that made PKZIP crash.

The files we have that make PKZIP fail also fail with ARJ and LHA.

Doug

P.S. This is a rare bug, I think it will be some faulty batch of cache chips.

#: 40459 S11/PKWare  [PCVENC]
    01-Apr-92  15:20:07
Sb: #40356-PKZip 2.0
Fm: PKWARE - Douglas Hay 75300,730
To: 'Fred' Kirby  [UK] 70734,126 (X)

I am sure PK will use some other number, but I don't think he will tell anyone
what that number is.

WHEN PKZIP 2.0 IS RELEASED, IT WILL BE IN THE PKWARE SECTION IN PCVENC.  So if
you don't see it here, don't believe what you see.  I will also make some type
of announcement a few days before hand.  It won't be a big secret like the
hacked versions are.

Maybe we should just call it 3.0!

Doug

#: 40744 S11/PKWare  [PCVENC]
    03-Apr-92  12:51:39
Sb: #40301-PKZip 2.0
Fm: Mick Ghazey 70550,1762
To: PKWARE - Douglas Hay 75300,730

Doug,

For what it's worth, I have experienced problems problems with Gateway 2000
486 33's that involved a memcpy but not a disk read. The problem occurred
when I was performing an LZW decode into my frame buffer. This is all my own
code, running with the PharLap DOS extender and VMM virtual memory. The
memcpy failed while copiyng from one location in the frame buffer to another.

The frame buffer sits on the XT bus. Is this a clue?

-Mick