Repository: facebook/zstd

Problem statement:
Assert in ZSTD_buildSeqStore when using ZSTD_compressContinue with a dictionary
Hello, I'm hitting the assert on [zstd_compress.c:2837 (v1.5.2)](https://github.com/Cyan4973/zstd/blob/v1.5.2/lib/compress/zstd_compress.c#L2837) when calling `ZSTD_compressContinue` with a dictionary (attached using `ZSTD_compressBegin_usingCDict`).

This code uses a manually-allocated circular buffer of `1 << (windowLog+1)` (I've reproduced with multiple window sizes including `windowLog == 17`) and compresses no more than half the buffer in each call to `ZSTD_compressContinue` to ensure that `1 << windowLog` bytes of previous data are always available.  The assert is only hit when a dictionary is attached (in this case the dictionary varies in size <= 128KB).

Does this sound like correct use of the buffer-less streaming API?  Are there any additional requirements/caveats around using this API with a dictionary?  Tracing through the code around this assert it looks like there are cases where the dictionary cannot be used -- when that happens, is there a potential impact on compression?

Thanks in advance for your help.  I've attached a log captured with `DEBUGLEVEL=6` and can work on trying to come up with a minimal test case to reproduce if needed.

Cheers,
 Greg

[test.log.gz](https://github.com/facebook/zstd/files/8336303/test.log.gz)


Hints:
An example of your usage of the compression API would be useful for debugging, though probably not 100% necessary.

What is your use-case for the buffer-less API? Could you use the streaming API `ZSTD_compressStream2()` instead? The buffer-less API is not super well supported, and I generally wouldn't recommend using it.

At some point in the future, I would like to deprecate the buffer-less API. And capture those use cases with the streaming API. So hearing how & why you're using this API would be super helpful.
Hi, and thanks for the response.  My eventual goal is to avoid large internal allocations and use a collection of externally provided buffers where I don't control the size, so it wouldn't be possible to just configure a custom memory allocator with `ZSTD_compressStream2`.

Patch:
diff --git a/lib/compress/zstd_compress_internal.h b/lib/compress/zstd_compress_internal.h
index 3b888acfa94..5b2792da3a2 100644
--- a/lib/compress/zstd_compress_internal.h
+++ b/lib/compress/zstd_compress_internal.h
@@ -1172,10 +1172,15 @@ ZSTD_checkDictValidity(const ZSTD_window_t* window,
                     (unsigned)blockEndIdx, (unsigned)maxDist, (unsigned)loadedDictEnd);
         assert(blockEndIdx >= loadedDictEnd);
 
-        if (blockEndIdx > loadedDictEnd + maxDist) {
+        if (blockEndIdx > loadedDictEnd + maxDist || loadedDictEnd != window->dictLimit) {
             /* On reaching window size, dictionaries are invalidated.
              * For simplification, if window size is reached anywhere within next block,
              * the dictionary is invalidated for the full block.
+             *
+             * We also have to invalidate the dictionary if ZSTD_window_update() has detected
+             * non-contiguous segments, which means that loadedDictEnd != window->dictLimit.
+             * loadedDictEnd may be 0, if forceWindow is true, but in that case we never use
+             * dictMatchState, so setting it to NULL is not a problem.
              */
             DEBUGLOG(6, "invalidating dictionary for current block (distance > windowSize)");
             *loadedDictEndPtr = 0;


Test patch:
diff --git a/tests/fuzzer.c b/tests/fuzzer.c
index 4a091c8972b..802b3937c56 100644
--- a/tests/fuzzer.c
+++ b/tests/fuzzer.c
@@ -2592,6 +2592,27 @@ static int basicUnitTests(U32 const seed, double compressibility)
         }
         DISPLAYLEVEL(3, "OK \n");
 
+        DISPLAYLEVEL(3, "test%3d : bufferless api with cdict : ", testNb++);
+        {   ZSTD_CDict* const cdict = ZSTD_createCDict(dictBuffer, dictSize, 1);
+            ZSTD_DCtx* const dctx = ZSTD_createDCtx();
+            ZSTD_frameParameters const fParams = { 0, 1, 0 };
+            size_t cBlockSize;
+            cSize = 0;
+            CHECK_Z(ZSTD_compressBegin_usingCDict_advanced(cctx, cdict, fParams, ZSTD_CONTENTSIZE_UNKNOWN));
+            cBlockSize = ZSTD_compressContinue(cctx, (char*)compressedBuffer + cSize, compressedBufferSize - cSize, CNBuffer, 1000);
+            CHECK_Z(cBlockSize);
+            cSize += cBlockSize;
+            cBlockSize = ZSTD_compressEnd(cctx, (char*)compressedBuffer + cSize, compressedBufferSize - cSize, (char const*)CNBuffer + 2000, 1000);
+            CHECK_Z(cBlockSize);
+            cSize += cBlockSize;
+
+            CHECK_Z(ZSTD_decompress_usingDict(dctx, decodedBuffer, CNBuffSize, compressedBuffer, cSize, dictBuffer, dictSize));
+
+            ZSTD_freeCDict(cdict);
+            ZSTD_freeDCtx(dctx);
+        }
+        DISPLAYLEVEL(3, "OK \n");
+
         DISPLAYLEVEL(3, "test%3i : Building cdict w/ ZSTD_dct_fullDict on a good dictionary : ", testNb++);
         {   ZSTD_compressionParameters const cParams = ZSTD_getCParams(1, CNBuffSize, dictSize);
             ZSTD_CDict* const cdict = ZSTD_createCDict_advanced(dictBuffer, dictSize, ZSTD_dlm_byRef, ZSTD_dct_fullDict, cParams, ZSTD_defaultCMem);