Repository: facebook/zstd

Problem statement:
Fast Mode UX is Poor
Every recent benchmark of ZSTD that I can find skips "fast mode" ([lzbench][lzbench], [squash][squash], [peazip][peazip], ***[zstd][zstd]***).

This is partly because most of the material about ZSTD was written before the negative ranges were added.  People just don't know this and you need to advertise this capability.  Maybe you could break your benchmark table and graphs into 3 ranges, like is done [here](https://engineering.fb.com/core-data/zstandard/):

![image](https://user-images.githubusercontent.com/44454/80927595-9d5ad880-8d53-11ea-9495-d82acb5016c4.png)

The other issue is that users don't read manuals.  While I understand hiding the highest compression levels behind `--ultra` due to spiking client memory consumption, the `--fast=[#]` is just there to accommodate CLI formatting convention and hides the higher speed levels from end users:

> No extra compression settings were used to fine tune the compression, neither for Brotli nor for Zstandard, compression was defined exclusively setting the compression level parameter. Both algorithms features plenty fine-tuning options, which are out of the scope of this benchmark.
> -[PeaZIP Blog][peazip]

I would strongly suggest the following:

- [ ]  Accept negative integers (i.e. `--5` and maybe `- -5`) for the compression level.
- [ ]  Deprecate (but probably never actually remove) the `--fast` flag to discourage this being seen as fine tuning.
- [ ] Update all [documentation](https://github.com/facebook/zstd/blob/4a4018870f48ae859a5fb730ef4ff1e1bd79e6ba/programs/zstd.1) to state the full range of levels.
- [ ] Highlight the full range on the website and readme.md, perhaps by breaking up the benchmarks into fast, normal, and ultra sections.  Make sure to explicitly state the range being from -5 to 22.

[lzbench]: https://github.com/inikep/lzbench/issues/68
[squash]: https://github.com/quixdb/squash/issues/251
[peazip]: https://www.peazip.org/fast-compression-benchmark-brotli-zstandard.html
[zstd]: https://github.com/facebook/zstd/issues/2098
Fast Mode UX is Poor
Every recent benchmark of ZSTD that I can find skips "fast mode" ([lzbench][lzbench], [squash][squash], [peazip][peazip], ***[zstd][zstd]***).

This is partly because most of the material about ZSTD was written before the negative ranges were added.  People just don't know this and you need to advertise this capability.  Maybe you could break your benchmark table and graphs into 3 ranges, like is done [here](https://engineering.fb.com/core-data/zstandard/):

![image](https://user-images.githubusercontent.com/44454/80927595-9d5ad880-8d53-11ea-9495-d82acb5016c4.png)

The other issue is that users don't read manuals.  While I understand hiding the highest compression levels behind `--ultra` due to spiking client memory consumption, the `--fast=[#]` is just there to accommodate CLI formatting convention and hides the higher speed levels from end users:

> No extra compression settings were used to fine tune the compression, neither for Brotli nor for Zstandard, compression was defined exclusively setting the compression level parameter. Both algorithms features plenty fine-tuning options, which are out of the scope of this benchmark.
> -[PeaZIP Blog][peazip]

I would strongly suggest the following:

- [ ]  Accept negative integers (i.e. `--5` and maybe `- -5`) for the compression level.
- [ ]  Deprecate (but probably never actually remove) the `--fast` flag to discourage this being seen as fine tuning.
- [ ] Update all [documentation](https://github.com/facebook/zstd/blob/4a4018870f48ae859a5fb730ef4ff1e1bd79e6ba/programs/zstd.1) to state the full range of levels.
- [ ] Highlight the full range on the website and readme.md, perhaps by breaking up the benchmarks into fast, normal, and ultra sections.  Make sure to explicitly state the range being from -5 to 22.

[lzbench]: https://github.com/inikep/lzbench/issues/68
[squash]: https://github.com/quixdb/squash/issues/251
[peazip]: https://www.peazip.org/fast-compression-benchmark-brotli-zstandard.html
[zstd]: https://github.com/facebook/zstd/issues/2098


Hints:
We could certainly improve visibility of the `--fast` mode feature, through a combination of better and more prominent documentation, like the benchmark table.

However, the proposed command line change `--5` is problematic, as `--anything` is a commonly accepted idiom meaning "this is a long command", which purpose is to be descriptive.
`- -5` is very different. It currently has the following meaning : "read from `stdin` and apply level 5". The meaning can't be changed without breaking interface.

With regards to existing restrictions, `--fast=#` still looks the better option.
> However, the proposed command line change --5 is problematic, as --anything is a commonly accepted idiom meaning "this is a long command", which purpose is to be descriptive.

I know, but this is **just** convention, right?  It's not going to break anything or cause Dennis Ritchie to send down lightning bolts or anything 😸.

> `- -5` is very different. It currently has the following meaning : "read from stdin and apply level 5".

Yeah, that was a stupid suggestion on my part.
> I know, but this is just convention, right? It's not going to break anything or cause Dennis Ritchie to send down lightning bolts or anything 😸.

Except it would break `-abc`.  What about a `level=[-5, 20]` and `L=[-5, 20]`?
I get that the faster versions are considered the "negative" levels, but do they need to be notated as this for the CLI? Maybe a different identifier could be used to denote using a fast level. Maybe `#`? (i.e. `zstd -#2` for `zstd --fast=2`)
What's wrong with `--fast=N` ?
Mainly, any other functionality that is expecting a level (`--adapt`, `-b` and `-e`, etc) can't use fast speeds.
They can : 
`zstd -b --fast=3 file`
`zstd --adapt --fast=3 file`

On the other hand, `-e` is indeed not compatible.
But what if I want to limit my adapt to -3 to 3? `--adapt-min=-3,max=3` doesn't work (no negative numbers work).

The `--fast` works for basic functionality, but you can't combine any of the "advanced" features (adapt, fast, ultra, benchmarking, etc) together.

I get that fast is likely to be less used than the normal levels, but allowing for the "negative" levels to be used in all areas would be nice.
It might be possible to add support of negative compression levels for explicit integer parameters such as `--long-param=-3`.

What's less interesting is specifically the initial `--3` proposal, which conflicts with `--long-command` convention. But this one can still be expressed as `--fast=3`.
We could certainly improve visibility of the `--fast` mode feature, through a combination of better and more prominent documentation, like the benchmark table.

However, the proposed command line change `--5` is problematic, as `--anything` is a commonly accepted idiom meaning "this is a long command", which purpose is to be descriptive.
`- -5` is very different. It currently has the following meaning : "read from `stdin` and apply level 5". The meaning can't be changed without breaking interface.

With regards to existing restrictions, `--fast=#` still looks the better option.
> However, the proposed command line change --5 is problematic, as --anything is a commonly accepted idiom meaning "this is a long command", which purpose is to be descriptive.

I know, but this is **just** convention, right?  It's not going to break anything or cause Dennis Ritchie to send down lightning bolts or anything 😸.

> `- -5` is very different. It currently has the following meaning : "read from stdin and apply level 5".

Yeah, that was a stupid suggestion on my part.
> I know, but this is just convention, right? It's not going to break anything or cause Dennis Ritchie to send down lightning bolts or anything 😸.

Except it would break `-abc`.  What about a `level=[-5, 20]` and `L=[-5, 20]`?
I get that the faster versions are considered the "negative" levels, but do they need to be notated as this for the CLI? Maybe a different identifier could be used to denote using a fast level. Maybe `#`? (i.e. `zstd -#2` for `zstd --fast=2`)
What's wrong with `--fast=N` ?
Mainly, any other functionality that is expecting a level (`--adapt`, `-b` and `-e`, etc) can't use fast speeds.
They can : 
`zstd -b --fast=3 file`
`zstd --adapt --fast=3 file`

On the other hand, `-e` is indeed not compatible.
But what if I want to limit my adapt to -3 to 3? `--adapt-min=-3,max=3` doesn't work (no negative numbers work).

The `--fast` works for basic functionality, but you can't combine any of the "advanced" features (adapt, fast, ultra, benchmarking, etc) together.

I get that fast is likely to be less used than the normal levels, but allowing for the "negative" levels to be used in all areas would be nice.
It might be possible to add support of negative compression levels for explicit integer parameters such as `--long-param=-3`.

What's less interesting is specifically the initial `--3` proposal, which conflicts with `--long-command` convention. But this one can still be expressed as `--fast=3`.

Patch:
diff --git a/programs/zstdcli.c b/programs/zstdcli.c
index 9e2133c4f86..28fc980a2bf 100644
--- a/programs/zstdcli.c
+++ b/programs/zstdcli.c
@@ -206,6 +206,7 @@ static void usage_advanced(const char* programName)
     DISPLAYOUT( "--ultra : enable levels beyond %i, up to %i (requires more memory) \n", ZSTDCLI_CLEVEL_MAX, ZSTD_maxCLevel());
     DISPLAYOUT( "--long[=#]: enable long distance matching with given window log (default: %u) \n", g_defaultMaxWindowLog);
     DISPLAYOUT( "--fast[=#]: switch to very fast compression levels (default: %u) \n", 1);
+    DISPLAYOUT( "--long-param=#: specify compression level, accepts negative values as fast compression levels \n");
     DISPLAYOUT( "--adapt : dynamically adapt compression level to I/O conditions \n");
     DISPLAYOUT( "--[no-]row-match-finder : force enable/disable usage of fast row-based matchfinder for greedy, lazy, and lazy2 strategies \n");
     DISPLAYOUT( "--patch-from=FILE : specify the file to be used as a reference point for zstd's diff engine. \n");
@@ -354,6 +355,25 @@ static unsigned readU32FromChar(const char** stringPtr) {
     return result;
 }
 
+#ifndef ZSTD_NOCOMPRESS
+/*! readIntFromChar() :
+ * @return : signed integer value read from input in `char` format.
+ *  allows and interprets K, KB, KiB, M, MB and MiB suffix.
+ *  Will also modify `*stringPtr`, advancing it to position where it stopped reading.
+ *  Note : function will exit() program if digit sequence overflows */
+static int readIntFromChar(const char** stringPtr) {
+    static const char errorMsg[] = "error: numeric value overflows 32-bit int";
+    int sign = 1;
+    unsigned result;
+    if (**stringPtr=='-') {
+        (*stringPtr)++;
+        sign = -1;
+    }
+    if (readU32FromCharChecked(stringPtr, &result)) { errorOut(errorMsg); }
+    return (int) result * sign;
+}
+#endif
+
 /*! readSizeTFromCharChecked() :
  * @return 0 if success, and store the result in *value.
  *  allows and interprets K, KB, KiB, M, MB and MiB suffix.
@@ -940,23 +960,6 @@ int main(int const argCount, const char* argv[])
                 if (longCommandWArg(&argument, "--trace")) { char const* traceFile; NEXT_FIELD(traceFile); TRACE_enable(traceFile); continue; }
 #endif
                 if (longCommandWArg(&argument, "--patch-from")) { NEXT_FIELD(patchFromDictFileName); continue; }
-                if (longCommandWArg(&argument, "--long")) {
-                    unsigned ldmWindowLog = 0;
-                    ldmFlag = 1;
-                    /* Parse optional window log */
-                    if (*argument == '=') {
-                        ++argument;
-                        ldmWindowLog = readU32FromChar(&argument);
-                    } else if (*argument != 0) {
-                        /* Invalid character following --long */
-                        badusage(programName);
-                        CLEAN_RETURN(1);
-                    }
-                    /* Only set windowLog if not already set by --zstd */
-                    if (compressionParams.windowLog == 0)
-                        compressionParams.windowLog = ldmWindowLog;
-                    continue;
-                }
 #ifndef ZSTD_NOCOMPRESS   /* linking ZSTD_minCLevel() requires compression support */
                 if (longCommandWArg(&argument, "--fast")) {
                     /* Parse optional acceleration factor */
@@ -981,8 +984,44 @@ int main(int const argCount, const char* argv[])
                     }
                     continue;
                 }
+
+                if (longCommandWArg(&argument, "--long-param")) {
+                    if (*argument == '=') {
+                        int maxLevel = ZSTD_maxCLevel();
+                        int minLevel = ZSTD_minCLevel();
+                        int readLevel;
+                        ++argument;
+                        readLevel = readIntFromChar(&argument);
+                        if (readLevel > maxLevel) readLevel = maxLevel;
+                        if (readLevel < minLevel) readLevel = minLevel;
+                        cLevel = readLevel;
+                    } else {
+                        /* --long-param requires an argument */
+                        badusage(programName);
+                        CLEAN_RETURN(1);
+                    }
+                    continue;
+                }
 #endif
 
+                if (longCommandWArg(&argument, "--long")) {
+                    unsigned ldmWindowLog = 0;
+                    ldmFlag = 1;
+                    /* Parse optional window log */
+                    if (*argument == '=') {
+                        ++argument;
+                        ldmWindowLog = readU32FromChar(&argument);
+                    } else if (*argument != 0) {
+                        /* Invalid character following --long */
+                        badusage(programName);
+                        CLEAN_RETURN(1);
+                    }
+                    /* Only set windowLog if not already set by --zstd */
+                    if (compressionParams.windowLog == 0)
+                        compressionParams.windowLog = ldmWindowLog;
+                    continue;
+                }
+
                 if (longCommandWArg(&argument, "--filelist")) {
                     const char* listName;
                     NEXT_FIELD(listName);


Test patch:
diff --git a/tests/playTests.sh b/tests/playTests.sh
index 25293900656..09dae1a297f 100755
--- a/tests/playTests.sh
+++ b/tests/playTests.sh
@@ -191,6 +191,13 @@ zstd --fast=3 -f tmp  # == -3
 zstd --fast=200000 -f tmp  # too low compression level, automatic fixed
 zstd --fast=5000000000 -f tmp && die "too large numeric value : must fail"
 zstd -c --fast=0 tmp > $INTOVOID && die "--fast must not accept value 0"
+println "test : --long-param compression levels"
+zstd --long-param=1 -f tmp
+zstd --long-param=0 -f tmp
+zstd --long-param=-1 -f tmp
+zstd --long-param=-10000 -f tmp # too low, automatic fixed
+zstd --long-param=10000 -f tmp # too high, automatic fixed
+zstd --long-param -f tmp > $INTOVOID && die "--long-param must be given a value"
 println "test : too large numeric argument"
 zstd --fast=9999999999 -f tmp  && die "should have refused numeric value"
 println "test : set compression level with environment variable ZSTD_CLEVEL"