neulab/tldr huggingface dataset:
• Nolan Woods <nolan_w@sfu.ca>
• Gautam Kumar <25435568+gautamajay52@users.noreply.github.com>
• Chris Macklin <chris.macklin@10xgenomics.com>
• Antoon Prins <antoon.prins@surfsara.nl>
• Alexey Ivanov <rbtz@dropbox.com>
• Serge Pouliquen <sp31415@free.fr>
• acsfer <carlos@reendex.com>
• Tom <tom@tom-fitzhenry.me.uk>
• Tyson Moore <tyson@tyson.me>
• database64128 <free122448@hotmail.com>
• Chris Lu <chrislusf@users.noreply.github.com>
• Reid Buzby <reid@rethink.software>
• darrenrhs <darrenrhs@gmail.com>
• Florian Penzkofer <fp@nullptr.de>
• Xuanchen Wu <117010292@link.cuhk.edu.cn>
• partev <petrosyan@gmail.com>
• Dmitry Sitnikov <fo2@inbox.ru>
• Haochen Tong <i@hexchain.org>
• Michael Hanselmann <public@hansmi.ch>
• Chuan Zh <zhchuan7@gmail.com>
• Antoine GIRARD <antoine.girard@sapk.fr>
• Justin Winokur (Jwink3101) <Jwink3101@users.noreply.github.com>
• Mariano Absatz (git) <scm@baby.com.ar>
• Greg Sadetsky <lepetitg@gmail.com>
• yedamo <logindaveye@gmail.com>
• hota <lindwurm.q@gmail.com>
• vinibali <vinibali1@gmail.com>
• Ken Enrique Morel <ken.morel.santana@gmail.com>
• Justin Hellings <justin.hellings@gmail.com>
• Parth Shukla <pparth@pparth.net>
• wzl <wangzl31@outlook.com>
• HNGamingUK <connor@earnshawhome.co.uk>
• Jonta <359397+Jonta@users.noreply.github.com>
• YenForYang <YenForYang@users.noreply.github.com>
• Joda Stößer <stoesser@yay-digital.de> <services+github@simjo.st>
• Logeshwaran <waranlogesh@gmail.com>
• Rajat Goel <rajat@dropbox.com>
• r0kk3rz <r0kk3rz@gmail.com>
• Matthew Sevey <mjsevey@gmail.com>
• Filip Rysavy <fil@siasky.net>
• Ian Levesque <ian@ianlevesque.org>
• Thomas Stachl <thomas@stachl.me>
• Dmitry Bogatov <git#v1@kaction.cc>
• trevyn <trevyn-git@protonmail.com>
• David Liu <david.yx.liu@oracle.com>
Contact the rclone project Forum Forum for questions and general discussion:
• https://forum.rclone.org
GitHub repository The project's repository is located at:
• https://github.com/rclone/rclone
There you can file bug reports or contribute with pull requests.
Twitter You can also follow me on twitter for rclone announcements:
• [@njcw](https://twitter.com/njcw)
Email Or if all else fails or you want to ask something private or confidential email Nick Craig-Wood (mailto:nick@craig-wood.com).  Please don't email me requests for help - those are better directed to the forum.  Thanks!
Nick Craig-Wood.
steghide(1)                  General Commands Manual                 steghide(1)
steghide - a steganography program
steghide command [ arguments ]
Steghide is a steganography program that is able to hide data in various kinds of image- and audio-files. The color- respectivly sample- frequencies are not changed thus making the embedding resistant against first-order statistical tests.
Features include the compression of the embedded data, encryption of the embedded data and automatic integrity checking using a checksum. The JPEG, BMP, WAV and AU file formats are supported for use as cover file. There are no restrictions on the format of the secret data.
Steghide uses a graph-theoretic approach to steganography. You do not need to know anything about graph theory to use steghide and you can safely skip the rest of this paragraph if you are not interested in the technical details. The embedding algorithm roughly works as follows: At first, the secret data is compressed and encrypted. Then a sequence of postions of pixels in the cover file is created based on a pseudo-random number generator initialized with the passphrase (the secret data will be embedded in the pixels at these positions). Of these positions those that do not need to be changed (because they already contain the correct value by chance) are sorted out. Then a graph-theoretic matching algorithm finds pairs of positions such that exchanging their values has the effect of embedding the corresponding part of the secret data. If the algorithm cannot find any more such pairs all exchanges are actually performed. The pixels at the remaining positions (the positions that are not part of such a pair) are also modified to contain the embedded data (but this is done by overwriting them, not by exchanging them with other pixels).  The fact that (most of) the embedding is done by exchanging pixel values implies that the first-order statistics (i.e. the number of times a color occurs in the picture) is not changed. For audio files the algorithm is the same, except that audio samples are used instead of pixels.
The default encryption algorithm is Rijndael with a key size of 128 bits (which is AES - the advanced encryption standard) in the cipher block chaining mode. If you do not trust this combination for whatever reason feel free to choose another algorithm/mode combination (information about all possible algorithms and modes is displayed by the encinfo command). The checksum is calculated using the CRC32 algorithm.
In this section the commands for steghide are listed. The first argument must always be one of these commands. You can supply additional arguments to the embed, extract and info commands. The other commands to not take any arguments.
embed, --embed Embed secret data in a cover file thereby creating a stego file.
extract, --extract Extract secret data from a stego file.
info, --info Display information about a cover or stego file.
encinfo, --encinfo Display a list of encryption algorithms and modes that can be used. No arguments required.
version, --version Display short version information. No arguments required.
license, --license Display steghide's license. No arguments required.
help, --help Display a help screen. No arguments required.
You should use the embed command if you want to embed secret data in a cover file. The following arguments can be used with the embed command:
-ef, --embedfile filename Specify the file that will be embedded (the file that contains the secret message). Note that steghide embeds the original file name in the stego file. When extracting data (see below) the default behaviour is to save the embedded file into the current directory under its original name. If this argument is omitted or filename is -, steghide will read the secret data from standard input.
-cf, --coverfile filename Specify the cover file that will be used to embed data. The cover file must be in one of the following formats: AU, BMP, JPEG or WAV. The file-format will be detected automatically based on header information (the extension is not relevant). If this argument is omitted or filename is -, steghide will read the cover file from standard input.
-sf, --stegofile filename Specify the name for the stego file that will be created. If this argument is omitted when calling steghide with the embed command, then the modifications to embed the secret data will be made directly to the cover file without saving it under a new name.
-e, --encryption algo [ mode ] | mode [ algo ] Specify encryption parameters. This option must be followed by one or two strings that identify an encryption algorithm and/or mode. You can get the names of all available algorithms and supported modes with the encinfo command. The default encryption is rijndael-128 (AES) in the cbc mode.  If you do not want to use any encryption, use -e none.
-z, --compress level Specify the compression level. The compression level can be any number in 1...9 where 1 means best speed and 9 means best compression.
-Z, --dontcompress Do not compress the secret data before embedding it.
-K, --nochecksum Do not embed a CRC32 checksum. You can use this if the secret data already contains some type of checksum or if you do not want to embed those extra 32 bits needed for the checksum.
-N, --dontembedname Do not embed the file name of the secret file. If this option is used, the extractor needs to specify a filename to tell steghide where to write the embedded data.
If you have received a file that contains a message that has been embedded with steghide, use the extract command to extract it. The following arguments can be used with this command.
-sf, --stegofile filename Specify the stego file (the file that contains embedded data). If this argument is omitted or filename is -, steghide will read a stego file from standard input.
-xf, --extractfile filename Create a file with the name filename and write the data that is embedded in the stego file to it. This option overrides the filename that is embedded int the stego file. If this argument is omitted, the embedded data will be saved to the current directory under its original name.
GETTING INFORMATION ABOUT A COVER/STEGO FILE You can use the info command to get some information about a cover or stego file (for example the capacity). You might want to use this if you have received a file and you are not sure if it contains an embedded message or if you consider using a certain file as cover file and want to find out its capacity.
The command line steghide info <filename> will print information about <filename> and then ask you if you would like to get information about data that is embedded in that file. If you answer with yes you have to supply the passphrase that was used to embed the data in that file.
You can also supply the -p, --passphrase argument (see below) to the info command which has the effect that steghide will automatically try to get information about the data that has been embedded using the given passphrase.
The following options can be used with all commands (where it makes sense).
-p, --passphrase Use the string following this argument as the passphrase. If your passphrase contains whitespace, you have to enclose it in quotes, for example: -p "a very long passphrase".
-v, --verbose Display detailed information about the status of the embedding or extracting process.
-q, --quiet Supress information messages.
-f, --force Always overwrite existing files.
All file name arguments (-cf, -ef, -sf, -xf) also accept - as a filename which makes steghide use standard input or standard output (whichever makes sense). Omitting the corresponding file name argument will have the same effect as using - with two exceptions: If -sf is omitted for the embed command, then the modifications will be done directly in the cover file. If -xf is omitted for extraction, then the embedded data will be saved under the file name that is embedded in the stego file.  So when you want to be sure that standard input/output is used, use - as filename.
The basic usage is as follows:
$ steghide embed -cf picture.jpg -ef secret.txt Enter passphrase: Re-Enter passphrase: embedding "secret.txt" in "picture.jpg"... done
This command will embed the file secret.txt in the cover file picture.jpg.
After you have embedded your secret data as shown above you can send the file picture.jpg to the person who should receive the secret message. The receiver has to use steghide in the following way:
$ steghide extract -sf picture.jpg Enter passphrase: wrote extracted data to "secret.txt".
If the supplied passphrase is correct, the contents of the original file secret.txt will be extracted from the stego file picture.jpg and saved in the current directory.
If you have received a file that contains embedded data and you want to get some information about it before extracting it, use the info command:
$ steghide info received_file.wav "received_file.wav": format: wave audio, PCM encoding capacity: 3.5 KB Try to get information about embedded data ? (y/n) y Enter passphrase: embedded file "secret.txt": size: 1.6 KB encrypted: rijndael-128, cbc compressed: yes
After printing some general information about the stego file (format, capacity) you will be asked if steghide should try to get information about the embedded data. If you answer with yes you have to supply a passphrase. Steghide will then try to extract the embedded data with that passphrase and - if it succeeds - print some information about it.
Steghide returns 0 on success and 1 if a failure occured and it had to terminate before completion of the requested operation. Warnings do not have an effect on the return value.
Stefan Hetzl <shetzl@chello.at>
GUIX(PACKAGE)                           1                          GUIX(PACKAGE)
guix package (GNU Guix) - manual page for guix package (GNU Guix) 1.2.0rc2
guix package [OPTION]...
Install, remove, or upgrade packages in a single transaction.
-i, --install PACKAGE ... install PACKAGEs
-e, --install-from-expression=EXP install the package EXP evaluates to
-f, --install-from-file=FILE install the package that the code within FILE evaluates to
-r, --remove PACKAGE ... remove PACKAGEs
-u, --upgrade[=REGEXP] upgrade all the installed packages matching REGEXP
-m, --manifest=FILE create a new profile generation with the manifest from FILE
--do-not-upgrade[=REGEXP] do not upgrade any packages matching REGEXP
--roll-back roll back to the previous generation
--search-paths[=KIND] display needed environment variable definitions
-l, --list-generations[=PATTERN] list generations matching PATTERN
-d, --delete-generations[=PATTERN] delete generations matching PATTERN
-S, --switch-generation=PATTERN switch to a generation matching PATTERN
-p, --profile=PROFILE use PROFILE instead of the user's default profile
--list-profiles list the user's profiles
--allow-collisions do not treat collisions in the profile as an error
--bootstrap use the bootstrap Guile to build the profile
-v, --verbosity=LEVEL use the given verbosity LEVEL
-s, --search=REGEXP search in synopsis and description using REGEXP
-I, --list-installed[=REGEXP] list installed packages matching REGEXP
-A, --list-available[=REGEXP] list available packages matching REGEXP
--show=PACKAGE show details about PACKAGE
-L, --load-path=DIR prepend DIR to the package module search path
-K, --keep-failed keep build tree of failed builds
-k, --keep-going keep going when some of the derivations fail
-n, --dry-run do not build the derivations
--fallback fall back to building when the substituter fails
--no-substitutes build instead of resorting to pre-built substitutes
--substitute-urls=URLS fetch substitute from URLS if they are authorized
--no-grafts do not graft packages
--no-offload do not attempt to offload builds
--max-silent-time=SECONDS mark the build as failed after SECONDS of silence
--timeout=SECONDS mark the build as failed after SECONDS of activity
--rounds=N build N times in a row to detect non-determinism
-c, --cores=N allow the use of up to N CPU cores for the build
-M, --max-jobs=N allow at most N build jobs
--debug=LEVEL produce debugging output at LEVEL
--help-transform list package transformation options not shown here
-V, --version display version information and exit
Report bugs to: bug-guix@gnu.org. GNU Guix home page: <https://www.gnu.org/software/guix/> General help using Guix and GNU software: <https://guix.gnu.org/help/>
Copyright Â© 2020 the Guix authors License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.
The full documentation for guix package (GNU Guix) is maintained as a Texinfo manual.  If the info and guix package (GNU Guix) programs are properly installed at your site, the command
info guix
POPD(8)                      System Manager's Manual                     POPD(8)
>> Included manual page: /usr/lib/mh/tmac.h
popd - the POP server
/usr/etc/popd [-p portno] (under /etc/rc.local)
The popd server implements the Post Office Protocol (version 3), as described in RFC1081 and RFC1082.  Basically, the server listens on the TCP port named “pop” for connections and enters the POP upon establishing a connection.  The `-p' option overrides the default TCP port.
If the POP2 configuration option is defined, then the server also implements version 2 of the protocol.
If the APOP configuration option is defined, then the server supports a non-standard mechanism for identity-establishment in which authentication credentials are used to provide for origin authentication and reply protection, but which do not involve sending a password in the clear over the network.  See popauth(8) for more details.  ^/usr/spool/pop/POP~^POP database None Post Office Protocol - version 3 (aka RFC-1081), Post Office Protocol - version 3: Extended service offerings (RFC-1082), pop(5) None None For historical reasons, the MH POP defaults to using the port named “pop” (109) instead of its newly assigned port named “pop3” (110).  See the POPSERVICE configuration option for more details.
Previous versions of the server (10/28/84) had the restriction that the POP client may retrieve messages for login users only.  This restriction has been lifted, and true POB support is available (sending mail to a mailbox on the POP service host which does not map to a user-id in the password file).
DATAMASH(1)                       User Commands                      DATAMASH(1)
datamash - command-line calculations
datamash [OPTION] op [fld] [op fld ...]
Performs numeric/string operations on input from stdin.
'op' is the operation to perform.  If a primary operation is used, it must be listed first, optionally followed by other operations.  'fld' is the input field to use.  'fld' can be a number (1=first field), or a field name when using the -H or --header-in options.  Multiple fields can be listed with a comma (e.g. 1,6,8).  A range of fields can be listed with a dash (e.g. 2-8).  Use colons for operations which require a pair of fields (e.g. 'pcov 2:6').
Primary operations: groupby, crosstab, transpose, reverse, check
Line-Filtering operations: rmdup
Per-Line operations: base64, debase64, md5, sha1, sha224, sha256, sha384, sha512, bin, strbin, round, floor, ceil, trunc, frac, dirname, basename, barename, extname, getnum, cut
Numeric Grouping operations: sum, min, max, absmin, absmax, range
Textual/Numeric Grouping operations: count, first, last, rand, unique, collapse, countunique
Statistical Grouping operations: mean, geomean, harmmean, trimmean, median, q1, q3, iqr, perc, mode, antimode, pstdev, sstdev, pvar, svar, mad, madraw, pskew, sskew, pkurt, skurt, dpo, jarque, scov, pcov, spearson, ppearson
Grouping Options: -C, --skip-comments skip comment lines (starting with '#' or ';' and optional whitespace)
-f, --full print entire input line before op results (default: print only the grouped keys)
-g, --group=X[,Y,Z] group via fields X,[Y,Z]; equivalent to primary operation 'groupby'
--header-in first input line is column headers
--header-out print column headers as first line
-H, --headers same as '--header-in --header-out'
-i, --ignore-case ignore upper/lower case when comparing text; this affects grouping, and string operations
-s, --sort sort the input before grouping; this removes the need to manually pipe the input through 'sort'
File Operation Options: --no-strict allow lines with varying number of fields
--filler=X fill missing values with X (default %s)
General Options: -t, --field-separator=X use X instead of TAB as field delimiter
--format=FORMAT print numeric values with printf style floating-point FORMAT.
--output-delimiter=X use X instead as output field delimiter (default: use same delimiter as -t/-W)
--narm skip NA/NaN values
-R, --round=N round numeric output to N decimal places
-W, --whitespace use whitespace (one or more spaces and/or tabs) for field delimiters
-z, --zero-terminated end lines with 0 byte, not newline
Primary Operations Primary operations affect the way the file is processed. If used, the primary operation must be listed first. Some operations require field numbers (groupby, crosstab) while others do not (reverse,check,transpose).  If primary operation is not listed the entire file is processed - either line-by-line (for 'per-line' operations) or all lines as one group (for grouping operations). See Examples section below.
groupby X,Y,... op fld ... group the file by given fields. Equivalent to option '-g'. For each group perform operation op on field fld.
crosstab X,Y [op fld ...] cross-tabulate a file by two fields (cross-tabulation is also known as pivot tables). If no operation is specified, counts how many incidents exist of X,Y.
transpose   transpose rows, columns of the input file
reverse     reverse field order in each line
check [N lines] [N fields] verify the input file has same number of fields in all lines, or the expected number of lines/fields.  number of lines and fields are printed to STDOUT. Exits with non-zero code and prints the offending line if there's a mismatch in the number of lines/ fields.
Line-Filtering operations rmdup       remove lines with duplicated key value
Per-Line operations base64      Encode the field as base64
debase64    Decode the field as base64, exit with error if invalid base64 string
md5/sha1/sha224/sha256/sha384/sha512 Calculate md5/sha1/sha224/sha256/sha384/sha512 hash of the field value
bin[:BUCKET-SIZE] bin numeric values into buckets of size BUCKET-SIZE (defaults to 100).
strbin[:BUCKET-SIZE] hashes the input and returns a numeric integer value between zero and BUCKET-SIZE (defaults to 10).
round/floor/ceil/trunc/frac numeric rounding operations. round (round half away from zero), floor (round up), ceil (ceiling, round down), trunc (truncate, round towards zero), frac (fraction, return fraction part of a decimal-point value).
dirname/basename extract the directory name and the base file name from a given string (same as to dirname(1) and basename(1)).
extname     extract the extension of the file name (without the '.').
barename    extract the base file name without the extension.
getnum[:TYPE] extract a number from the field. TYPE is optional single letter option n/i/d/p/h/o (see examples below).
cut         copy input field to output field (similar to cut(1)).
Numeric Grouping operations sum         sum the of values
min         minimum value
max         maximum value
absmin      minimum of the absolute values
absmax      maximum of the absolute values
range       the values range (max-min)
Textual/Numeric Grouping operations count       count number of elements in the group
first       the first value of the group
last        the last value of the group
rand        one random value from the group
unique      comma-separated sorted list of unique values
collapse    comma-separated list of all input values
countunique number of unique/distinct values
Statistical Grouping operations A p/s prefix indicates the varient: population or sample.  Typically, the sample variant is equivalent with GNU R's internal functions (e.g datamash's sstdev operation is equivalent to R's sd() function).
mean        mean of the values
geomean     geometric mean of the values
harmmean    harmonic mean of the values
trimmean[:PERCENT] trimmed mean of the values. PERCENT should be between 0 and 0.5.  (trimmean:0 is equivalent to mean. trimmean:0.5 is equivalent to median).
median      median value
q1          1st quartile value
q3          3rd quartile value
iqr         inter-quartile range
perc[:PERCENTILE] percentile value PERCENTILE (defaults to 95).
mode        mode value (most common value)
antimode    anti-mode value (least common value)
pstdev/sstdev population/sample standard deviation
pvar/svar   population/sample variance
mad         median absolute deviation, scaled by constant 1.4826 for normal distributions
madraw      median absolute deviation, unscaled
pskew/sskew skewness of the group values x reported by 'sskew' and 'pskew' operations: x > 0       -  positively skewed / skewed right 0 > x           -  negatively skewed / skewed left x > 1       -  highly skewed right 1 > x >  0.5    -  moderately skewed right 0.5 > x > -0.5    -  approximately symmetric -0.5 > x > -1      -  moderately skewed left -1 > x           -  highly skewed left
pkurt/skurt excess Kurtosis of the group
jarque/dpo  p-value of the Jarque-Beta (jarque) and D'Agostino-Pearson Omnibus (dpo) tests for normality: null hypothesis is normality; low p-Values indicate non-normal data; high p-Values indicate null-hypothesis cannot be rejected.
pcov/scov [X:Y] covariance of fields X and Y
ppearson/spearson [X:Y] Pearson product-moment correlation coefficient [Pearson's R] of fields X and Y
Basic usage Print the sum and the mean of values from field 1:
$ seq 10 | datamash sum 1 mean 1 55  5.5
Group input based on field 1, and sum values (per group) on field 2:
$ cat example.txt A  10 A  5 B  9 B  11
$ datamash -g 1 sum 2 < example.txt A  15 B  20
$ datamash groupby 1 sum 2 < example.txt A  15 B  20
Unsorted input must be sorted (with '-s'):
$ cat example.txt A  10 C  4 B  9 C  1 A  5 B  11
$ datamash -s -g1 sum 2 < example.txt A  15 B  20 C  5
Which is equivalent to:
$ cat example.txt | sort -k1,1 | datamash -g 1 sum 2
Header lines Use -h (--headers) if the input file has a header line:
# Given a file with student name, field, test score... $ head -n5 scores_h.txt Name           Major            Score Shawn          Engineering      47 Caleb          Business         87 Christian      Business         88 Derek          Arts             60
# Calculate the mean and standard devian for each major $ datamash --sort --headers --group 2 mean 3 pstdev 3 < scores_h.txt
(or use short form)
$ datamash -sH -g2 mean 3 pstdev 3 < scores_h.txt
(or use named fields)
$ datamash -sH -g Major mean Score pstdev Score < scores_h.txt GroupBy(Major)    mean(Score)   pstdev(Score) Arts              68.9          10.1 Business          87.3           4.9 Engineering       66.5          19.1 Health-Medicine   90.6           8.8 Life-Sciences     55.3          19.7 Social-Sciences   60.2          16.6
Field names must be escaped with a backslash if they start with a digit or contain special characters (dash/minus, colons, commas).  Note the interplay between escaping with backslash and shell quoting.  The following equivalent command sum the values of a field named "FOO-BAR":
$ datamash -H sum FOO\\-BAR < input.txt $ datamash -H sum 'FOO\-BAR' < input.txt $ datamash -H sum "FOO\\-BAR" < input.txt
Skipping comment lines Use -C (--skip-comments) to skip lines starting with ´#´ or ´;´ characters (and optional whitespace before them):
$ cat in.txt #foo   3 bar     5 ;baz    7
$ datamash sum 2 < in.txt 15
$ datamash -C sum 2 < in.txt 5
Multiple fields Use comma or dash to specify multiple fields. The following are equivalent: $ seq 9 | paste - - - 1   2   3 4   5   6 7   8   9
$ seq 9 | paste - - - | datamash sum 1 sum 2 sum 3 12  15  18
$ seq 9 | paste - - - | datamash sum 1,2,3 12  15  18
$ seq 9 | paste - - - | datamash sum 1-3 12  15  18
Rounding The following demonstrate the different rounding operations: $ ( echo X ; seq -1.25 0.25 1.25 ) \ | datamash --full -H round 1 ceil 1 floor 1 trunc 1 frac 1
X     round(X)  ceil(X)  floor(X)  trunc(X)   frac(X) -1.25   -1        -1       -2        -1         -0.25 -1.00   -1        -1       -1        -1          0 -0.75   -1         0       -1         0         -0.75 -0.50   -1         0       -1         0         -0.5 -0.25    0         0       -1         0         -0.25 0.00    0         0        0         0          0 0.25    0         1        0         0          0.25 0.50    1         1        0         0          0.5 0.75    1         1        0         0          0.75 1.00    1         1        1         1          0 1.25    1         2        1         1          0.25
Reversing fields $ seq 6 | paste - - | datamash reverse 2    1 4    3 6    5
Transposing a file $ seq 6 | paste - - | datamash transpose 1    3    5 2    4    6
Removing Duplicated lines Remove lines with duplicate key value from field 1 (Unlike first,last operations, rmdup is much faster and does not require sorting the file with -s):
# Given a list of files and sample IDs: $ cat INPUT SampleID  File 2         cc.txt 3         dd.txt 1         ab.txt 2         ee.txt 3         ff.txt
# Remove lines with duplicated Sample-ID (field 1): $ datamash rmdup 1 < INPUT
# or use named field: $ datamash -H rmdup SampleID < INPUT SampleID  File 2         cc.txt 3         dd.txt 1         ab.txt
Checksums Calculate the sha1 hash value of each TXT file, after calculating the sha1 value of each file's content:
$ sha1sum *.txt | datamash -Wf sha1 2
Check file structure Check the structure of the input file: ensure all lines have the same number of fields, or expected number of lines/fields:
$ seq 10 | paste - - | datamash check && echo ok || echo fail 5 lines, 2 fields ok
$ seq 13 | paste - - - | datamash check && echo ok || echo fail line 4 (3 fields): 10  11  12 line 5 (2 fields): 13 datamash: check failed: line 5 has 2 fields (previous line had 3) fail
$ seq 10 | paste - - | datamash check 2 fields 5 lines 5 lines, 2 fields
$ seq 10 | paste - - | datamash check 4 fields line 1 (2 fields): 1     2 datamash: check failed: line 1 has 2 fields (expecting 4)
$ seq 10 | paste - - | datamash check 7 lines datamash: check failed: input had 5 lines (expecting 7)
Cross-Tabulation Cross-tabulation compares the relationship between two fields.  Given the following input file: $ cat input.txt a    x    3 a    y    7 b    x    21 a    x    40
Show cross-tabulation between the first field (a/b) and the second field (x/y) - counting how many times each pair appears (note: sorting is required):
$ datamash -s crosstab 1,2 < input.txt x    y a    2    1 b    1    N/A
An optional grouping operation can be used instead of counting:
$ datamash -s crosstab 1,2 sum 3 < input.txt x    y a    43   7 b    21   N/A
$ datamash -s crosstab 1,2 unique 3 < input.txt x    y a    3,40 7 b    21   N/A
Binning numeric values Bin input values into buckets of size 5:
$  ( echo X ; seq -10 2.5 10 ) \ | datamash -H --full bin:5 1 X  bin(X) -10.0    -15 -7.5    -10 -5.0    -10 -2.5     -5 0.0      0 2.5      0 5.0      5 7.5      5 10.0     10
Binning string values Hash any input value into a numeric integer.  A typical usage would be to split an input file into N chunks, ensuring that all values of a certain key will be stored in the same chunk:
$ cat input.txt PatientA   10 PatientB   11 PatientC   12 PatientA   14 PatientC   15
Each patient ID is hashed into a bin between 0 and 9 and printed in the last field:
$ datamash --full strbin 1 < input.txt PatientA   10    5 PatientB   11    6 PatientC   12    7 PatientA   14    5 PatientC   15    7
Splitting the input into chunks can be done with awk:
$ cat input.txt \ | datamash --full strbin 1 \ | awk '{print > $NF ".txt"}'
Extracting numbers with getnum The 'getnum' operation extracts a numeric value from the field:
$ echo zoom-123.45xyz | datamash getnum 1 123.45
getnum accepts an optional single-letter TYPE option: getnum:n - natural numbers (positive integers, including zero) getnum:i - integers getnum:d - decimal point numbers getnum:p - positive decimal point numbers (this is the default) getnum:h - hex numbers getnum:o - octal numbers
Examples: $ echo zoom-123.45xyz | datamash getnum 1 123.45
$ echo zoom-123.45xyz | datamash getnum:n 1 123
$ echo zoom-123.45xyz | datamash getnum:i 1 -123
$ echo zoom-123.45xyz | datamash getnum:d 1 123.45
$ echo zoom-123.45xyz | datamash getnum:p 1 -123.45
# Hex 0x123 = 291 Decimal $ echo zoom-123.45xyz | datamash getnum:h 1 291
# Octal 0123 = 83 Decimal $ echo zoom-123.45xyz | datamash getnum:o 1 83
See GNU Datamash Website (https://www.gnu.org/software/datamash)
Copyright © 2020 Assaf Gordon License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.
The full documentation for datamash is maintained as a Texinfo manual. If the info and datamash programs are properly installed at your site, the command
info datamash
GIT-FOR-EACH-REPO(1)               Git Manual               GIT-FOR-EACH-REPO(1)
git-for-each-repo - Run a Git command on a list of repositories
git for-each-repo --config=<config> [--] <arguments>
Run a Git command on a list of repositories. The arguments after the known options or -- indicator are used as the arguments for the Git subprocess.
For example, we could run maintenance on each of a list of repositories stored in a maintenance.repo config variable using
git for-each-repo --config=maintenance.repo maintenance run
This will run git -C <repo> maintenance run for each value <repo> in the multi-valued config variable maintenance.repo.
--config=<config> Use the given config variable as a multi-valued list storing absolute path names. Iterate on that list of paths to run the given arguments.
These config values are loaded from system, global, and local Git config, as available. If git for-each-repo is run in a directory that is not a Git repository, then only the system and global config is used.
If any git -C <repo> <arguments> subprocess returns a non-zero exit code, then the git for-each-repo process returns that exit code without running more subprocesses.
Each git -C <repo> <arguments> subprocess inherits the standard file descriptors stdin, stdout, and stderr.
jstack(1)                    General Commands Manual                   jstack(1)
Name jstack - Stack Trace
jstack [ option ] pid jstack [ option ] executable core jstack [ option ] [server-id@]remote-hostname-or-IP
Options are mutually exclusive. Option, if used, should follow immediately after the command name. See OPTIONS.
pid process id for which the stack trace is to be printed. The process must be a Java process. To get a list of Java processes running on a machine, jps(1) may be used.
core core file for which the stack trace is to be printed.
jstack prints Java stack traces of Java threads for a given Java process or core file or a remote debug server. For each Java frame, the full class name, method name, 'bci' (byte code index) and line number, if available, are printed. With the -m option, jstack prints both Java and native frames of all threads along with the 'pc' (program counter). For each native frame, the closest native symbol to 'pc', if available, is printed. C++ mangled names are not demangled. To demangle C++ names, the output of this command may be piped to c++filt. If the given process is running on a 64-bit VM, you may need to specify the -J-d64 option, e.g.:
jstack -J-d64 -m pid
NOTE - This utility is unsupported and may or may not be available in future versions of the JDK. In Windows Systems where dbgeng.dll is not present, 'Debugging Tools For Windows' needs to be installed to have these tools working. Also, PATH environment variable should contain the location of jvm.dll used by the target process or the location from which the Crash Dump file was produced.
-F Force a stack dump when 'jstack [-l] pid' does not respond.
-l Long listing. Prints additional information about locks such as list of owned java.util.concurrent ownable synchronizers @ http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/locks/AbstractOwnableSynchronizer.html.
-m prints mixed mode (both Java and native C/C++ frames) stack trace.
-h prints a help message.
-help prints a help message
o pstack(1)
o c++filt(1)
Mixed mode stack trace, the -m option, does not work with the remote debug server.
HTTPRY(1)                    General Commands Manual                   HTTPRY(1)
httpry - HTTP logging and information retrieval tool
httpry [ -dFpq ] [ -b file ] [ -f format ] [ -i device ] [ -m methods ] [ -n count ] [ -o file ] [ -P file ] [ -r file ] [ -S bytes ] [ -u user ] [ 'expression' ] httpry -s [ -l threshold ] [ -t seconds ] httpry -h
httpry is a tool designed for displaying and logging HTTP traffic. It is not designed to perform analysis itself, but instead to capture, parse and log the traffic for later analysis. It can be run in real-time displaying the live traffic on the wire, or as a daemon process that logs to an output file.
-b file Write all processed HTTP packets to a binary pcap dump file. Useful for further analysis of logged data.
-d     Run the program as a daemon process. All program status output will be sent to syslog. A pid file is created for the process in /var/run/httpry.pid by default. Requires an output file specified with -o.
-f format Provide a comma-delimited string specifying the parsed HTTP data to output.  See the doc/format-string file for further information regarding available options and syntax.
-F     Disable all output buffering. This may be helpful when piping httpry output into another program.
-h     Display a brief description of these options.
-i device Specify an ethernet interface for the program to listen on. If not specified, the program will poll the system for a list of interfaces and select the first one found.
-l threshold Specify a requests per second rate threshold value when running in rate statistics mode (-s). Only hosts with a rps value greater than or equal to this number will be displayed. Defaults to 1.
-m methods Provide a comma-delimited string that specifies the request methods to parse.  The program defaults to parsing all of the standard RFC2616 method strings if this option is not set. See the doc/method-string file for more information.
-n count Parse this number of HTTP packets and then exit. Defaults to 0, which means loop forever.
-o file Specify an output file for writing parsed packet data.
-p     Do not put the NIC in promiscuous mode on startup. Note that the NIC could already be in that mode for another reason.
-P file Specify a path and filename for creating the PID file in daemon mode.
-q     Suppress non-critical output (startup banner, statistics, etc.).
-r file Provide an input capture file to read from instead of performing a live capture. This option does not require root privileges.
-s     Run httpry in an HTTP request per second display mode. This periodically displays the rate per active host and total rate at a specified interval.
-S     Specify a number of bytes to skip in the ethernet header. This allows for custom header offsets to be accounted for.
-t seconds >>>>>>> HEAD~0 Specify the host statistics display interval in seconds when running in rate statistics mode (-s). Defaults to 5 seconds.
-u user Specify an alternate user to take ownership of the process and any output files. You will need root privileges to do this; it will switch to the new user after initialization.
'expression' Specify a bpf-style capture filter, overriding the default. Here are a few basic examples starting with the default filter:
'tcp port 80 or 8080' 'tcp dst port 80' 'tcp dst port 80 and src host 192.168.1.1'
These filters will capture all web traffic both directions on two common ports, capture only requests made to port 80, and capture requests to port 80 by a particular host, respectively. See 'man tcpdump' for further information on the syntax and available primitives.
httpry was written by Jason Bittel <jason.bittel@gmail.com>. See included COPYING file for specific licensing information
dos2unix(1)                        2020-10-12                        dos2unix(1)
dos2unix - DOS/Mac to Unix and vice versa text file format converter
dos2unix [options] [FILE ...] [-n INFILE OUTFILE ...] unix2dos [options] [FILE ...] [-n INFILE OUTFILE ...]
The Dos2unix package includes utilities "dos2unix" and "unix2dos" to convert plain text files in DOS or Mac format to Unix format and vice versa.
In DOS/Windows text files a line break, also known as newline, is a combination of two characters: a Carriage Return (CR) followed by a Line Feed (LF). In Unix text files a line break is a single character: the Line Feed (LF). In Mac text files, prior to Mac OS X, a line break was single Carriage Return (CR) character. Nowadays Mac OS uses Unix style (LF) line breaks.
Besides line breaks Dos2unix can also convert the encoding of files. A few DOS code pages can be converted to Unix Latin-1. And Windows Unicode (UTF-16) files can be converted to Unix Unicode (UTF-8) files.
Binary files are automatically skipped, unless conversion is forced.
Non-regular files, such as directories and FIFOs, are automatically skipped.
Symbolic links and their targets are by default kept untouched.  Symbolic links can optionally be replaced, or the output can be written to the symbolic link target.  Writing to a symbolic link target is not supported on Windows.
Dos2unix was modelled after dos2unix under SunOS/Solaris.  There is one important difference with the original SunOS/Solaris version. This version does by default in-place conversion (old file mode), while the original SunOS/Solaris version only supports paired conversion (new file mode). See also options "-o" and "-n". Another difference is that the SunOS/Solaris version uses by default iso mode conversion while this version uses by default ascii mode conversion.
--  Treat all following options as file names. Use this option if you want to convert files whose names start with a dash. For instance to convert a file named "-foo", you can use this command:
dos2unix -- -foo
Or in new file mode:
dos2unix -n -- -foo out.txt
--allow-chown Allow file ownership change in old file mode.
When this option is used, the conversion will not be aborted when the user and/or group ownership of the original file can't be preserved in old file mode. Conversion will continue and the converted file will get the same new ownership as if it was converted in new file mode. See also options "-o" and "-n". This option is only available if dos2unix has support for preserving the user and group ownership of files.
-ascii Convert only line breaks. This is the default conversion mode.
-iso Conversion between DOS and ISO-8859-1 character set. See also section CONVERSION MODES.
-1252 Use Windows code page 1252 (Western European).
-437 Use DOS code page 437 (US). This is the default code page used for ISO conversion.
-850 Use DOS code page 850 (Western European).
-860 Use DOS code page 860 (Portuguese).
-863 Use DOS code page 863 (French Canadian).
-865 Use DOS code page 865 (Nordic).
-7  Convert 8 bit characters to 7 bit space.
-b, --keep-bom Keep Byte Order Mark (BOM). When the input file has a BOM, write a BOM in the output file. This is the default behavior when converting to DOS line breaks. See also option "-r".
-c, --convmode CONVMODE Set conversion mode. Where CONVMODE is one of: ascii, 7bit, iso, mac with ascii being the default.
-D, --display-enc ENCODING Set encoding of displayed text. Where ENCODING is one of: ansi, unicode, unicodebom, utf8, utf8bom with ansi being the default.
This option is only available in dos2unix for Windows with Unicode file name support. This option has no effect on the actual file names read and written, only on how they are displayed.
There are several methods for displaying text in a Windows console based on the encoding of the text. They all have their own advantages and disadvantages.
ansi Dos2unix's default method is to use ANSI encoded text. The advantage is that it is backwards compatible. It works with raster and TrueType fonts. In some regions you may need to change the active DOS OEM code page to the Windows system ANSI code page using the "chcp" command, because dos2unix uses the Windows system code page.
The disadvantage of ansi is that international file names with characters not inside the system default code page are not displayed properly. You will see a question mark, or a wrong symbol instead. When you don't work with foreign file names this method is OK.
unicode, unicodebom The advantage of unicode (the Windows name for UTF-16) encoding is that text is usually properly displayed. There is no need to change the active code page.  You may need to set the console's font to a TrueType font to have international characters displayed properly. When a character is not included in the TrueType font you usually see a small square, sometimes with a question mark in it.
When you use the ConEmu console all text is displayed properly, because ConEmu automatically selects a good font.
The disadvantage of unicode is that it is not compatible with ASCII. The output is not easy to handle when you redirect it to another program.
When method "unicodebom" is used the Unicode text will be preceded with a BOM (Byte Order Mark). A BOM is required for correct redirection or piping in PowerShell.
utf8, utf8bom The advantage of utf8 is that it is compatible with ASCII. You need to set the console's font to a TrueType font. With a TrueType font the text is displayed similar as with the "unicode" encoding.
The disadvantage is that when you use the default raster font all non-ASCII characters are displayed wrong. Not only unicode file names, but also translated messages become unreadable. On Windows configured for an East-Asian region you may see a lot of flickering of the console when the messages are displayed.
In a ConEmu console the utf8 encoding method works well.
When method "utf8bom" is used the UTF-8 text will be preceded with a BOM (Byte Order Mark). A BOM is required for correct redirection or piping in PowerShell.
The default encoding can be changed with environment variable DOS2UNIX_DISPLAY_ENC by setting it to "unicode", "unicodebom", "utf8", or "utf8bom".
-f, --force Force conversion of binary files.
-gb, --gb18030 On Windows UTF-16 files are by default converted to UTF-8, regardless of the locale setting. Use this option to convert UTF-16 files to GB18030. This option is only available on Windows. See also section GB18030.
-i[FLAGS], --info[=FLAGS] FILE ... Display file information. No conversion is done.
The following information is printed, in this order: number of DOS line breaks, number of Unix line breaks, number of Mac line breaks, byte order mark, text or binary, file name.
6       0       0  no_bom    text    dos.txt 0       6       0  no_bom    text    unix.txt 0       0       6  no_bom    text    mac.txt 6       6       6  no_bom    text    mixed.txt 50       0       0  UTF-16LE  text    utf16le.txt 0      50       0  no_bom    text    utf8unix.txt 50       0       0  UTF-8     text    utf8dos.txt 2     418     219  no_bom    binary  dos2unix.exe
Note that sometimes a binary file can be mistaken for a text file. See also option "-s".
Optionally extra flags can be set to change the output. One or more flags can be added.
0   Print the file information lines followed by a null character instead of a newline character. This enables correct interpretation of file names with spaces or quotes when flag c is used. Use this flag in combination with xargs(1) option "-0" or "--null".
d   Print number of DOS line breaks.
u   Print number of Unix line breaks.
m   Print number of Mac line breaks.
b   Print the byte order mark.
t   Print if file is text or binary.
c   Print only the files that would be converted.
With the "c" flag dos2unix will print only the files that contain DOS line breaks, unix2dos will print only file names that have Unix line breaks.
h   Print a header.
p   Show file names without path.
Show information for all *.txt files:
dos2unix -i *.txt
Show only the number of DOS line breaks and Unix line breaks:
dos2unix -idu *.txt
Show only the byte order mark:
dos2unix --info=b *.txt
List the files that have DOS line breaks:
dos2unix -ic *.txt
List the files that have Unix line breaks:
unix2dos -ic *.txt
Convert only files that have DOS line breaks and leave the other files untouched:
dos2unix -ic0 *.txt | xargs -0 dos2unix
Find text files that have DOS line breaks:
find -name '*.txt' -print0 | xargs -0 dos2unix -ic
-k, --keepdate Keep the date stamp of output file same as input file.
-L, --license Display program's license.
-l, --newline Add additional newline.
dos2unix: Only DOS line breaks are changed to two Unix line breaks. In Mac mode only Mac line breaks are changed to two Unix line breaks.
unix2dos: Only Unix line breaks are changed to two DOS line breaks. In Mac mode Unix line breaks are changed to two Mac line breaks.
-m, --add-bom Write a Byte Order Mark (BOM) in the output file. By default an UTF-8 BOM is written.
When the input file is UTF-16, and the option "-u" is used, an UTF-16 BOM will be written.
Never use this option when the output encoding is other than UTF-8, UTF-16, or GB18030. See also section UNICODE.
-n, --newfile INFILE OUTFILE ... New file mode. Convert file INFILE and write output to file OUTFILE. File names must be given in pairs and wildcard names should not be used or you will lose your files.
The person who starts the conversion in new file (paired) mode will be the owner of the converted file. The read/write permissions of the new file will be the permissions of the original file minus the umask(1) of the person who runs the conversion.
--no-allow-chown Don't allow file ownership change in old file mode (default).
Abort conversion when the user and/or group ownership of the original file can't be preserved in old file mode. See also options "-o" and "-n". This option is only available if dos2unix has support for preserving the user and group ownership of files.