neulab/tldr huggingface dataset:
<http://library.gnome.org/devel/gtk/stable/gtk-running.html> <http://library.gnome.org/devel/gtk/stable/gtk-x11.html>
Chromium obeys the following environment variables:
all_proxy Shorthand for specifying all of http_proxy, https_proxy, ftp_proxy
http_proxy, https_proxy, ftp_proxy The proxy servers used for HTTP, HTTPS, and FTP.  Note: because Gnome/KDE proxy settings may propagate into these variables in some terminals, this variable is ignored (in preference for actual system proxy settings) when running under Gnome or KDE.  Use the command-line flags to set these when you want to force their values.
auto_proxy Specify proxy autoconfiguration.  Defined and empty autodetects; otherwise, it should be an autoconfig URL.  But see above note about Gnome/KDE.
SOCKS_SERVER SOCKS proxy server (defaults to SOCKS v4, also set SOCKS_VERSION=5 to use SOCKS v5).
no_proxy Comma separated list of hosts or patterns to bypass proxying.
$HOME/.config/chromium Default directory for configuration data.
$HOME/.cache/chromium Default directory for cache data.  (Why?  See <http://standards.freedesktop.org/basedir-spec/latest/> .)
Bug tracker:
Be sure to do your search within "All Issues" before reporting bugs, and be sure to pick the "Defect on Linux" template when filing a new one.
The Chromium team - <http://www.chromium.org>
NTH()                                                                      NTH()
NTH
AnyType NTH( ListType ListObject, NumericType Index )
Returns the Index (base count 1) element of the list ListObject.
Lst = list( a, list( b, c ), d ); Lst2 = NTH( Lst, 2 );
and now Lst2 is equal to 'list( b, c )'.  See also NREF.
MICROCOM(1)                  General Commands Manual                 MICROCOM(1)
microcom - A minimalistic terminal program
microcom [-d] [-f] [-p devfile|-t host:port] [-s speed] microcom -c interface:rx_id:tx_id microcom -h
This manual page documents briefly the microcom command.
microcom is a is a minimalistic terminal program for accessing devices (e.g. switches) via a serial connection.  It features connection via RS232 serial interfaces (including setting of transferrates) as well as in `telnetmode' as specified in rfc2217 and a (Linux specific) can mode.
The escape character to enter the program menu is crtl-\. There are several commands available, among them quit (to exit microcom), exit (to return to normal mode) and speed (to set terminal speed)
microcom follows the usual GNU command line syntax, with long options starting with two dashes (`-'). A summary of options is included below.
-d     output debugging info to stdout.
-f     ignore an already existing lock file.
-p devfile, --port=devfile use the specified serial port device (default /dev/ttyS0).
-s speed, --speed=speed use specified baudrate (default 115200).
-t host:port, --telnet=host:port work in telnet (rfc2217) mode.
-c interface:rx_id:tx_id, --can=interface:rx_id:tx_id work in CAN mode (default: can0:200:200)
This manual page was written by Uwe Kleine-KÃ¶nig based on work initially done by Alexander Reichle-Schmehl.
GIT-GC(1)                          Git Manual                          GIT-GC(1)
git-gc - Cleanup unnecessary files and optimize the local repository
git gc [--aggressive] [--auto] [--quiet] [--prune=<date> | --no-prune] [--force] [--keep-largest-pack]
Runs a number of housekeeping tasks within the current repository, such as compressing file revisions (to reduce disk space and increase performance), removing unreachable objects which may have been created from prior invocations of git add, packing refs, pruning reflog, rerere metadata or stale working trees. May also update ancillary indexes such as the commit-graph.
When common porcelain operations that create objects are run, they will check whether the repository has grown substantially since the last maintenance, and if so run git gc automatically. See gc.auto below for how to disable this behavior.
Running git gc manually should only be needed when adding objects to a repository without regularly running such porcelain commands, to do a one-off repository optimization, or e.g. to clean up a suboptimal mass-import. See the "PACKFILE OPTIMIZATION" section in git-fast- import(1) for more details on the import case.
--aggressive Usually git gc runs very quickly while providing good disk space utilization and performance. This option will cause git gc to more aggressively optimize the repository at the expense of taking much more time. The effects of this optimization are mostly persistent. See the "AGGRESSIVE" section below for details.
--auto With this option, git gc checks whether any housekeeping is required; if not, it exits without performing any work.
See the gc.auto option in the "CONFIGURATION" section below for how this heuristic works.
Once housekeeping is triggered by exceeding the limits of configuration options such as gc.auto and gc.autoPackLimit, all other housekeeping tasks (e.g. rerere, working trees, reflog...) will be performed as well.
--prune=<date> Prune loose objects older than date (default is 2 weeks ago, overridable by the config variable gc.pruneExpire). --prune=now prunes loose objects regardless of their age and increases the risk of corruption if another process is writing to the repository concurrently; see "NOTES" below. --prune is on by default.
--no-prune Do not prune any loose objects.
--quiet Suppress all progress reports.
--force Force git gc to run even if there may be another git gc instance running on this repository.
--keep-largest-pack All packs except the largest pack and those marked with a .keep files are consolidated into a single pack. When this option is used, gc.bigPackThreshold is ignored.
When the --aggressive option is supplied, git-repack(1) will be invoked with the -f flag, which in turn will pass --no-reuse-delta to git-pack- objects(1). This will throw away any existing deltas and re-compute them, at the expense of spending much more time on the repacking.
The effects of this are mostly persistent, e.g. when packs and loose objects are coalesced into one another pack the existing deltas in that pack might get re-used, but there are also various cases where we might pick a sub-optimal delta from a newer pack instead.
Furthermore, supplying --aggressive will tweak the --depth and --window options passed to git-repack(1). See the gc.aggressiveDepth and gc.aggressiveWindow settings below. By using a larger window size we’re more likely to find more optimal deltas.
It’s probably not worth it to use this option on a given repository without running tailored performance benchmarks on it. It takes a lot more time, and the resulting space/delta optimization may or may not be worth it. Not using this at all is the right trade-off for most users and their repositories.
The below documentation is the same as what’s found in git-config(1):
git gc tries very hard not to delete objects that are referenced anywhere in your repository. In particular, it will keep not only objects referenced by your current set of branches and tags, but also objects referenced by the index, remote-tracking branches, reflogs (which may reference commits in branches that were later amended or rewound), and anything else in the refs/* namespace. Note that a note (of the kind created by git notes) attached to an object does not contribute in keeping the object alive. If you are expecting some objects to be deleted and they aren’t, check all of those locations and decide whether it makes sense in your case to remove those references.
On the other hand, when git gc runs concurrently with another process, there is a risk of it deleting an object that the other process is using but hasn’t created a reference to. This may just cause the other process to fail or may corrupt the repository if the other process later adds a reference to the deleted object. Git has two features that significantly mitigate this problem:
1. Any object with modification time newer than the --prune date is kept, along with everything reachable from it.
2. Most operations that add an object to the database update the modification time of the object if it is already present so that #1 applies.
However, these features fall short of a complete solution, so users who run commands concurrently have to live with some risk of corruption (which seems to be low in practice).
The git gc --auto command will run the pre-auto-gc hook. See githooks(5) for more information.
git-prune(1) git-reflog(1) git-repack(1) git-rerere(1)
BERKS(1)                          User Commands                         BERKS(1)
berks - Chef addon to manage cookbooks and their dependencies
Commands: berks apply ENVIRONMENT # Apply version locks from Berksfile.lock to a Chef environment
berks contingent COOKBOOK # List all cookbooks that depend on the given cookbook in your Berksfile
berks cookbook NAME [PATH] # Create a skeleton for a new cookbook
berks help [COMMAND] # Describe available commands or one specific command
berks info [COOKBOOK] # Display name, author, copyright, and dependency information about a cookbook
berks init [PATH] # Initialize Berkshelf in the given directory
berks install # Install the cookbooks specified in the Berksfile
berks list # List cookbooks and their dependencies specified by your Berksfile
berks outdated [COOKBOOKS] # List dependencies that have new versions available that satisfy their constraints
berks package [PATH] # Vendor and archive the dependencies of a Berksfile
berks search NAME # Search the remote source for cookbooks matching the partial name
berks shelf SUBCOMMAND # Interact with the cookbook store
berks show [COOKBOOK] # Display the path to a cookbook on disk
berks update [COOKBOOKS] # Update the cookbooks (and dependencies) specified in the Berksfile
berks upload [COOKBOOKS] # Upload the cookbook specified in the Berksfile to the Chef Server
berks vendor [PATH] # Vendor the cookbooks specified by the Berksfile into a directory
berks verify # Perform a quick validation on the contents of your resolved cookbooks
berks version # Display version
berks viz # Visualize the dependency graph
-c, [--config=PATH] # Path to Berkshelf configuration to use.
-F, [--format=FORMAT] # Output format to use.  # Default: human
-q, [--quiet], [--no-quiet] # Silence all informational output.
-d, [--debug], [--no-debug] # Output debug information
IDATE(1)                          User Commands                         IDATE(1)
ical - A Hijri/Islamic calendar (and converter)
ical [--gregorian yyyymmdd] [--hijri yyyymmdd] [--umm_alqura] [--fixed_view] [--dual] [--help]
The ical program is a Hijri/Islamic calendar displayer.  It utilizes and includes a Gregorian to Hijri (and vice-versa) date converter.  The application uses and offers multiple calculation methods with not all of them agreeing at all times.  The reason for this multiplicity is due to not having one agreed upon method and so various entities develop and advocate their calculations.
ical is able to comprehend and calculate both pre-epoch or pre-Hijrah, denoted as "B.H", as well as post-epoch or post-Hijrah, denoted as "A.H", dates.  ical also utilizes Gregorian's pre-epoch "B.C" and post-epoch "A.D" dates and notes them per its output.  When entering pre-epoch years, negative numbers ought to be utilized.
ical when run without any command-line options uses the host machine's Gregorian date and converts it to Hijri to display that month's view. Entries enclosed by [] denote exact day specified.
ical follows the usual GNU command line syntax, with long options starting with two dashes (`-').  A summary of all options is noted below:
-g, --gregorian yyyymmdd Specify the Gregorian date to be converted where 'y' stands for year, 'm' for month and 'd' for day
-hi, --hijri yyyymmdd Specify the Hijri date to be converted where 'y' stands for year, 'm' for month and 'd' for day
-u, --umm_alqura Specify to use the Umm Al-Qura calculation method (used mostly in Saudi Arabia)
-f, --fixed_view Show a fixed week view (ie. start on Sun and end on Sat) else default to showing the preferred week view of the resulting calendar (Gregorian starts on Sunday, Hijri starts on Saturday)
-d, --dual Show both conversion from and to calendar months simultaneously
The Hijri calendar is used in most of the Arab world and is the symbolic calendar of the Islamic faithed worldwide.  This calendar is known as the "Hijri" (based on the word "Hijrah" - denoting migration in Arabic) to signal Prophet Mohammed's (PBUH) migration from Makkah to Medinah on Thursday, July 15, 622 AD (Julian).
The Islamic Hijri calendar is strictly lunar (ie. moon-based) with twelve lunar months which do not correspond or track their solar counterparts (the Gregorian calendar is a solar or sun-based calendar).  Lunar years and thus Hijri years are, on average, about 354 days long resulting in a Hijri year being roughly about 11 days shorter than its Gregorian counterpart.
There is much discussion and confusion regarding how best to track the Hijri calendar.  A great deal of that confusion is based on the fact that many relay on a human moon sighting to denote the start (or end) of a month (each month of the Hijri calendar starts when a new moon's crescent is observed or is made visible at sunset) as opposed to using an empirical mathematic certainty.  The methods presented in this application and its underlying ITL library are strictly arithmetic in nature and do NOT take moon-phases into consideration (in short, observational approximation is not used).
There is currently no Umm Al-Qura support.
Report bugs on the web using http://bugs.arabeyes.org
Written by Nadim Shaikli as part of the Arabeyes.org project.
ical is subject to the GNU General Public License (GPL). Copyright Â© 2005, Arabeyes, Nadim Shaikli.
The ITL (Islamic Tools Library).  It is the underlying requirement for ical to function.  The ITL library was created and is hosted at www.arabeyes.org.
This option is internally converted to git version ...  and accepts the same options as the git-version(1) command. If --help is also given, it takes precedence over --version.
GIT_REF_PARANOIA If set to 0, ignore broken or badly named refs when iterating over lists of refs. Normally Git will try to include any such refs, which may cause some operations to fail. This is usually preferable, as potentially destructive operations (e.g., git-prune(1)) are better off aborting rather than ignoring broken refs (and thus considering the history they point to as not worth saving). The default value is 1 (i.e., be paranoid about detecting and aborting all operations). You should not normally need to set this to 0, but it may be useful when trying to salvage data from a corrupted repository.
Note that servers may need to be configured to allow this variable to pass over some transports. It will be propagated automatically when accessing local repositories (i.e., file:// or a filesystem path), as well as over the git:// protocol. For git-over-http, it should work automatically in most configurations, but see the discussion in git- http-backend(1). For git-over-ssh, the ssh server may need to be configured to allow clients to pass this variable (e.g., by using AcceptEnv GIT_PROTOCOL with OpenSSH).
This configuration is optional. If the variable is not propagated, then clients will fall back to the original "v0" protocol (but may miss out on some performance improvements or features). This variable currently only affects clones and fetches; it is not yet used for pushes (but may be in the future).
GNOME-TERMINAL(1)                 User Commands                GNOME-TERMINAL(1)
gnome-terminal - A terminal emulator for GNOME
gnome-terminal [OPTION...] [-- PROGRAM [ARG...]]
gnome-terminal is a terminal emulator application for accessing a UNIX shell environment which can be used to run programs available on your system. It supports several profiles, multiple tabs and implements several keyboard shortcuts.
--help, -h Show a brief overview of all the options.
--help-all Show all the options in detail.
--help-gtk Show all the GTK options.
--help-terminal Show all the options to select between new terminal tabs or windows.
--help-terminal-options Show all the options to change the attributes of terminals regardless of whether they are in separate tabs or windows.
--help-window-options Show all the options to change the attributes of windows containing terminals.
--load-config=FILE Restore the application to a previously saved state by loading it from a configuration file.
--preferences Show the preferences window.
--print-environment, -p Print the environment variables to interact with newly created terminals.
--quiet, -q Suppress diagnostics.
--verbose, -v Increase diagnostic verbosity.
--tab Open a new tab containing a terminal in the last-opened window with the default profile.
--window Open a new window with a tab containing a terminal with the default profile.
--command, -e=COMMAND Split the argument to this option into a program and arguments in the same way a shell would, and execute the resulting command-line inside the terminal.
This option is deprecated. Instead, use -- to terminate the options, and put the program and arguments to execute after it: for example, instead of gnome-terminal -e "python3 -q", prefer to use gnome-terminal -- python3 -q.
Note that the COMMAND is not run via a shell: it is split into words and executed as a program. If shell syntax is required, use the form gnome-terminal -- sh -c '...'.
--execute PROGRAM [ARGS], -x PROGRAM [ARGS] Stop parsing options at this point, and interpret all subsequent options as a program and arguments to execute inside the terminal.
This option is deprecated: use -- instead. For example, instead of gnome-terminal -x python3 -q, prefer to use gnome-terminal -- python3 -q.
--fd=FD Forward file descriptor.
--profile=PROFILE-NAME Use the given profile instead of the default profile.
--title, -t=TITLE Set the initial terminal title.
--wait Wait until the terminal's child exits.
--working-directory=DIRNAME Set the terminal's working directory.
--zoom=ZOOM Set the terminal's zoom factor. 1.0 is normal size.
--active Set the last specified tab as the active one in its window.
--full-screen Full-screen the window.
--geometry=GEOMETRY Set the window size as COLSxROWS+X+Y. For example, 80x24 or 80x24+200+200.
--hide-menubar Turn off the menubar for the window.
--show-menubar Turn on the menubar for the window.
--maximize Maximize the window.
--role=ROLE Set the X window role.
--class=CLASS Program class as used by the window manager.
--display=DISPLAY X display to use.
--g-fatal-warnings Make all warnings fatal.
--gdk-debug=FLAGS GDK debugging flags to set.
--gdk-no-debug=FLAGS GDK debugging flags to unset.
--gtk-debug=FLAGS GTK debugging flags to set.
--gtk-no-debug=FLAGS GTK debugging flags to unset.
--gtk-module=MODULES Load additional GTK modules.
--name=NAME Program name as used by the window manager.
Please read https://wiki.gnome.org/Apps/Terminal/ReportingBugs on how to report bugs.
To run a terminal containing an interactive Python prompt:
gnome-terminal --title=Python -- python3 -q
To interpret shell syntax in a terminal, either write it in a separate shell script, or use sh -c:
gnome-terminal -- sh -c 'if [ "$(id -u)" = 0 ]; then ...'
For further information, visit the website https://wiki.gnome.org/Apps/Terminal. There's a list of frequently asked questions at https://wiki.gnome.org/Apps/Terminal/FAQ.
duperemove(8)                System Manager's Manual               duperemove(8)
duperemove - Find duplicate extents and print them to stdout
duperemove [options] files...
duperemove is a simple tool for finding duplicated extents and submitting them for deduplication. When given a list of files it will hash the contents of their extents and compare those hashes to each other, finding and categorizing extents that match each other. When given the -d option, duperemove will submit those extents for deduplication using the Linux kernel extent-same ioctl.
duperemove can store the hashes it computes in a hashfile. If given an existing hashfile, duperemove will only compute hashes for those files which have changed since the last run.  Thus you can run duperemove repeatedly on your data as it changes, without having to re-checksum unchanged data.  For more on hashfiles see the --hashfile option below as well as the Examples section.
duperemove can also take input from the fdupes program, see the --fdupes option below.
Duperemove has two major modes of operation one of which is a subset of the other.
Readonly / Non-deduplicating Mode When run without -d (the default) duperemove will print out one or more tables of matching extents it has determined would be ideal candidates for deduplication. As a result, readonly mode is useful for seeing what duperemove might do when run with -d.
Generally, duperemove does not concern itself with the underlying representation of the extents it processes. Some of them could be compressed, undergoing I/O, or even have already been deduplicated. In dedupe mode, the kernel handles those details and therefore we try not to replicate that work.
Deduping Mode This functions similarly to readonly mode with the exception that the duplicated extents found in our "read, hash, and compare" step will actually be submitted for deduplication. Extents that have already been deduped will be skipped. An estimate of the total data deduplicated will be printed after the operation is complete. This estimate is calculated by comparing the total amount of shared bytes in each file before and after the dedupe.
files can refer to a list of regular files and directories or be a hyphen (-) to read them from standard input.  If a directory is specified, all regular files within it will also be scanned. Duperemove can also be told to recursively scan directories with the '-r' switch.
-r     Enable recursive dir traversal.
-d     De-dupe the results - only works on btrfs and xfs.
-A     Opens files readonly when deduping. Primarily for use by privileged users on readonly snapshots.
-h     Print numbers in human-readable format.
-q     Quiet mode. Duperemove will only print errors and a short summary of any dedupe.
--hashfile=hashfile Use a file for storage of hashes instead of memory. This option drastically reduces the memory footprint of duperemove and is recommended when your data set is more than a few files large. Hashfiles are also reusable, allowing you to further reduce the amount of hashing done on subsequent dedupe runs.
If hashfile does not exist it will be created.  If it exists, duperemove will check the file paths stored inside of it for changes.  Files which have changed will be rescanned and their updated hashes will be written to the hashfile.  Deleted files will be removed from the hashfile.
New files are only added to the hashfile if they are discoverable via the files argument.  For that reason you probably want to provide the same files list and -r arguments on each run of duperemove.  The file discovery algorithm is efficient and will only visit each file once, even if it is already in the hashfile.
Adding a new path to a hashfile is as simple as adding it to the files argument.
When deduping from a hashfile, duperemove will avoid deduping files which have not changed since the last dedupe.
-L     Print all files in the hashfile and exit. Requires the --hashfile option.  Will print additional information about each file when run with -v.
-R [file] Remove file from the db and exit. Can be specified multiple times. Duperemove will read the list from standard input if a hyphen (-) is provided. Requires the --hashfile option.
Note: If you are piping filenames from another duperemove instance it is advisable to do so into a temporary file first as running duperemove simultaneously on the same hashfile may corrupt that hashfile.
--fdupes Run in fdupes mode. With this option you can pipe the output of fdupes to duperemove to dedupe any duplicate files found. When receiving a file list in this manner, duperemove will skip the hashing phase.
-v     Be verbose.
--skip-zeroes Read data blocks and skip any zeroed blocks, useful for speedup duperemove, but can prevent deduplication of zeroed files.
-b size Use the specified block size for reading file extents. Defaults to 128K.
--io-threads=N Use N threads for I/O. This is used by the file hashing and dedupe stages. Default is automatically detected based on number of host cpus.
--cpu-threads=N Use N threads for CPU bound tasks. This is used by the duplicate extent finding stage. Default is automatically detected based on number of host cpus.
Note: Hyperthreading can adversely affect performance of the extent finding stage. If duperemove detects an Intel CPU with hyperthreading it will use half the number of cores reported by the system for cpu bound tasks.
--dedupe-options=options Comma separated list of options which alter how we dedupe. Prepend 'no' to an option in order to turn it off.
[no]partial Duperemove can often find more dedupe by comparing portions of extents to each other. This can be a lengthy, CPU intensive task so it is turned off by default.
The code behind this option is under active development and as a result the semantics of the partial argument may change.
[no]same Defaults to off. Allow dedupe of extents within the same file.
[no]fiemap Defaults to on. Duperemove uses the fiemap ioctl during the dedupe stage to optimize out already deduped extents as well as to provide an estimate of the space saved after dedupe operations are complete.
Unfortunately, some versions of Btrfs exhibit extremely poor performance in fiemap as the number of references on a file extent goes up. If you are experiencing the dedupe phase slowing down or 'locking up' this option may give you a significant amount of performance back.
Note: This does not turn off all usage of fiemap, to disable fiemap during the file scan stage, you will also want to use the --lookup-extents=no option.
[no]block Deprecated.
--help Prints help text.
--lookup-extents=[yes|no] Defaults to yes. Allows duperemove to skip checksumming some blocks by checking their extent state.
--read-hashes=hashfile This option is primarily for testing. See the --hashfile option if you want to use hashfiles.
Read hashes from a hashfile. A file list is not required with this option. Dedupe can be done if duperemove is run from the same base directory as is stored in the hash file (basically duperemove has to be able to find the files).
--write-hashes=hashfile This option is primarily for testing. See the --hashfile option if you want to use hashfiles.
Write hashes to a hashfile. These can be read in at a later date and deduped from.
--debug Print debug messages, forces -v if selected.
--hash-threads=N Deprecated, see --io-threads above.
--hash=alg You can choose between murmur3 and xxhash. The default is murmur3 as it is very fast and can generate 128 bit digests for a very small chance of collision. Xxhash may be faster but generates only 64 bit digests. Both hashes are fast enough that the default should work well for the overwhelming majority of users.
--exclude=PATTERN You an exclude certain files and folders from the deduplication process. This might be benefical for skipping subvolume snapshot mounts, for instance. You need to provide full path for exclusion. For example providing just a file name with a wildcard i.e duperemove --exclude file-* won't ever match because internally duperemove works with absolute paths. Another thing to keep in mind is that shells usually expand glob pattern so the passed in pattern ought to also be quoted. Taking everything into consideration the correct way to pass an exclusion pattern is duperemove --exclude "/path/to/dir/file*" /path/to/dir
Simple Usage Dedupe the files in directory /foo, recurse into all subdirectories. You only want to use this for small data sets.
duperemove -dr /foo
Use duperemove with fdupes to dedupe identical files below directory foo.
fdupes -r /foo | duperemove --fdupes
Using Hashfiles Duperemove can optionally store the hashes it calculates in a hashfile. Hashfiles have two primary advantages - memory usage and re-usability. When using a hashfile, duperemove will stream computed hashes to it, instead of main memory.
If Duperemove is run with an existing hashfile, it will only scan those files which have changed since the last time the hashfile was updated. The files argument controls which directories duperemove will scan for newly added files. In the simplest usage, you rerun duperemove with the same parameters and it will only scan changed or newly added files - see the first example below.
Dedupe the files in directory foo, storing hashes in foo.hash. We can run this command multiple times and duperemove will only checksum and dedupe changed or newly added files.
duperemove -dr --hashfile=foo.hash foo/
Don't scan for new files, only update changed or deleted files, then dedupe.
duperemove -dr --hashfile=foo.hash
Add directory bar to our hashfile and discover any files that were recently added to foo.
duperemove -dr --hashfile=foo.hash foo/ bar/
List the files tracked by foo.hash.
duperemove -L --hashfile=foo.hash
Is there an upper limit to the amount of data duperemove can process? Duperemove v0.11 is fast at reading and cataloging data. Dedupe runs will be memory limited unless the '--hashfile' option is used. '--hashfile' allows duperemove to temporarily store duplicated hashes to disk, thus removing the large memory overhead and allowing for a far larger amount of data to be scanned and deduped. Realistically though you will be limited by the speed of your disks and cpu. In those situations where resources are limited you may have success by breaking up the input data set into smaller pieces.
When using a hashfile, duperemove will only store duplicate hashes in memory. During normal operation then the hash tree will make up the largest portion of duperemove memory usage. As of Duperemove v0.11 hash entries are 88 bytes in size. If you know the number of duplicate blocks in your data set you can get a rough approximation of memory usage by multiplying with the hash entry size.
Actual performance numbers are dependent on hardware - up to date testing information is kept on the duperemove wiki (see below for the link).
How large of a hashfile will duperemove create? Hashfiles are essentially sqlite3 database files with several tables, the largest of which are the files and extents tables. Each extents table entry is about 72 bytes though that may grow as features are added. The size of a files table entry depends on the file path but a good estimate is around 270 bytes per file. The number of extents in a data set is directly proportional to file fragmentation level.
If you know the total number of extents and files in your data set then you can calculate the hashfile size as:
Hashfile Size = Num Hashes X 72 + Num Files X 270
Using a real world example of 1TB (8388608 128K blocks) of data over 1000 files:
8388608 * 72 + 270 * 1000 = 755244720 or about 720MB for 1TB spread over 1000 files.
Note that none of this takes database overhead into account.
Is is safe to interrupt the program (Ctrl-C)? Yes, Duperemove uses a transactional database engine and organizes db changes to take advantage of those features. The result is that you should be able to ctrl-c the program at any point and re-run without experiencing corruption of your hashfile.
I got two identical files, why are they not deduped? Duperemove by default works on extent granularity. What this means is if there are two files which are logically identical (have the same content) but are laid out on disk with different extent structure they won't be deduped. For example if 2 files are 128k each and their content are identical but one of them consists of a single 128k extent and the other of 2 x 64k extents then they won't be deduped. This behavior is dependent on the current implementation and is subject to change as duperemove is being improved.
How can I find out my space savings after a dedupe? Duperemove will print out an estimate of the saved space after a dedupe operation for you.
You can get a more accurate picture by running 'btrfs fi df' before and after each duperemove run.
Be careful about using the 'df' tool on btrfs - it is common for space reporting to be 'behind' while delayed updates get processed, so an immediate df after deduping might not show any savings.
Why is the total deduped data report an estimate? At the moment duperemove can detect that some underlying extents are shared with other files, but it can not resolve which files those extents are shared with.
Imagine duperemove is examing a series of files and it notes a shared data region in one of them. That data could be shared with a file outside of the series. Since duperemove can't resolve that information it will account the shared data against our dedupe operation while in reality, the kernel might deduplicate it further for us.
Why are my files showing dedupe but my disk space is not shrinking? This is a little complicated, but it comes down to a feature in Btrfs called _bookending_. The Btrfs wiki explains this in detail: http://en.wikipedia.org/wiki/Btrfs#Extents.
Essentially though, the underlying representation of an extent in Btrfs can not be split (with small exception). So sometimes we can end up in a situation where a file extent gets partially deduped (and the extents marked as shared) but the underlying extent item is not freed or truncated.
Is duperemove safe for my data? Yes. To be specific, duperemove does not deduplicate the data itself.  It simply finds candidates for dedupe and submits them to the Linux kernel extent-same ioctl. In order to ensure data integrity, the kernel locks out other access to the file and does a byte-by-byte compare before proceeding with the dedupe.
What is the cost of deduplication? Deduplication will lead to increased fragmentation. The blocksize chosen can have an effect on this. Larger blocksizes will fragment less but may not save you as much space. Conversely, smaller block sizes may save more space at the cost of increased fragmentation.
Deduplication is currently only supported by the btrfs and xfs filesystem.
The Duperemove project page can be found at https://github.com/markfasheh/duperemove
There is also a wiki at https://github.com/markfasheh/duperemove/wiki
hashstats(8) filesystems(5) btrfs(8) xfs(8) fdupes(1)
STREAMLINK(1)                      Streamlink                      STREAMLINK(1)
streamlink - extracts streams from various services and pipes them into a video player of choice
streamlink [OPTIONS] <URL> [STREAM]
streamlink --loglevel debug youtu.be/VIDEO-ID best streamlink --player mpv --player-args '--no-border --no-keepaspect-window' twitch.tv/CHANNEL 1080p60 streamlink --player-external-http --player-external-http-port 8888 URL STREAM streamlink --output /path/to/file --http-timeout 60 URL STREAM streamlink --stdout URL STREAM | ffmpeg -i pipe:0 ... streamlink --http-header 'Authorization=OAuth TOKEN' --http-header 'Referer=URL' URL STREAM streamlink --hls-live-edge 5 --hls-segment-threads 5 'hls://https://host/playlist.m3u8' best streamlink --twitch-low-latency -p mpv -a '--cache=yes --demuxer-max-bytes=750k' twitch.tv/CHANNEL best
URL    A URL to attempt to extract streams from.
Usually, the protocol of http(s) URLs can be omitted ("https://"), depending on the implementation of the plugin being used.
Alternatively, the URL can also be specified by using the --url option.
STREAM Stream to play.
Use best or worst for selecting the highest or lowest available quality.
Fallback streams can be specified by using a comma-separated list:
"720p,480p,best"
If no stream is specified and --default-stream is not used, then a list of available streams will be printed.
--help Show this help message and exit.
--version Show version number and exit.
--plugins Print a list of all currently installed plugins.
--plugin-dirs DIRECTORY Attempts to load plugins from these directories.
Multiple directories can be used by separating them with a comma.
--can-handle-url URL Check if Streamlink has a plugin that can handle the specified URL.
Returns status code 1 for false and 0 for true.
Useful for external scripting.
--can-handle-url-no-redirect URL Same as --can-handle-url but without following redirects when looking up the URL.
--config FILENAME Load options from this config file.
Can be repeated to load multiple files, in which case the options are merged on top of each other where the last config has highest priority.
-l LEVEL
--loglevel LEVEL Set the log message threshold.
Valid levels are: none, error, warning, info, debug, trace
--logfile FILE Append log output to FILE instead of writing to stdout/stderr.
User prompts and download progress won't be written to FILE.
A value of - will set the file name to an ISO8601-like string and will choose the following default log directories.
%TEMP%\streamlink\logs
${HOME}/Library/Logs/streamlink
Linux/BSD:
${XDG_STATE_HOME:-${HOME}/.local/state}/streamlink/logs
--quiet Hide all log output.
Alias for "--loglevel none".
--json Output JSON representations instead of the normal text output.
--auto-version-check {yes,true,1,on,no,false,0,off} Enable or disable the automatic check for a new version of Streamlink.
Default is: "no".
--version-check Runs a version check and exits.
--locale LOCALE The preferred locale setting, for selecting the preferred subtitle and audio language.
The locale is formatted as [language_code]_[country_code], eg. en_US or es_ES.
Default is: system locale.
--interface INTERFACE Set the network interface.
--ipv4 Resolve address names to IPv4 only. This option overrides -6.
--ipv6 Resolve address names to IPv6 only. This option overrides -4.
-p COMMAND
--player COMMAND Player to feed stream data to. By default, VLC will be used if it can be found in its default location.
This is a shell-like syntax to support using a specific player:
streamlink --player=vlc <url> [stream]
Absolute or relative paths can also be passed via this option in the event the player's executable can not be resolved:
streamlink --player=/path/to/vlc <url> [stream] streamlink --player=./vlc-player/vlc <url> [stream]
To use a player that is located in a path with spaces you must quote the parameter or its value:
streamlink "--player=/path/with spaces/vlc" <url> [stream] streamlink --player "C:\path\with spaces\mpc-hc64.exe" <url> [stream]
Options may also be passed to the player. For example:
streamlink --player "vlc --file-caching=5000" <url> [stream]
As an alternative to this, see the --player-args parameter, which does not log any custom player arguments.
-a ARGUMENTS
--player-args ARGUMENTS This option allows you to customize the default arguments which are put together with the value of --player to create a command to execute.
It's usually enough to only use --player instead of this unless you need to add arguments after the player's input argument or if you don't want any of the player arguments to be logged.
The value can contain formatting variables surrounded by curly braces, { and }. If you need to include a brace character, it can be escaped by doubling, e.g. {{ and }}.
Formatting variables available:
{playerinput} This is the input that the player will use. For standard input (stdin), it is -, but it can also be a URL, depending on the options used.
{filename} The old fallback variable name with the same functionality.
streamlink -p vlc -a "--play-and-exit {playerinput}" <url> [stream]
NOTE: When neither of the variables are found, {playerinput} will be appended to the whole parameter value, to ensure that the player always receives an input argument.
--verbose-player Allow the player to display its console output.
--player-fifo
--fifo Make the player read the stream through a named pipe instead of the stdin pipe.
--player-http Make the player read the stream through HTTP instead of the stdin pipe.
--player-continuous-http Make the player read the stream through HTTP, but unlike --player-http it will continuously try to open the stream if the player requests it.
This makes it possible to handle stream disconnects if your player is capable of reconnecting to a HTTP stream. This is usually done by setting your player to a "repeat mode".
--player-external-http Serve stream data through HTTP without running any player. This is useful to allow external devices like smartphones or streaming boxes to watch streams they wouldn't be able to otherwise.
Behavior will be similar to the continuous HTTP option, but no player program will be started, and the server will listen on all available connections instead of just in the local (loopback) interface.
The URLs that can be used to access the stream will be printed to the console, and the server can be interrupted using CTRL-C.
--player-external-http-port PORT A fixed port to use for the external HTTP server if that mode is enabled. Omit or set to 0 to use a random high ( >1024) port.
--player-passthrough TYPES A comma-delimited list of stream types to pass to the player as a URL to let it handle the transport of the stream instead.
Stream types that can be converted into a playable URL are:
• hls
• rtmp
Make sure your player can handle the stream type when using this.
--player-no-close By default Streamlink will close the player when the stream ends. This is to avoid "dead" GUI players lingering after a stream ends.
It does however have the side-effect of sometimes closing a player before it has played back all of its cached data.
This option will instead let the player decide when to exit.
-t TITLE
--title TITLE This option allows you to supply a title to be displayed in the title bar of the window that the video player is launched in.
This value can contain formatting variables surrounded by curly braces, { and }. If you need to include a brace character, it can be escaped by doubling, e.g. {{ and }}.
This option is only supported for the following players: mpv, potplayer, vlc.
VLC specific information: VLC has certain codes you can use inside your title.  These are accessible inside --title by using a backslash before the dollar sign VLC uses to denote a format character.
e.g. to put the current date in your VLC window title, the string "$A" could be inserted inside your --title string.
A full list of the format codes VLC uses is available here: https://wiki.videolan.org/Documentation:Format_String/
mpv specific information: mpv has certain codes you can use inside your title.  These are accessible inside --title by using a backslash before the dollar sign mpv uses to denote a format character.
e.g. to put the current version of mpv running inside your mpv window title, the string "${{mpv-version}}" could be inserted inside your --title string.
A full list of the format codes mpv uses is available here: https://mpv.io/manual/stable/#property-list
Formatting variables available to use in --title:
{title} If available, this is the title of the stream.  Otherwise, it is the string "Unknown Title"
{author} If available, this is the author of the stream.  Otherwise, it is the string "Unknown Author"
{category} If available, this is the category the stream has been placed into.
• For Twitch, this is the game being played
• For YouTube, it's the category e.g. Gaming, Sports, Music...
Otherwise, it is the string "No Category"
{game} This is just a synonym for {category} which may make more sense for gaming oriented platforms. "Game being played" is a way to categorize the stream, so it doesn't need its own separate handling.
{url}  URL of the stream.
{time} The current timestamp, which can optionally be formatted via {time:format}.  This format parameter string is passed to Python's datetime.strftime() method, so all usual time directives are available. The default format is "%Y-%m-%d_%H-%M-%S".
streamlink -p vlc --title "{title} -!- {author} -!- {category} \$A" <url> [stream] streamlink -p mpv --title "{title} -- {author} -- {category} -- (\${{mpv-version}})" <url> [stream]
-o FILENAME
--output FILENAME Write stream data to FILENAME instead of playing it.
You will be prompted if the file already exists.
The formatting variables available for the --title option may be used.  Unsupported characters in substituted variables will be replaced with an underscore.
--force When using -o or -r, always write to file even if it already exists.
--force-progress When using -o or -r, show the download progress bar even if there is no terminal.
--stdout Write stream data to stdout instead of playing it.
-r FILENAME
--record FILENAME Open the stream in the player, while at the same time writing it to FILENAME.
-R FILENAME
--record-and-pipe FILENAME Write stream data to stdout, while at the same time writing it to FILENAME.
--fs-safe-rules The rules used to make formatting variables filesystem-safe are chosen automatically according to the type of system in use. This overrides the automatic detection.
Intended for use when Streamlink is running on a UNIX-like OS but writing to Windows filesystems such as NTFS; USB devices using VFAT or exFAT; CIFS shares that are enforcing Windows filename limitations, etc.
These characters are replaced with an underscore for the rules in use:
POSIX  : \x00-\x1F / Windows: \x00-\x1F \x7F " * / : < > ? \ |
--url URL A URL to attempt to extract streams from.
Usually, the protocol of http(s) URLs can be omitted (https://), depending on the implementation of the plugin being used.
This is an alternative to setting the URL using a positional argument and can be useful if set in a config file.
--default-stream STREAM Stream to play.
This is an alternative to setting the stream using a positional argument and can be useful if set in a config file.
--stream-url If possible, translate the resolved stream to a URL and print it.
--retry-streams DELAY Retry fetching the list of available streams until streams are found while waiting DELAY second(s) between each attempt. If unset, only one attempt will be made to fetch the list of streams available.
The number of fetch retry attempts can be capped with --retry-max.
--retry-max COUNT When using --retry-streams, stop retrying the fetch after COUNT retry attempt(s). Fetch will retry infinitely if COUNT is zero or unset.
If --retry-max is set without setting --retry-streams, the delay between retries will default to 1 second.
--retry-open ATTEMPTS After a successful fetch, try ATTEMPTS time(s) to open the stream until giving up.
Default is: 1.
--stream-types TYPES
--stream-priority TYPES A comma-delimited list of stream types to allow.
The order will be used to separate streams when there are multiple streams with the same name but different stream types. Any stream type not listed will be omitted from the available streams list. A * can be used as a wildcard to match any other type of stream, eg. muxed-stream.
Default is: "rtmp,hls,hds,http,akamaihd,*".
--stream-sorting-excludes STREAMS Fine tune the best and worst stream name synonyms by excluding unwanted streams.
If all of the available streams get excluded, best and worst will become inaccessible and new special stream synonyms best-unfiltered and worst-unfiltered can be used as a fallback selection method.
Uses a filter expression in the format:
[operator]<value>
Valid operators are >, >=, < and <=. If no operator is specified then equality is tested.
For example this will exclude streams ranked higher than "480p":
">480p"
Multiple filters can be used by separating each expression with a comma.
For example this will exclude streams from two quality types:
">480p,>medium"
--ringbuffer-size SIZE The maximum size of the ringbuffer. Mega- or kilobytes can be specified via the M or K suffix respectively.
The ringbuffer is used as a temporary storage between the stream and the player.  This allows Streamlink to download the stream faster than the player which reads the data from the ringbuffer.
The smaller the size of the ringbuffer, the higher the chance of the player buffering if the download speed decreases, and the higher the size, the more data can be use as a storage to recover from volatile download speeds.
Most players have their own additional cache and will read the ringbuffer's content as soon as data is available.  If the player stops reading data while playback is paused, Streamlink will continue to download the stream in the background as long as the ringbuffer doesn't get full.
Default is: "16M".
NOTE: A smaller size is recommended on lower end systems (such as Raspberry Pi) when playing stream types that require some extra processing (such as HDS) to avoid unnecessary background processing.
--stream-segment-attempts ATTEMPTS How many attempts should be done to download each segment before giving up.
This applies to all different kinds of segmented stream types, such as DASH, HDS, HLS, etc.
Default is: 3.