Package: kolombos
CLI application for visualising usually invisible characters and bytes:whitespace characters;ASCII control characters;ANSI escape sequences;UTF-8 encoded characters;binary data.InstallationViapipxpipx install kolombosWithoutpipxSystem-wide install (sudo)python -m pip install kolombosUser install (nosudo)python -m pip install --user kolombos
export PATH="${PATH}:${HOME}/.local/bin/"UsageApplication can be useful for a variety of tasks, e.g. browsing unknown data formats, searching for patterns or debugging combinations of SGR sequences.USAGE                                                                                                                                                   
  kolombos [[--text] | --binary] [<options>] [--demo | <file>]     
  
INPUT
  <file>                  file to read from; if empty or "-", read stdin
                          instead; ignored if --demo is present
  -M, --demo              show output examples and exit; see --legend for the
                          description
OPERATING MODE
  -t, --text              open file in text mode [this is a default]
  -b, --binary            open file in binary mode
  -l, --legend            show annotation symbol list and exit
  -v, --version           show app version and exit
  -h, --help              show this help message and exit 

[...]Text mode and binary modekolomboscan work in two primary modes: text and binary. The differences between them are line-by-line input reading in text mode vs. fixed size byte chunk reading in binary mode, and extended output in binary mode, which consists of text representation (similar to text mode) and hexademical byte values.As you can see, some of the settings are shared between both modes, while the others are unique for one or another.GENERIC OPTIONS
  -f, --buffer <size>     read buffer size, in bytes [default: 4096]
  -L, --max-lines <num>   stop after reading <num> lines [default: no limit]
  -B, --max-bytes <num>   stop after reading <num> bytes [default: no limit]
  -D, --debug             enable debug mode; can be used from 1 to 4 times,
                          each level increases verbosity (-D|DD|DDD|DDDD)
  --color-markers         apply SGR marker format to themselves

TEXT MODE OPTIONS
  -m, --marker <details>  marker details: 0 is none, 1 is brief, 2 is full
                          [default: 0]
  --no-separators         do not print ⢸separators⡇ around escape sequences
  --no-line-numbers       do not print line numbers

BINARY MODE OPTIONS
  -w, --columns <num>     format output as <num>-columns wide table [default: auto]
  -d, --decode            decode valid UTF-8 sequences, print as unicode chars
  --decimal-offsets       output offsets in decimal format [default: hex format]
  --no-offsets            do not print offsets

[...]Character classesThere are 6 differentcharacter classes, and each of those can be displayed normally, highlighted (orfocused) or ignored.outputcharacter classbyte rangesfocus flagignore flagexampleswhitespace09-0d20-s-Sspace, line feed, carriage returncontrol char01-080e-1f-c-Cnull byte, backspace, deleteprintable char21-7e-p-PASCII alphanumeric and punctuation characters: A-Z, a-z, 0-9, (), []escape sequence1b[..]-e-EANSI escape sequences controlling cursor position, color, font styling, and other terminal optionsUTF-8 sequencevarious-u-Uvalid UTF-8 byte sequences that can be decoded into Unicode charactersbinary data80-ff-i-Istandalone non-(7 bit)-ASCII bytesExamplesControl and whitespace charactersLet's take a look at one of the files from somebody's home directory —.psql_history. At the first sight it's a regular text file:But what if we look a bit more deeper into it?kolombosshows us hidden until now characters — not only spaces and line breaks, but even more: some control characters, namely01START OF HEADINGASCII bytes, whichpostgresqluses to store multiline queries.Red symbol is an example ofmarker— special sigil that indicates invisibile character in the input. Sigils were selected with a focus on dissimilarity and noticeability, which helps to detect them as soon as possible. Control char and escape sequence markers also provide some details about original input byte(s); there are three different levels of these details in text mode.Level 0 is no details, just the marker itself.Level 1 is medium details (this is a default) — one extra character for control chars and varying amount for escape sequences. For most of the control characters the second char corresponds to their caret notation, e.g.ⱯAshould be read as^A[wiki].Level 2 is maximuim amount of verbosity. For control chars it's their 2-digit hexademical value. Also note-coption in the last example below — which tells the application to highlight control characters and make them even more noticable.Some of the control characters has unique sigils — for example, null byte (seeLegend).A few more examples of option combinations. First one is--focus-spaceflag, or-s, which can be useful for a situations where whitespaces are the points of interest, but input is a mess of different character classes.Second example is a result of running the app with--ignore-spaceand--ignore-printableoptions; as you can see, now almost nothing is in the way of observing our precious control characters (if that's what you were after, that is):ANSI escape sequecncesEscape sequences and their overlapping combinations were the main reason for me to develop this application. For those who doesn't know much about them here's some comprehensive materials:[one][two].kolomboscan distiguish a few types of escape sequences, but most interesting and frequent type isSGR sequence, which consists of escape control character1b, square bracket[, one or more digit params separated by;andmcharacter (terminator). Let me illustrate.SGR sequences are used for terminal text coloring and formatting. Consider this command with the following output:kolomboscan show us what exactly is happening out there:There are 3 different types of markers in the example above:ǝis a sigil for regular SGR sequence (which for example sets the color of the following text to red);θis aresetSGR sequence (ESC[0m) which completely disables all previously set colors and effects;ϽisCSI sequence(more common sequence class which includes SGRs) — they also begin withESC[, but have different terminator characters; in general, they control cursor position.Other types are listed inLegendsection.For this example binary more would be more convenient.As a rule of a thumb, the only underlined bytes inkolombo's output are the bytes that correspond to escape sequences' params, introducers or terminators (but not the1b|ESCcharacter itself, though).Now it's clear where and which sequences are located:ǝ[35m— SGR that sets text color tomagenta;Ͻ[K— CSI that erases all characters from cursor to the end of the current line;θ[m— SGR that resets, or disables all formatting;ǝ[01;91m— SGR that sets text style toboldand text color tobright red, etc.There is an option of highlighting SGR sequences with their own colors:--color-markers, which is disabled by default. In this particular case, even more clear picture can be seen after launching the app with-Poption (--ignore-printable):Also notice that in binary mode each byte of input corresponds strictly to one hex value and one text representation character. That means that option-mis always equal to 2 (maximum verbosity) and cannot be changed.UTF-8 and binary dataThere is no limitation for input bytes range inkolombostext mode — binary data will be displayed with the replacement character --Ḇ:But it's better and faster to work with binary data in binary mode. Valid UTF-8 sequences and escape sequences can be seen even in completely random byte data:UTF-8 sequences in text mode are automatically decoded and displayed as Unicode characters. In binary mode for faster data processing they are displayed as boxes by default, but still can be decoded with-d|--decodeoption (note the same requirement as for escape sequence markers — hex value length must always correspond to text representation length):LegendEven more information can be seen after runningkolombos --legend.Changelogv1.5.4FIX: reverted default column amount in--demomodev1.5.3FIX: errors while processing SGR with subparams (e.g.4:3;)v1.5.2UPDATE: icon redrawv1.5.1FIX: packaging assetsv1.5NEW:--demomodev1.4.1Temporarily injectedpytermorv2.1v1.4REFACTOR: base colorsREFACTOR: extended legendDOCS:  update README and screenshotsv1.3Swap -D and -d (debug/decode)Make '--marker 0' default (was 1)Update legendUpgradepytermorto 2.1v1.2.1Minor update.v1.2Separators additional styling.Separators auto-hide from-m0.--no-sep[arators]launch option.rundev script for quick launch of repo versions.Updated output format of SGR color prefixes.SGR labels are now getting colored instead of marker details (if-m0is set).Updated legend.v1.1Additional separators around escape seqs (in text mode) for better readability.v1.0.2Added logos.Fixed pipy README images.v1.0.1First public version.