Package: puren-tonbo
-- coding: utf-8 --puren_tonboPure Plain Text Notes... with optional encryption.https://github.com/clach04/puren_tonbo/IMPORTANTbefore using the optionally encryption features,
ensure that it is legal in your country to use the specific encryption ciphers.
Some countries have also have restrictions on import, export, and usage seehttp://www.cryptolaw.org/cls-sum.htmBackgroundFeaturesGetting StartedRegular installWithout a source code checkoutFrom a source code checkoutExamplesptcatptgrepptigSample ptig sessionptig configpttkviewptpyvimptcipherTombo Blowfish CHIccrypt CPTOpenPGP - gpg / pgpOpenSSL 1.1.0 AESAES-256 zipVimCryptptcat/ptcipher with text editors like vimreadonly pipe into editorSciTE lua pluginSciTE lua install without extension managerSciTE lua install with an extension managerSciTE Python pluginvim pluginvim demovim plugin installvim plugin install linux/unixvim plugin install Microsoft Windowsvim config notesDevelopment and testingRun test suiteHigh Level OverviewThanksBackgroundPlain text notes search/edit tool that supports encrypted files, formats:AES-256 ZIP AE-1/AE-2created with 7z (does NOT support encrypted 7z files), WinZIP, WinRAR - AES256 (under Python 3 can also read (but not write) the original ZipCrypto zip format)AES-256-CTR PBKDF2 (iterations 1000)ccrypt- Rijndael-256 (no authentication)GnuPG (OpenPGP, gpg)-symmetricseehttps://tutonics.com/articles/gpg-encryption-guide-part-4-symmetric-encryption/#:~:text=Another%20type%20of%20cryptographic%20solution,also%20called%20a%20shared%20secret.OpenSSL 1.1.0 aes-256-cbc-symmetricAES-256-CBC encryption with pbkdf2.Tombo (chi)- blowfish-ECBvim VimCryptencrypted files READ ONLY - VimCrypt (1-3) zip, blowfish, and blowfish2ãƒ—ãƒ¬ãƒ¼ãƒ³ ãƒˆãƒ³ãƒœ
PurÄ“ntonboå¹³æ˜“ èœ»è›‰http://tombo.sourceforge.jp/En/https://github.com/clach04/chi_iohttps://hg.sr.ht/~clach04/pytombo- NOTE Puren Tonbo is intended to replace PyTomboFeaturesPlain text files notes (potentially with no formatting or in Markdown, reStructuredText, etc.)Nested directories of notesSupports reading and writing from/to encrypted files that are compatible with other formats/tools (there is no intention to create a new crypto format/algorithm in this tool)Currently limited to local file system and stdin/out for filesCommand line tools;ptcatandptcipherto encrypt/decrypt and view plain text filesptcipher- process raw binary files, controlled via command line and environment variablesptcat- in addition to command line and environment variables, also has an (optional) config file and the concept of a root directory of notesptgrep- a grep,ack,ripgrep,silver-searcher,psslike tool that works on encrypted (and plain text) filesptigan interactive grep like tool that can also view/editptpyvima vim-like editor that works on encrypted (and plain text) filesGetting Startedsudo apt-get install python-tk
sudo apt-get install ccryptRegular installpip install "puren_tonbo[all]"Without a source code checkoutPicking up the latest versionpip uninstall puren_tonbo ; python -m pip install --upgrade git+https://github.com/clach04/chi_io.git  git+https://github.com/clach04/puren_tonbo.git

# sanity check, and dump sample config to stdout
ptconfig
python -m puren_tonbo.tools.ptconfigFrom a source code checkout# pip uninstall puren_tonbo
# python -m pip install -r requirements.txt
# TODO requirements_optional.txt
python -m pip install -e .


# sanity check, and dump sample config to stdout
ptconfig
python -m puren_tonbo.tools.ptconfigExamplesptconfig
python -m puren_tonbo.tools.ptconfigptcatptcat  puren_tonbo/tests/data/aesop.txt
ptcat --list-formats
python -m puren_tonbo.tools.ptcat --note-root . puren_tonbo/tests/data/aesop.txt
python -m puren_tonbo.tools.ptcat --note-root . puren_tonbo/tests/data/aesop.chi
python -m puren_tonbo.tools.ptcat  puren_tonbo/tests/data/aesop.txt

ptcat -p password puren_tonbo/tests/data/aesop_linux_7z.aes256.zip
python -m puren_tonbo.tools.ptcat -p password puren_tonbo/tests/data/aesop_linux_7z.aes256.zipptgrepA grep,ack,ripgrep,silver-searcher,psslike tool that works on encrypted (and plain text) files.Has similar parameters for ease of switching.Python 2.7 note for Windows. Non-ascii characters can cause Python exception/crashes UnicodeEncodeError when attempting to print Unicode characters, where as Python 3 does not. ptgrep implements a translation feature/hack which can be disabled (or tweaked) via the Operating System environment variablePTGREP_STDOUT_MODE. Valid options aredisabled,utf8, andascii:backslashreplace. For Python 2.7 under Microsoft Windows **only** (when neitherPYTHONIOENCODINGnorPYTHONUTF8have been set)ascii:backslashreplaceis the default and works in a similar fashion toPYTHONIOENCODING`.ptgrep better
ptgrep -i better
python -m puren_tonbo.tools.ptgrep -i better
python -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -i better
python -m puren_tonbo.tools.ptgrep -e -p password Better
python -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -e -p password BetterFind all instances of "king", case-insensitive (note; matchestaking):python -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -i kingFind all words "king", case-insensitive:python -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -i -r \bking\bfind different words with regexpython -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data    -r "cruel|better"
python -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -i -r "cruel|better"
python -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data    -r "cru.l|b.tter"find filenames with regexpython -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -y -r ^aesopfind filenames that have an ISO date in either dirname or filenamepython -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -y -r "202[0-9]-[0-9][0-9]-[0-9][0-9]"find filenames encrypted with regexpython -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -y -e -r ^aesopfind filenames ONLY encrypted with regexpython -m puren_tonbo.tools.ptgrep --note-root=puren_tonbo/tests/data -y -k -r ^aesopptigCommand line interactive search tool, that also supports viewing and editing.
Relies on ptgrep, seePTGREP_STDOUT_MODEnote.Also seehttps://github.com/clach04/puren_tonbo/wiki/tool-ptigptig
python -m puren_tonbo.tools.ptigSample ptig session$ ptig --note-root=puren_tonbo/tests/data
3.8.10 (default, Mar 15 2022, 12:22:08)
[GCC 9.4.0]

Puren Tonbo puren_tonbo version 0.0.3.git
Formats:

              txt - RawFile - Raw file, no encryption support
               md - RawFile - Raw file, no encryption support
              chi - TomboBlowfish - Tombo Blowfish ECB (not recommended)
              gpg - GnuPG - gpg (GnuPG) symmetric 1.x and 2.x, does NOT uses keys
              asc - GnuPGascii - gpg (GnuPG) symmetric 1.x and 2.x, does NOT uses keys
              cpt - Ccrypt - ccrypt symmetric Rijndael
          aes.zip - ZipAES - AES-256 ZIP AE-1 DEFLATED (regular compression)
       aes256.zip - ZipAES - AES-256 ZIP AE-1 DEFLATED (regular compression)
           aeszip - ZipAES - AES-256 ZIP AE-1 DEFLATED (regular compression)
          old.zip - ZipAES - AES-256 ZIP AE-1 DEFLATED (regular compression)
openssl_aes256cbc_pbkdf2_10k - OpenSslEnc10k - OpenSSL 1.1.0 pbkdf2 iterations 10000 aes-256-cbc
 aes256stored.zip - ZipNoCompressionAES - AES-256 ZIP AE-1 STORED (uncompressed)
    oldstored.zip - ZipNoCompressionAES - AES-256 ZIP AE-1 STORED (uncompressed)
   aes256lzma.zip - ZipLzmaAES - AES-256 ZIP AE-1 LZMA
  aes256bzip2.zip - ZipBzip2AES - AES-256 ZIP AE-1 BZIP2
         vimcrypt - VimDecrypt - vimcrypt 1, 2, 3
        vimcrypt1 - VimDecrypt - vimcrypt 1, 2, 3
        vimcrypt2 - VimDecrypt - vimcrypt 1, 2, 3
        vimcrypt3 - VimDecrypt - vimcrypt 1, 2, 3

Libs:
        chi_io.implementation: using PyCrypto 3.15.0
        python-gnupg version: 0.5.0
        gpg version: (2, 2, 19)
        pyzipper version: 0.3.6

ptig: ðŸ”Ž rg better
Query time: 0.01 seconds
ptig: ðŸ”Ž set ic
ptig: ðŸ”Ž rg better
[1] puren_tonbo/tests/data/aesop.txt
7:Better no rule than cruel rule.
Query time: 0.01 seconds
ptig: ðŸ”Ž find ccrypt
[1] puren_tonbo/tests/data/aesop_win_ccrypt.cpt
Query time: 0.00 seconds
ptig: ðŸ”Ž f ccrypt
[1] puren_tonbo/tests/data/aesop_win_ccrypt.cpt
Query time: 0.00 seconds
ptig: ðŸ”Ž cat 0
Password for file aesop_win_ccrypt.cpt:
ptig: ðŸ”Ž set search_encrypted=True
ptig: ðŸ”Ž rg better
[1] puren_tonbo/tests/data/aesop.chi
7:Better no rule than cruel rule.
..... Truncated
[21] puren_tonbo/tests/data/aesop_win_winrar.aes256stored.zip
7:Better no rule than cruel rule.
Query time: 0.32 seconds
ptig: ðŸ”Ž help

Documented commands (type help <topic>):
========================================
EOF  c    config  edit  f     g     help  ptpyvim  quit     rg   ver      vi
bye  cat  e       exit  find  grep  info  pyvim    results  set  version  vim

ptig: ðŸ”Ž exit
Quitting...ptig configConfig for ptig is the regular config filept.json, with additional (optional) ptig section:{
    "_version_created_with": "0.0.dev3",
    "codec": [
        "utf8",
        "cp1252"
    ],
    "default_encryption_ext": "chi",
    "default_text_ext": "txt",
    "note_root": "C:\\Users\\yourname\\tombo",
    "ptig": {
        "#init": ["set ic", "set enc"],
        "init": ["set ic"],
        "editor": "start scite",
        "editors": {
            "encscite": "C:\\programs\\encscite\\prog\\encscite.bat",
            "pttkview": "pttkview",
            "scite": "scite",
            "gvim": "gvim",
            "vim": "vim"
        },
        "file_browser": "explorer",
        "prompt": "ptig: ? ",
        "use_pager": false
    }
}NOTE options for editor along withinitwhich is alistof commands to issue on start up.
For example, enabling case insensitive search. There is a commented out example which enables encrypted file search.pttkviewA simple GUI viewer using the tk toolkit:python -m puren_tonbo.tools.pttkview -h
pttkview -hptpyvimIf pyvim is available, ptpyvim wraps encryption/decryption support.ptpyvim
python -m puren_tonbo.tools.ptpyvimptcipherptcipher is a tool for dealing with raw (binary, i.e. bytes rather than characters) files for encryption/decryption.
All options are controlled via command line flag and operating system environment variables.Assuming installed:ptcipher -hFrom source code checkout:python -m puren_tonbo.tools.ptcipher -h
python2 -m puren_tonbo.tools.ptcipher -h
python3 -m puren_tonbo.tools.ptcipher -hQuick demo:ptcipher --password password --decrypt puren_tonbo/tests/data/aesop.chi
ptcipher --password password --decrypt puren_tonbo/tests/data/aesop_linux_7z.aes256.zip
python -m puren_tonbo.tools.ptcipher --password password --decrypt puren_tonbo/tests/data/aesop.chiTombo Blowfish CHISymmetric encryption/decryption from passphase.Compatible withhttp://tombo.osdn.jp/En/(and others, for example, Kumagusu on Android).ptcipher -e -p test README.md -o README.chi

ptcipher -v -p test README.chiThe chi file can also be read/written by Tombohttp://tombo.sourceforge.jp/En/and clonesccrypt CPTSymmetric encryption/decryption from passphase.Tested with ccrypt 1.11 and 1.10 (32-bit and 64-bit Intel x86/x64 and arm).Requires a ccrypt binary, download fromhttps://ccrypt.sourceforge.net/(or debian apt). ccrypt binary/executable needs to be in the path or
the environment variable CCRYPT_EXE needs to have the (full) path.
NOTE Under Microsoft Windows, is the ccrypt.exe is in a path with
spaces, do NOT use double quotes in the SET. Example:set CCRYPT_EXE=C:\3rd party bins\ccrypt.exepython -m puren_tonbo.tools.ptcipher --password password puren_tonbo/tests/data/aesop_win_ccrypt.cpt
ptcipher --cipher=cpt -e -p test README.md -o README.cpt
python -m puren_tonbo.tools.ptcipher --cipher=cpt -e -p test README.md -o README.cpt

ccrypt -c README.cpt
ccrypt -c -K test README.cptOpenPGP - gpg / pgpSymmetric encryption/decryption from passphase, key support not explictly implemented.Requires a gpg binary, download fromhttps://gnupg.org/download/python -m puren_tonbo.tools.ptcipher --cipher=asc -e -p test README.md -o README.asc
python -m puren_tonbo.tools.ptcipher --cipher=gpg -e -p test README.md -o README.gpg

gpg  --pinentry-mode=loopback --decrypt  --passphrase test README.gpgAlso seeencryptclifromhttps://github.com/evpo/EncryptPad/OpenSSL 1.1.0 AESOpenSSL 1.1.0+ compatible (with a very small subset of options).ptcat --note-root=. puren_tonbo/tests/data/aesop_win.openssl_aes256cbc_pbkdf2_10kIntended to allow decryption of files generated with OpenSSL 1.1.0 and vice-versa. Supported OpenSSL flags/formats:openssl enc -e aes-256-cbc -salt -pbkdf2 -iter 10000 -in in_file -base64 -out out_file
openssl dec -d aes-256-cbc -salt -pbkdf2 -iter 10000 -in in_file -base64 -out out_file

echo hello| openssl enc -e aes-256-cbc -salt -pbkdf2 -iter 10000 -in - -base64 -out - -pass pass:password
echo hello| openssl enc -e -aes-256-cbc -in - -out - -salt -pbkdf2 -iter 10000  -pass pass:passwordNOTE PBKDF2 iteration count of 10,000 is the default in OpenSSL 1.1.1 and is considered too few in 2023.
Older versions of OpenSSL did not support; PBKDF2 (and ergo iterations) and salt and used a much weaker KDF.Supports binary/raw and base64 encoded/ASCII armored files.
ONLY supports aes-256-cbc with salt and pbkdf2 KDF with 10,000 interations.AES-256 zipSymmetric encryption/decryption from passphase.ptcipher -e -p test README.md -o README.aes256.zip

ptcipher -p test README.aes256.zipThe aes256.zip file can also be read/written by 7-Zip, WinRAR, WinZIP, etc. that support AES zip files.For example, 7z can read and write AES zip files:7z a -tzip -mem=AES256 -ptest README.aes256.zip README.md
7z x -ptest README.aes256.zipVimCryptSymmetric encryption/decryption from passphase.NOTE not implemented in nvim / neovim.In vim the easiest way to get the newest encryption mode/format, for a file:vim -c ":setlocal cm=blowfish2"  test.vimcrypt3then issue::Xwill be prompted for password, can then edit/save as per normal.To see encryption mode::setlocal cm?ptcipher demo:python -m puren_tonbo.tools.ptcipher -p test test.vimcrypt3ptcat/ptcipher with text editors like vimreadonly pipe into editorQuick and easy view/read ONLY of encrypted file with vim, without updating vim config.NOTE call vim (or neovim) with options to set "private" mode:-to read from stdin instead of a filename, avoid plaintext hitting the disk-nturns off swap file - use memory only-iturns off .viminfoTODO disable undo fileUse a pipe (cross platform)ptcat FILE | vim - -n -i "NONE"Use bash shellprocess substitutionfeature for editors that don't support stdin (Linux/Unix only, avoids directly calling mkfifo and cleaning up named pipes)scite < (ptcat FILE)
   ptcat FILE | scite /dev/stdin
   scite <(python -m puren_tonbo.tools.ptcat  puren_tonbo/tests/data/aesop.txt)Puren Tonbo will prompt for passwords and the decrypted content should not hit the file system.Option 1 can be used with other tools that take in stdin, option 2 can be used with any tool that takes in a filename.Caution!don't save the raw fileensure now backup, swap, undo file, etc.. get createdhttps://vi.stackexchange.com/questions/6177/the-simplest-way-to-start-vim-in-private-modeSciTE lua pluginTested with versions: v4.0.0, 4.1.5, and 5.3.5Seept_scite.lua- read and write support (tested Windows).
NOTE due to lua popen() any helpful error text/information is missing. Diagnosing failures can be difficult, even for trivial errors like directory/disk does not exist as this is not reported to the lua runtime.
Also see SciTE Python plugin for read and write support with stderror support.By default usesptcipherin path, override viaPTCIPHER_EXEenvironment variable.Does NOT prompt for password, requires settingPT_PASSWORDenvironment variable or use of keyring.Can be used standalone or with ParsKorata (http://lua-users.org/wiki/ParsKorata) mini/simple ExtMan compatible scriptUntested with ExtMan -http://lua-users.org/wiki/SciteExtManNeed to edit SciTEUser.properties:Windows%USERPROFILE%"\SciTEUser.propertiesor%APPDATA%\scite\SciTEUser.propertiesUnix~/.SciTEUser.propertiesor$HOME/.SciTEUser.propertiesAlternatively, launch scite, then open Options, Open User (or GLobal) Options File.Windows NOTE to avoid a (typically black) CMD/Command window showing up use SciTE 4.4.4 or later and setcreate.hidden.consolein SciTEUser.properties:# https://groups.google.com/g/scite-interest/c/QOhizNSEejU/m/qXslloxnCgAJ
# SciTE 4.4.4 on Windows adds create.hidden.console option to stop console window flashing when Lua script calls os.execute or io.popen.
create.hidden.console=1
# TODO see if this can be set in lua code, to make config easierSciTE lua install without extension managerIf not using a plugin extension manager can simply setpt_scite.luaas starting lua script.
Edit SciTEUser.properties to set lua script:...
ext.lua.startup.script=C:\code\py\puren_tonbo\pt_scite.lua
create.hidden.console=1
...SciTE lua install with an extension managerIf using an extman like system:...
# This is a simplified ExtMan
# requires each plugin/add-on to be "registered" or declared, white listed, etc.
ext.lua.startup.script=C:\code\scite\extman\parskorata_extman.lua
...Then editparskorata_extman.luato add to end:mgr:load_files{'pt_scite.lua'}NOTE not needed with original full ExtMan.SciTE Python pluginSeescite with Python README- read and write support, Windows only (needs polishing).By default usesptcipherin path, override viaPTCIPHER_EXEenvironment variable.Does NOT prompt for password, requires settingPT_PASSWORDenvironment variable.vim pluginTested under Linux with vim 8.0 and 8.1. under x86, x64, and arm 32-bit.Seept.vim- Linux/Unix/Cygwin only for now.By default usesptcipherin path, override viaPTCIPHER_EXEenvironment variable.Will prompt for password, which can be skipped by settingPT_PASSWORDenvironment variable.vim demoAssuming puren_tonbo has been installed andptcipheris in the
path (and in source code checkout):vim -u pt.vim  puren_tonbo/tests/data/aesop.chiAnd enter in the test password,password.vim plugin installAs per example above, can use the-uparameter but this overrides existing settings.Vim 8 supports plugins packs, to install:vim plugin install linux/unixIf you do not already have any plugin packs, need to create directory, e.g.:mkdir -p ~/.vim/pack/bundle/start/Wherebundleis user decided. For the rest of the documentation replacebundlewith your directory name.Install from checkout:mkdir -p ~/.vim/pack/bundle/start/puren_tonbo/plugin/
cp pt.vim ~/.vim/pack/bundle/start/puren_tonbo/plugin/Then can call vim without-u:vim puren_tonbo/tests/data/aesop.chi
gvim puren_tonbo/tests/data/aesop.chivim plugin install Microsoft WindowsNOTE not working under Windows :-(
Appear to be buffered files issues with vim/python interaction.See linux notes, instead of~/.vimuse %USERPROFILE%\vimfiles (or$VIM_INSTALLATION_FOLDER\vimfiles)mkdir %USERPROFILE%\vimfiles\pack\bundle\start\puren_tonbo\plugin
copy pt.vim %USERPROFILE%\vimfiles\pack\bundle\start\puren_tonbo\pluginvim config notesSeehttps://vim.fandom.com/wiki/Encryptionfor how to configure vim with external tools for (view and edit) of encrypted files with autocmd.
NOTE under Windows buffered IO can interfere with vim interactions.
TODO consider using (OpenSSL)https://www.vim.org/scripts/script.php?script_id=2012as a model for vim plugin (uses functions), also see:https://aweirdimagination.net/2019/03/24/encrypted-files-in-vim/https://git.aweirdimagination.net/perelman/openssl.vimhttps://github.com/MoserMichael/vimcrypt2.https://github.com/MoserMichael/vimcryptDevelopment and testingPuren Tonbo is implemented in Python, with support for Python 3.x and 2.7.Run test suitepython -m puren_tonbo.tests.testsuite
python -m puren_tonbo.tests.testsuite -v 2>&1 |grep -i skippedHigh Level OverviewAll encryption/decryption is file object based.
Low level routines (EncryptedFile) use file-like objects, for in-memory encryption/decryption use BytesIO
(see test suite,puren_tonbo/tests/testsuite.py).There is also the note abstraction (FileSystemNotes) which is filename based.ThanksThanks and kudos to:Tomohisa Hirami - the original creator of Tombomaxpat78for the Python 2 (and 3) fall back code for AES zip support,relicensed with permissionfromhttps://github.com/maxpat78/CryptoPadNoah Spurrier who's Public Domain OpenSSL vim plugin is the inspiration for the PT vim support (using functions)https://www.vim.org/scripts/script.php?script_id=2012