Package: pydbase
pydbase (see blockchain at the end)High speed database with key / dataThe motivation was to create a no frills way of saving / retrieving data.
It is fast, and the time test shows that this is an order of magnitude
faster than most mainstream databases. This is due to the engine's simplicity.
It avoids expensive computations in favor of quickly saving data.Fast data save / retrieveMostly ready for production. All tests pass. Please use caution, as this is new.
The command line tester can drive most aspects of this API; and it is somewhat
complete. It is also  good way to see the API / Module in action.APIThe module 'twincore' uses two data files and a lock file. The file
names are generated from the base name of the data file;
name.pydb for data; name.pidx for the index, name.lock for the lock file.
In case of frozen process the lock file times out in xx seconds
and breaks the lock. If the locking process (id in lockfile) does
not exist, the lock breaks immediately.Setting verbosity and debug level:twincore.core_quiet   = quiet
twincore.core_verbose = verbose
twincore.core_pgdebug = pgdebug
twincore.core_showdel = sdelx(Setting before data creation will display mesages from the construtor)Example db creation:core = twincore.TwinCore(datafile_name)Some basic ops:dbsize = core.getdbsize()

core.save_data(keyx, datax)
rec_arr = core.retrieve(keyx, ncount)
print("rec_arr", rec_arr)Structure of the data:32 byte header, starting with FILESIG

4 bytes    4 bytes          4 bytes         Variable
------------------------------------------------------------
RECSIG     Hash_of_key      Len_of_key      DATA_for_key
RECSEP     Hash_of_payload  Len_of_payload  DATA_for_payload

    .
    .

RECSIG     Hash_of_key      Len_of_key      DATA_for_key
RECSEP     Hash_of_payload  Len_of_payload  DATA_for_payload

where:
RECSIG="RECB" (record begin here)
RECSEP="RECS" (record separated here)
RECDEL="RECX" (record deleted)

Deleted records are marked with the RECSIG mutated from RECB to RECX

  Vacuum will remove the deleted records; Make sure your database has no
pending ops; or non atomic opts;

    (like: find keys - delete keys in two ops)

  New data is appended to the end, no duplicate filtering is done.
Retrieval is searched from reverse, the latest record with this key
is retrieved first. Most of the times this behavior is what we
want; also the record history is kept this way, also a desirable
behavior.The db exerciser executable script 'pydbase.py':The file pydbase.py exercises most of the twincore functionality. It also
provides examples of how to drive it.The command line utility's help response:Usage: pydebase.py [options] [arg_key arg_data]
 Options: -h         help (this screen)   -|-  -i   show deleted on dump
          -V         print version        -|-  -q   quiet on
          -d         debug level (0-10)   -|-  -v   increment verbosity level
          -r         randomize data       -|-  -w   write fixed record(s)
          -z         dump backwards(s)    -|-  -i   show deleted record(s)
          -U         Vacuum DB            -|-  -R   reindex / recover DB
          -I         DB Integrity check   -|-  -c   set check integrity flag
          -s         Skip to count recs   -|-  -K   list keys only
          -y  key    find by key          -|-  -m   dump data to console
          -o  offs   get data from offset -|-  -e  offs   delete at offset
          -u  rec    delete at position   -|-  -g  num    get number of recs.
          -k  key    key to save          -|-  -a  str    data to save
          -S         print num recs       -|-  -D  key    delete by key
          -n  num    number of records    -|-  -t  key    retrieve by key
          -p  num    skip number of records on get
          -l  lim    limit number of records on get
          -x  max    limit max number of records to get
          -f  file   input or output file (default: 'data/pydbase.pydb')
The default action is to dump records to screen in reverse order.
On the command line, use quotes for multi word arguments.Comparison to other databases:This comparison is to show the time it takes to write 500 records.
In the tests the record size is about the same (Hello, 1  /vs/ "Hello", 1)
Please see the sqlite_test.sql for details of data output;The test can be repeated with running the 'time.sh' script file.
Please note the the time.sh clears all files test_data/* for a fair test.sqlite time test, writing 500 records ...
    real	0m1.730s
    user	0m0.110s
    sys	0m0.455s

    pydbase time test, writing 500 records ...
    real	0m0.120s
    user	0m0.075s
    sys	0m0.044s

    -rw-r--r-- 1 peterglen users  4032 Feb  7 15:35 pydb_test.pidx
    -rw-r--r-- 1 peterglen users 15032 Feb  7 15:35 pydb_test.pydb
    -rw-r--r-- 1 peterglen users 20480 Feb  7 15:35 sqlite_test.dbPlease mind the fact that the sqlite engine has to do a lot of parsing which we
skip doing; That is why pydbase is more than an order of magnitude faster ...Saving more complex dataThe database saves a key / value pair. However, the key can be mutated to contain
meta data. (for example adding a string in front of it.)
[like: CUST_  for customer data / details]
Also the key can be made unique by adding a UUID to it.The data can consist of any text / binary. The library pypacker.py can pack any data
into a string; A copy of pypacker is included here.pypacker.pyThis module can pack arbitrary python data into a string; which can be used to store
anything in the pydbase key / data sections.Example from running testpacker.py:org: (1, 2, 'aa', ['bb', b'dd'])
    packed: pg s4 'iisa' i4 1 i4 2 s2 'aa' a29 'pg s2 'sb' s2 'bb' b4 'ZGQ=' '
    unpacked: [1, 2, 'aa', ['bb', b'dd']]
    rec_arr: pg s4 'iisa' i4 1 i4 2 s2 'aa' a29 'pg s2 'sb' s2 'bb' b4 'ZGQ=' '
    rec_arr_upacked: [1, 2, 'aa', ['bb', b'dd']]
    (Note: the decode returns an array of data; use data[0] to get the original)There is also the option of using pypacker on the key itself. Because the key
is identified by its hash, there is no speed penalty; Note that the hash is a 32 bit
one; collisions are possible, however unlikely; To compensate, make sure you compare the
key proper with the returned key.MaintenanceThe DB can rebuild its index and purge all deleted records. In the test utility
the options are:./pydbase.py -U     for vacuum (add -v for verbosity)The database is re-built, the deleted entries are purged, the damaged data (if any)
is saved into a separate file, created with the same base name as the data base,
with the '.perr' extension../pydbase.py -R     for re-indexThe index is recreated; as of the current file contents. This is useful if
the index is lost (like copying the data only)If there is a data file without the index, the re-indexing is called
automatically.   In case of deleted data file, pydbase will recognize
the dangling index and nuke it by renaming it to
orgfilename.pidx.dangle (Tue 07.Feb.2023 just deleted it);Note about the 'garbage' and 'old_tries' directory ... older stuff I
tried; some are really useful; For instance take a look at the
simplifier: an array of indexes to save offsets and lengths; The
simplifier makes one range out of overlapping or close to each other
ranges. (min. dist=4)The database grows with every record added to it. It does not check if
the particular record already exists. It adds the new record version to
the end; Retrieving starts from the end, and the data retrieved
(for this particular key) is the last record saved. All the other records
of this key are also there in chronological (save) order. Miracle of
record history archived by default.To clean the old record history, one may delete all the records with
this same key, except the last one.Blockcain implementationThe database is extended with a blockhcain implementation. The new class
is called twinchain; and it is a class derived from twincore.To drive the blockchain, just use the append method. Th database will calculate
all the hashes, integrate it into the existing chain with the new item getting
a backlink field. This field is calulated based upon the previous record's
hash and the previous record frozen date. This assures that identical data
will have a different hash, so data cannot be anticipated based upon its hash
alone.To drive it:core = twinchain.TwinChain()    # Takes an optional file name
    #core.core_verbose = verbose    # Give details as it operates
    core.append("The payload")      # Arbitrary dataIntegrity checkTwo levels; Level one is checking if the record checksums are correct;
Level two checks if the linkage is correct.TODOSpeed this up by implementing this as a 'C' modulePyTestThe pytest passes with no errors;
The following (and more) test are created / executed:Test results:============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.0.0
rootdir: /home/peterglen/pgpygtk/pydbase
collected 32 items

test_bindata.py .                                                        [  3%]
test_create.py .....                                                     [ 18%]
test_del.py .                                                            [ 21%]
test_dump.py .                                                           [ 25%]
test_find.py ..                                                          [ 31%]
test_findrec.py ..                                                       [ 37%]
test_getrec.py .                                                         [ 40%]
test_integrity.py .                                                      [ 43%]
test_list.py ..                                                          [ 50%]
test_lockrel.py ..                                                       [ 56%]
test_multi.py ..                                                         [ 62%]
test_packer.py ......                                                    [ 81%]
test_randdata.py .                                                       [ 84%]
test_reindex.py .                                                        [ 87%]
test_search.py ...                                                       [ 96%]
test_vacuum.py .                                                         [100%]

============================== 32 passed in 3.24s ==============================
peterglen@getac110:~/pgpygtk/pydbase$History1.1         Tue 20.Feb.2024     Initial release
1.2.0       Mon 26.Feb.2024     Moved pip home to pydbase/
1.4.0       Tue 27.Feb.2024     Addedd pgdebug; EOF