Package: cindi
Cacheable Implementation Neutral Data IntermediateCINDIis aMeta Database Management Systemwhich provides a simple way for
front-end applications to performCRUDoperations  with various back-end
stores.CINDIis written inPython 3, and is available in thePyPI Indexas aPIPpackage.CINDIis a service to translateINDIlanguage statements to eitherSQL,MongoDB, orRedis.CINDIalso caches read queries in memory, so that clients asking for the same thing repetitively do not actually get to hit the backing-stores.CINDIis intended to be used during the early rapid-prototyping / idea phase, so that developers may focus on the layout, functionality, and usability of a front-end application.Think ofCINDIas a temporary substitute, until a proper back-end infrastructure is built.  This way, the initial pioneers of a project need only to focus their energy on the front-end(s).The idea is to forgo the traditional map of API endpoints usually provided by a back-end.  Instead, front-end code will submitINDIstatements directly to a single endpoint, and receiveJSONresponses accordingly. The result is, most application logic is shifted into the front-end code, which simplifies the rapid prototyping of applications.Later, when a minimal-viable-product had been demonstrated to stakeholders, and more developers can be assigned to the project,CINDIwould then be replaced with a traditional API back-end.Users may configureCINDIto use one, some, or all of the currently supported storage systems:PostgreSQLMySQLSQLite3MongoDBRedisAn easyImplementation Neutral Data Intermediatelanguage is provided in order to abstract-away the underlying chosen store(s).  TheINDIlanguage is similar toSQL, but re-worded to enforce theCRUDmentality.Brief INDI Language TutorialTheINDIlanguage uses four majorCRUDkeywords:CREATE,READ,UPDATE, andDELETE.CINDIwillcacheREADoperations in-memory, until it detects those result-sets are stale (due to anotherCREATE,UPDATE, orDELETEaffecting the data).In order to demonstrate theINDIlanguage, consider the followingSQLite3schema:CREATE TABLE nonsense (
 	id INTEGER NOT NULL PRIMARY KEY,
	nonsense_a TEXT,
	nonsense_b TEXT,
	nonsense_c TEXT
	);The aboveSQLstatement would be prepared (manually, by the user) forCINDIbefore starting. Then, after loadingCINDI, the fourCRUDoperations may be performed by submittingINDIstatements to the system.For example, here is anINDICREATEstatement:CREATE IN nonsense FIELDS (nonsense_a, nonsense_b, nonsense_c) VALUES ("hekkin big", "skare", "today and now")There would be no 'result' to the above query, other than the underlying store(s) being modified to reflect the insertion of new data.  Client-side, this is what would be received as a response:[[]]Next example, here is anINDIREADstatement:READ IN nonsense id 1 FIELDS nonsense_aTheREADhas a result, it would be a JSON array looking like this:[[['hekkin big']]]Now for a more comprehensiveREAD:READ IN nonsense ALL RECORDS FIELDS (id, nonsense_a, nonsense_b, nonsense_c)This retrieves all records from the nonsense table, including all columns:[[[1, 'hekkin big', 'skare', 'today and now']]]It appears that 'skare' is spelled wrong, so toUPDATEthat:UPDATE IN nonsense nonsense_b skare FIELDS (nonsense_b) VALUES (scare)The literal result is an empty set:[[]]Check to see just that column had been updated:READ IN nonsense id 1 FIELDS nonsense_bResult:[[['scare']]]Finally, toDELETEthe record:DELETE IN nonsense id 1A subsequentREADwould return an empty set.  Note theDELETEmay have queried by any field, not just the 'id'!VALUESquotes are optional, unless the string to be stored contains spaces.See thedoc/demo.txtfile for the verbose output from the above demo.Thedocfolder also contains schema examples for the other supported systems.Why ?The above demonstration may seem trivial, however, consider thatCINDIwill support managing theseINDIstatements even if the backing-store is aNoSQLsolution, such asMongoDBorRedis.  The results are consistent, and if multiple stores are enabled, then any cache-missedREADwill query all the stores, and compare the results to ensure consistency across the storage mechanisms.If clients are submitting the sameINDIqueries repetitively, then the result from the first time is served from an in-memoryPython dictionary.  That cached query is discarded when an affectingCREATE,UPDATE, orDELETEoccurs.In the case of theNoSQLsolutions,CINDImanages the concept of tables, columns, and primary keys.  In particular, theMongoDBsetup requires only creating a database, and theRedissetup requires nothing!CINDI will not automatically initialize SQL / Mongo schemas, this must be done manually prior to runtime. That is why the author has provided a coupleDockerdemos, read on!Checkcindi_tests.pyfor moreINDIlanguage examples.Installation, Easiest:cindi-liteDocker Compose PackPlease go hereand follow the instructions in order to haveDocker Composeorchestrate the setup ofultasun/cindi-lite, which will consume the least resources. Along the way, you'll learn everything there is to know about configuring theCINDIruntime for aminimalistSQLite3-only backing-store.Installation, Easier:cindi-plusDocker Compose PackPlease go hereand follow the instructions in order to haveDocker Composeorchestrate the complete setup and demonstration ofultasun/cindi, which will, by default, consume significantly more resources thanultasun/cindi-lite. Along the way, you'll learn everything there is to know about configuring theCINDIruntime for afull installation with all five supported backing-stores running simultaneouslyInstallation, Easy:cindi-liteDocker image (no orchestration)ADockerimageultasun/cindi-liteis available onDocker Hub, it will use aSQLite3setup, and provide a similar experience toInstalling & Basic Setup on SQLite3.After start-up, developmentFlaskserver willnothaveSSLenabled.Install the latest version ofDockeron your system.Open a terminal, and run the following command for a disposable demo:$ docker run -dp 36963:36963 --rm ultasun/cindi-liteOr, if you'd like to persist logs and data withnamed-mounts:$ docker run -dp 36963:36963 -v cindi-lite-data:/app/data -v cindi-lite-logs:/app/logs ultasun/cindi-liteFeel free to usebind-mountsfor easier access to the produced data.UseDocker Desktopto open a terminal into the running container, or otherwisedocker exec -itinto the container, and initialize theEXAMPLEdata:$ python>>> import cindi>>> cindi.quick_unit_tests()Verify the return value isTrue, which means all 17 tests passed!UsePostman(or something) toPOSTsomeINDIstatements tohttp://localhost:36963/evaluate, such as:READ IN nonsense ALL RECORDS FIELDS (id, nonsense_a, nonsense_b, nonsense_c)Back in thepythonconsole, the full list ofEXAMPLESmay be viewed:>>> cindi.print_2d_list(cindi.EXAMPLE_LIST)These are theEXAMPLESwhich were ran duringcindi.quick_unit_tests()There is no harm in running theDDLorDMLstatements again!Hard: Manual Installation & Basic Setup on SQLite3Installing without DockerDocker is not necessary to utilizeCINDI.  This installation demo will show manual installation from PyPI, and configuration using onlySQLite3, to keep it brief.  Please follow the detailed tutorial oncindi-plusto learn about configuring other backing-store systems.pip install cindiIn your current working directory, create a folder calledconfig:$ mkdir configIn theconfigfolder, create emptystores.txtandtables.txtfiles.$ cd config && touch stores.txt && touch tables.txt && cd ..Openstores.txtwith a text editor -- the file must contain a python dictionary, here is the basic example using onlySQLite3:{'sqlite3': {'db': 'db0', 'sqlite3_file_prefix': ''}}Opentables.txtwith a text editor -- the file must contain a python list (or tuple), here is the basic example:['nonsense']To open theSQLite3database for intialization, in your terminal, run:$ sqlite3 db0.dbYou may need to installSQLite3if the command is not found.In theSQLite3console, initialize the schema:sqlite> CREATE TABLE nonsense (id INTEGER NOT NULL PRIMARY KEY, nonsense_a TEXT, nonsense_b TEXT, nonsense_c TEXT);Save and quit:sqlite> .quitIn your terminal, run:$ python3Finally, try it out!  Populate the store(s) withEXAMPLEdata:>>> import cindi>>> cindi.quick_unit_tests()The above function returnsTrueif all tests succeed.>>> cindi.quick_cindi(cindi.EXAMPLE5)The above function returns a multi-dimensional list of values.LoggingSinceCINDIis in the alpha development stage, everyDMLstatement is logged in thelogs/directory. Whatever data you're submitting, a copy will be saved in that directory.This will consume disk space.This will expose sensitive data.There is oneINDIDMLstatement per file. Each filename ends with a nano-seconds-since-epoch timestamp. A filename prefixed withindiis aDMLexpression. At the time of this writing, there are no other files written to thelogs/directory.There is no simple way to disable this disk logging without modifying the source code. Just re-definedef fprint(w, x='', y='', z=''):to do nothing if you must.CINDIis in the alpha development stage, and it is designed to assist in developing apps which are also in the alpha development stage.SSLon theFlaskdevelopment server is not even enabled, and the development server is 'not for production use' anyway.TroubleshootingCINDIis in the alpha development stage. If you've encountered a problem, then you've probably found a bug in the translation routines, or your backing-stores are not configured correctly.CINDIwill generally exit thePython3process on most errors ('fail fast'). Thelogs/directory is ordred by nanoseconds-since-epoch timestamps, so it should be easy to work-backwards to locate theINDIDMLstatement which was translated inconsistently across multiple backing-stores. The author can't forsee any other reason forCINDIto crash, besides a backing-store connectivity issue.When thePython3process exits, variousexit codesare used. This helps frequent users/developers ofCINDIquickly identify from theDockerconsole (or elsewhere) why anexithad occurred.Here are the exit numbers and error messages which might be produced byCINDI:117:Corrupted store, please file a bug. Check the CINDI README.The chain ofINDIDMLstatements which were submitted to CINDI for execution over time, had managed to cause differentinconsistentresults to appear in the various backing stores.This will crash the system because, everyDQLstatement executed verifies consistency across all conneected backing stores ('fail fast'). SoCINDIhad a translation error during an earlierDMLstatement.Remedy Options:First,please submit a bug with the full contents oflogs/, and santize your data first if possible. This error is really not the fault of the user (you). TheVALUESmay not be so important, so a simple script to truncate after theVALUESin each file within thelogs/directory if you can't or won't share theVALUES.However, theVALUESwould be appreciated, becausesome parts of the translation code might do the wrong action if a value has too many single-or-double-quotes, it's a your-milage-may-vary scenario (at this time).Regular expressions might be used in the future to mitigate some of the forseen issues regarding single-and-double quotesBecause a proper recursive solution might be needed.Second,either:Manually adjust the backing-stores, using their native tools/clients, until the data is consistent.Success will be obvious whenDQLstatements regarding the 'offending table' no longer causeCINDIto exit.Wipe all backing stores, and reload all theINDIstatements from new.5:fprint failed to print to file: ...The log-to-disk mechanismfprintfailed to write a file.Remedy Options:Disk space full?Lost directory permissions during runtime?52:!! ERROR: The quantity of fields detected is not euqal to the number of values detected...The submittedINDIDMLstatement does not have the same quantity ofFIELDSandVALUES,or,The submittedINDIDMLstatement was unable to be parsed properly by the translation routine.Please submit a bug containing theINDIDMLstatement which caused this.Submitting the fulllogs/should not be necessary, because this error would be isolated to parsing a specificINDIDMLstatement before any writing to a backing-store occured.Remedy Options:Check theINDIsyntaxManipulate the single-quotes or double-quotes in theVALUESof the offending statementMany nested single-quotes or double-quotes might confuse the translation routine.Remove the quotes, and please submit a bug.Regular expressions will probably be used in the future to avoid problems like this.29:Redis SET failed.While performing aRedisSET, it failed.Remedy Options:WasCINDIdisconnected fromRedisin the middle of the transaction?DidCINDIlose write access to theRedisdatabase for some reason?Please submit a bug in any event! Thank you.2: Relating to a configuration 'File Not Found':config/stores.txt not found, please check the CINDI README.config/tables.txt not found, please check the CINDI README.Remedy Options:Theconfig/directory must be in the same directory.as where thePython3process was launched. This is an elementary user mistake. Please review this README file carefully, or check one of the publishedDockerimages for a proper example of how thePython3process needs to be launched.Theconfig/stores.txtorconfig/tables.txtfiles are missing or not named properly.61: Relating to a configuration file 'Invalid Syntax':config/stores.txt not a dictionary, check the CINDI README.Remedy Optionconfig/stores.txtmust be aPython3dictionary. Check the syntax of what is in the file, and try again.config/tables.txt must be a list or tuple, check CINDI README.Remedy Optionconfig/tables.txtmust be aPython3listortuple. Check the syntax of what is in the file, and try again.No stores defined in config/stores.txt! Check CINDI README.Remedy OptionTheconfig/stores.txtfile has an empty dictionary in it.Review this README file carefully, you must define at least one backing-store forCINDIto use!49:Failed to initialize XXXXX connection.Python3support for your chosen backing-store mechanism is not installed,or,The connection/credential to the backing store is bad,or, the database you had specified to connect to inconfig/stores.txtis not available.Remedy OptionVerify thePython3client library for the specified backing-store is installed.Verify the credentials to the backing-store are valid and specified correctly inconfig/stores.txtVerify the database configured inconfig/stores.txtexists, permissions are good, and so on....sqlite3 connection, file not found?Perhaps the database file was deleted, or access was lost.CINDIwill not automatically re-create theSQLite3database in this event.Either reconstruct theSQLite3database manually,or,Wipe out all backing-stores, and start again from new.Submit a bug if continuing to have difficulty connecting to a store, perhaps something changed in an underlying connectivity library!131:High level error: ...Some error, likely in a driver, occured.Remedy Option: If the solution is not obvious, then you should submit a bug.Especially if the problem is anOperationalError, complaining theSQLsyntax is invalid.22:Low level error: ...Some error, unforseen by the author, occured.Remedy Option: If the solution is not obvious, then you should submit a bug.Cache Duality CaveatIf you have multipleCINDIinstances running against the same backing-stores, then a cache in one instance will not have a way to know if aDMLCREATE/UPDATE/DELETEoccured in another instance.Please do not allow multiple clones ofCINDIto interact with the same backing-stores concurrently.A simple solution to this in a future release will be to have a/cache-clearendpoint inFlask,orA file on the file system is monitored, if it is changed, then other instances know to clear the cache.See the next section regardingSupport!SupportThe author is available for questions, comments and criticism in #cindi onLibera.Chat.CreditsPlease read theLICENSE.  This software was written by a single individual.  This is alpha-quality software, you are likely to find bugs!  Please submit feedback!Thank you for evaluatingCINDI!