Package: db-tqdm
db-tqdmA web based tqdm process bars.This TQDM based module is very useful for web, airflow, servers or similar environments which do not have console
to show the typical tqdm process bars.
In these environments are not useful to use the standard TQDM module because the access to the console or log files are
not easy, and if it writes into logs, this one are fulfilled with the progress bar information instead of the
real information log data.This module, by default, works the same way that the standard TQDM module. Only if the mode parameter is changed by
'mongo', or the environment variable 'TQDM_MODE' is defined with 'mongo', then the progress bar will store into a
database,instead of printing it in the standard error output.
This module also include the Uvicorn server in order to show this information with a browser like this:Contentdb-tqdm moduleInstall db-tqdmExecute the serverUse db-tqdm moduledb-tqdm environment variablesdb-tqdm dockerTo dodb-tqdm moduleYou can use db-tqdm as a usual Python module and integrate it into your Python projects. This section explains how toinstall, execute the server and test the db-tqdm module.Install db-tqdmTo install only need to do the following:pipinstalldb-tqdmIf you are using the pymongo database instead the standard TQDM module, you need to install:pipinstallmonutils~=0.1.3In the case, this module will be not installed, then, db-tqdm are going to work as a normal tqdm progress bar.If you are using also the web user interface, you need to install the following modules:pipinstallfastapi~=0.70.0Jinja2~=3.0.2uvicorn~=0.15.0importlib-resources~=5.1.3Execute the serverYou can execute the db-tqdm server in both, docker image or natively. If you choose the second option,
you can use the following command:usage:dbtqdm[-h][-HHOST][-PPORT][-tTYPE][--db_hostHOST][--db_portPORT][--root_pathPATH][-rNAME][-dNAME][-uUSER][-pPASS][--cert_key_fileFILE][--ca_fileFILE][--session_tokenSESSION][-iSECONDS][TITLE]Starttheservertoservethebarprogressdata.

positionalarguments:TITLEThewebpagetitle.Bydefault,"Process monitors".

optionalarguments:-h,--helpshowthishelpmessageandexit-HHOST,--hostHOSTTheserverhost.Bydefault,localhost.-PPORT,--portPORTTheserverport.Bydefault,5000.--root_pathPATHTheserverrootpath.Bydefault,"/".-tTYPE,--db_typeTYPEThedatabasehost.Bydefault,mongo.Availabledatabases:['mongo'].--db_hostHOSTThedatabasehost.Bydefault,localhost.--db_portPORTThedatabaseport.Bydefault,27017.-rNAME,--replicasetNAMEThereplicaset.Bydefault,none.-dNAME,--databaseNAMEThedatabasename.Bydefault,tqdm.-uUSER,--userUSERThedatabaseuser.Bydefault,none.-pPASS,--passwordPASSTheuserpassword.Bydefault,none.--cert_key_fileFILEThecertkeyfletoconnecttothedatabase.Bydefault,none.--ca_fileFILETheCAfiletoconnecttothedatabase.Bydefault,none.--session_tokenSESSIONThesessiontokentoconnecttothedatabase.Bydefault,none.-iSECONDS,--intervalSECONDSThedefaultwebrefreshinterval.Bydefault,5.For example:# Execute the server to connect with mongodb in localhost and port 27017# without authentication method and the database name "tqdm"dbtqdm# The same as above but defining database connection datadbtqdm--db_hostmymongoserver.com--db_port18-umymongouser-pmymongopass# Also, you can use environment variables instead command argumentsexportTQDM_DB_HOST=mymongoserver.comexportTQDM_DB_PORT=18exportTQDM_DB_USER=mymongouserexportTQDM_DB_PASSWORD=mymongopass

dbtqdmUse db-tqdm moduleThe use of db-tqdm is very similar to normal tqdm. For example, if you are using the progress bar into normal
console application or jupyter notebook, you only need to execute:fromdbtqdm.mongoimporttqdmfromtimeimportsleepforiintqdm(range(0,10),desc='Normal bar'):sleep(1)This code will show in the suitable output the following progress bar as usual:Normalbar:60%|██████|6/10[00:06<00:04,1.00s/it]You can change te normal tqdm progrsss bar by the database based one,
using the own db-tqdm parameters, specially mode and name, but not uniquely:fromdbtqdm.mongoimporttqdmfromtimeimportsleep# Create a tqdm based on MongoDB in localhost:27017 with the name "test1" and red colourfor_intqdm(range(0,5000),desc=f'Description of the progress bar 1',mode='mongo',name='test1',colour='red'):sleep(1)# The same as above but with the database connectionfor_intqdm(range(0,5000),desc=f'Description of the progress bar 1',mode='mongo',name='test1',colour='red',host='mymongoserver.com',port=18,user='mymongouser',password='mymongopassword'):sleep(1)This code will generate the following web-based bar progress:However, we strongly recommend to use environment variables instead of the function parameters because, this way,
your code will be the same if you use the tqdm locally or web-based. For example. this code will generate a normal
tqdm bar progress:fromdbtqdm.mongoimporttqdmfromtimeimportsleepfor_intqdm(range(0,5000),desc=f'Description of the progress bar 1',colour='red'):sleep(1)Nevertheless, if you define previously the follwoing environment variables, the same code will generate a web-based one:exportTQDM_MODE=mongoexportTQDM_NAME=test1But this approach has a problem in the case you want to use several progress bars at the same time,
because each progress bar is identified by its name. The solution is to change the previous code adding the suffix
parameter:fromdbtqdm.mongoimporttqdmfromtimeimportsleepfor_intqdm(range(0,10),desc=f'Description of the progress bar 1',colour='red',suffix='_main'):for_intqdm(range(0,20),desc=f'Description of the progress bar 1',colour='yellow',suffix='_secondary'):sleep(1)If you do not define the DB-TQDM environment variables, both progress bars will be normal ones,
ignoring the suffix parameter. However, if you define those variables, you will have two progress bars with
the name "<bar_name>_main" and "<bar_name>_secondary", respectively.Note:At the moment, do not put spaces neither, the bar name nor bar suffix.As we can see above, using environment variables instead the parameters allow you to use exactly the same code in
different platforms (for example, local workstation or in remote airflow server),
and depending on the defined environment variables can switch between different user interfaces.Note:At the moment, the argument 'db_type' is not supported, and it will be ignored.db-tqdm environment variablesYou can define environment variables to change the way tqdm progress bar appears
and the MondogDB connection data. All the environment variables you can define are:VariableDescriptionTQDM_MODEIf you want a normal or mongo bar progress. By default, normal.TQDM_NAMEThe bar progress name.TQDM_SUFFIXThe suffix of the bar progress name. This value is better to set by function parameter.TQDM_HOSTThe Web server host address. By default, localhost.TQDM_PORTThe Web server port. By default, 5000.TQDM_TYPEThe database type. In the current implementation only "mongo" is available. By default, mongo.TQDM_DB_HOSTThe database host address. By default, localhost.TQDM_DB_PORTThe database port. By default, 27017.TQDM_DB_REPLICASETThe mongo replicaset. By default, none.TQDM_DB_NAMEThe database name. By default, tqdm.TQDM_DB_USERThe database username. By default, none.TQDM_DB_PASSWORDThe user password. By default, none.TQDM_DB_CERT_KEY_FILEThe certificate key file. By default, none.TQDM_DB_CA_FILEThe CA file. By default, none.TQDM_INTERVALThe web refresh interval in seconds. By default, 5s.TQDM_TITLEThe web title. By default, 'Process monitors'.The TQDM_HOST and TQDM_PORT if only for the Web server.db-tqdm dockerYou can install the db-tqdm server by means a docker image.dockerpullialife/db-tqdm:1.1Remember you can define the [environment variables] to set the database connection.
For example, if you have the mongodb in your localhost, you can execute the following command to connect
the service in the db-tqdm docker with your mongodb:dockerrun-eTQDM_DB_USER=myusename-eTQDM_DB_PASSWORD=mypassword-eTQDM_DB_HOST=localhost\-eROOT_URL=http://localhost-eMONGO_URL=mongodb://localhost:27017--network="host"ialife/db-tqdmTo doMessage when there are any active process barShow a message when there are no active process bars.User and password for MongoDB connectionAdd the possibility to set the user and password for the MongoDB connection.Create a historical view of finished processesIn the collection_stats_are the historical information about the finished processes.
It could be interesting to use them in a view with this information. The idea is to use the home page, below
the progress bars in the main page, to add a section with a paged table with the finished processes ordered descending
by start time.Generalize to be able to use other database managersThe core of the module is ready to use another database managers creating classes inherited fromDatabaseTqdm.
However, the server process is not prepared to use other database managers, only MongoDB.
It could be interesting to refactor the code to add this functionality.Use Redis or KafkaThis can be adapted (but without historical information) to be used withRedisorKafkaplatforms.Pass collections to registersI made the design decision to create each process bar in a MongoDB collection instead of create a register for each
process. I do not sure because I made that decision, and I think it would be better to replace the different collections
by registers of a unique MongoDB collection.