Package: nimble-tk
nimble toolkitA collection of simple but powerful utilities for ML and Data Analytics workflows.Installationpip install nimble-tkExamplesLoggingThe below set of cells give some utility wrappers on python's logging functionality.
The following line will create a log file at/tmp/try_nimble.logAs the logs get written, the log file will go up to max50 MBin size by default and then will get rolled over.At most,10such files are maintained before removing the earliest file.Logs will also be written to the console (in this case the notebook console)ntk.init_file_logger(log_file_path='/tmp/try_nimble.log',console_log_on=True)Output:2023-12-09 11:10:43.270595 :: MainThread ::  Log file init at /tmp/try_nimble.logExample of an info log:ntk.log_info("some log message")Output:2023-12-09 11:10:43.289434 :: MainThread ::  some log messageExample of an error log:Catching an exception and logging the stack trace:try:tmp=1/0except:ntk.log_traceback("Error while running operation")Output:2023-12-09 11:10:43.342127 :: MainThread ::  Error while running operation :: ZeroDivisionError: division by zero ::
 Traceback (most recent call last):
  File "/tmp/ipykernel_3709/4084614229.py", line 2, in <module>
    tmp = 1/0
ZeroDivisionError: division by zeroAll logs are also written to the log file:! tail -10 /tmp/try_nimble.logOutput:2023-12-09 11:10:43,268 MainThread logger.py:88:  Log file init at /tmp/try_nimble.log
2023-12-09 11:10:43,284 MainThread logger.py:88:  some log message
2023-12-09 11:10:43,311 MainThread logger.py:88:  some error message
2023-12-09 11:10:43,334 MainThread logger.py:88:  Error while running operation :: ZeroDivisionError: division by zero ::
 Traceback (most recent call last):
  File "/tmp/ipykernel_3709/4084614229.py", line 2, in <module>
    tmp = 1/0
ZeroDivisionError: division by zeroLogs can also be directly written to the filewithout logging on the console. This helps in cases where we do not want to flood the console with a lot of logs.ntk.log_info_file("some log message")ntk.log_error_file("some error message")try:tmp=1/0except:ntk.log_traceback_file("Error while running operation")!tail-7/tmp/try_nimble.logOutput:2023-12-09 11:10:43,501 MainThread logger.py:88:  some log message
2023-12-09 11:10:43,505 MainThread logger.py:88:  some error message
2023-12-09 11:10:43,510 MainThread logger.py:88:   :: ZeroDivisionError: division by zero ::
 Traceback (most recent call last):
  File "/tmp/ipykernel_3709/3515687902.py", line 5, in <module>
    tmp = 1/0
ZeroDivisionError: division by zeroConcurrent ProcessingThentk.run_concurrently(...)method provides a wrapper around python's multi-processing and multi-threading functionality.Set up a list of functions to execute:defanalytic_function(customer_id):ntk.log_info_file(f"Running analytic_function for customer_id:{customer_id}")ifcustomer_id==0:# raising an error here to demonstrate error handlingraiseValueError(f"Invalid customer id{customer_id}")# - Query the DB# - Do feature engineering# - Run other analytics# - Write the result data to disk or return a DFreturnpd.DataFrame({'CUSTOMER_ID':[customer_id]*5,'OTHER_DATA':np.random.randint(0,9,5)})functions=[]forcustomer_idinrange(10):functions.append((analytic_function,{'customer_id':customer_id}))Run the functions concurrently with the following lines of code:results,errors=ntk.run_concurrently(functions,max_workers=ntk.get_num_cpus(),fork=True)df_result=pd.concat([result[1]forresultinresults])df_result.head(index=False)fork=Truewill launch multiple processes and is recommended to be used for CPU-bound tasks.fork=Falsewill launch multiple threads, should to be used for IO-bound tasks.Output:2023-12-09 11:10:46.767042 :: MainThread ::  Ran: 10 functions - Successfull: 9, Failed: 1CUSTOMER_IDOTHER_DATA1310131113......