Package: xetrack
Xetrackxetrack is a lightweight package to track experiments benchmarks, and monitor stractured data usingduckdbandsqlite.It is focuesed on simplicity and flexability.You create a "Tracker", and let it track data. You can retrive it later as pandas or connect to it as a database.Each instance of the tracker has a "track_id" which is a unique identifier for a single run.FeaturesSimpleEmbeddedFastPandas-likeSQL-likeObject store with deduplicationCLI for basic functionsMultiprocessing reads and writesLoguru logs integrationExperiment trackingModel monitoringInstallationpipinstallxetrackQuickstartfromxetrackimportTrackertracker=Tracker('database.db',params={'model':'resnet18'})tracker.log({"accuracy":0.9,"loss":0.1,"epoch":1})# All you really needtracker.latest{'accuracy':0.9,'loss':0.1,'epoch':1,'model':'resnet18','timestamp':'18-08-2023 11:02:35.162360','track_id':'cd8afc54-5992-4828-893d-a4cada28dba5'}tracker.to_df(all=True)# Retrive all the runs as dataframetimestamptrack_idmodellossepochaccuracy026-09-202312:17:00.342814398c985a-dc15-42da-88aa-6ac6cbf55794resnet180.110.9Paramsare values which are added to every future row:$tracker.set_params({'model':'resnet18','dataset':'cifar10'})$tracker.log({"accuracy":0.9,"loss":0.1,"epoch":2}){'accuracy':0.9,'loss':0.1,'epoch':2,'model':'resnet18','dataset':'cifar10','timestamp':'26-09-2023 12:18:40.151756','track_id':'398c985a-dc15-42da-88aa-6ac6cbf55794'}You can also set a value to an entire run withset_value("back in time"):tracker.set_value('test_accuracy',0.9)# Only known at the end of the experimenttracker.to_df()timestamptrack_idmodellossepochaccuracydatasettest_accuracy026-09-202312:17:00.342814398c985a-dc15-42da-88aa-6ac6cbf55794resnet180.110.9NaN0.9226-09-202312:18:40.151756398c985a-dc15-42da-88aa-6ac6cbf55794resnet180.120.9cifar100.9Track functionsYou can track any function.The return value is logged before returnedtracker=Tracker('database.db',log_system_params=True,log_network_params=True,measurement_interval=0.1)image=tracker.track(read_image,*args,**kwargs)tracker.latest{'result':571084,'name':'read_image','time':0.30797290802001953,'error':'','disk_percent':0.6,'p_memory_percent':0.496507,'cpu':0.0,'memory_percent':32.874608,'bytes_sent':0.0078125,'bytes_recv':0.583984375}Or with a wrapper:@tracker.wrap(params={'name':'foofoo'})deffoo(a:int,b:str):returna+len(b)result=foo(1,'hello')tracker.latest{'function_name':'foo','args':"[1, 'hello']",'kwargs':'{}','error':'','function_time':4.0531158447265625e-06,'function_result':6,'name':'foofoo','timestamp':'26-09-2023 12:21:02.200245','track_id':'398c985a-dc15-42da-88aa-6ac6cbf55794'}Track assets (Oriented for ML models)When you attempt to track a non primitive value which is not a list or a dict - xetrack saves it as assets with deduplication and log the object hash:Tips: If you plan to log the same object many times over, after the first time you log it, just insert the hash instead for future values to save time on encoding and hashing.$tracker=Tracker('database.db',params={'model':'logistic regression'})$lr=Logisticregression().fit(X_train,y_train)$tracker.log({'accuracy':float(lr.score(X_test,y_test)),'lr':lr}){'accuracy':0.9777777777777777,'lr':'53425a65a40a49f4',# <-- this is the model hash'dataset':'iris','model':'logistic regression','timestamp':'2023-12-27 12:21:00.727834','track_id':'wisteria-turkey-4392'}$model=tracker.get('53425a65a40a49f4')# retrive an object$model.score(X_test,y_test)0.9777777777777777You can retrive the model in CLI if you need only the model in production and mind carring the rest of the file# bashxtgetdatabase.db53425a65a40a49f4model.cloudpickle# pythonimportcloudpicklewithopen("model.cloudpickle",'rb')asf:model=cloudpickle.loads(f.read())# LogisticRegression()Tips and tricksTracker(Tracker.IN_MEMORY)Let you run only in memoryPandas-likeprint(tracker)_idtrack_iddatebaaccuracy048154ec7-1fe4-4896-ac66-89db54ddd12afd0bfe4f-7257-4ec3-8c6f-91fe8ae67d2016-08-202300:21:462.01.0NaN18a43000a-03a4-4822-98f8-4df671c2d410fd0bfe4f-7257-4ec3-8c6f-91fe8ae67d2016-08-202300:24:21NaNNaN1.0tracker['accuracy']# get accuracy columntracker.to_df()# get pandas dataframe of current runSQL-likeYou can filter the data using SQL-like syntax usingduckdb:The sqlite database is attached asdband the table isevents. Assts are in theassetstable.Pythontracker.conn.execute(f"SELECT * FROM db.events WHERE accuracy > 0.8").fetchall()Duckdb CLIduckdb
DATTACH'database.db'ASdb(TYPEsqlite);DSELECT*FROMdb.events;┌────────────────────────────┬──────────────────┬──────────┬───────┬──────────┬────────┐
│timestamp│track_id│model│epoch│accuracy│loss│
│varchar│varchar│varchar│int64│double│double│
├────────────────────────────┼──────────────────┼──────────┼───────┼──────────┼────────┤
│2023-12-2711:25:59.244003│fierce-pudu-1649│resnet18│1│0.9│0.1│
└────────────────────────────┴──────────────────┴──────────┴───────┴──────────┴────────┘Logger integrationThis is very useful in an environment where you can use normal logs, and don't want to manage a separate logger or file.On great use-case ismodel monitoring.logs_stdout=trueprint to stdout every tracked eventlogs_path='logs'writes logs to a file$Tracker(db=Tracker.IN_MEMORY,logs_path='logs',logs_stdout=True).log({"accuracy":0.9})2023-12-1421:46:55.290|TRACKING|xetrack.logging:log:69!📁!{"a":1,"b":2,"timestamp":"2023-12-14 21:46:55.290098","track_id":"marvellous-stork-4885"}$Reader.read_logs(path='logs')accuracytimestamptrack_id00.92023-12-1421:47:48.375258unnatural-polecat-1380AnalysisTo get the data of all runs in the database for analysis:Use this for further analysis and plotting.This works even while a another process is writing to the database.fromxetrackimportReaderdf=Reader('database.db').to_df()Model MonitoringHere is how we can save logs on any server and monitor them with xetrack:We want to print logs to a file orstdoutto be captured normally.We save memory by not inserting the data to the database (even though it's fine).
Later we can read the logs and do fancy visualisation, online/offline analysis, build dashboards etc.tracker=Tracker(db=Tracker.SKIP_INSERT,logs_path='logs',logs_stdout=True)tracker.logger.monitor("<dict or pandas DataFrame>")# -> write to logs in a structured way, consistent by schema, no database file neededdf=Reader.read_logs(path='logs')"""Run drift analysis and outlier detection on your logs:"""ML trackingtracker.logger.experiemnt(<modelevaluationandparams>)# -> prettily write to logsdf=Reader.read_logs(path='logs')"""Run fancy visualisation, online/offline analysis, build dashboards etc."""CLIFor basic and repetative needs.$xtheaddatabase.db--n=2||timestamp|track_id|model|accuracy|data|params||---:|:---------------------------|:-------------------------|:---------|-----------:|:-------|:-----------------||0|2023-12-2711:36:45.859668|crouching-groundhog-5046|xgboost|0.9|mnist|1b5b2294fc521d12||1|2023-12-2711:36:45.863888|crouching-groundhog-5046|xgboost|0.9|mnist|1b5b2294fc521d12|...


$xttaildatabase.db--n=1||timestamp|track_id|model|accuracy|data|params||---:|:---------------------------|:----------------|:---------|-----------:|:-------|:-----------------||0|2023-12-2711:37:30.627189|ebony-loon-6720|lightgbm|0.9|mnist|1b5b2294fc521d12|$xetsetaccuracy0.8--where-keyparams--where-value1b5b2294fc521d12--track-idebony-loon-6720

$xtdeletedatabase.dbebony-loon-6720# delete experiments wiht a given track_id# run any other SQL in a oneliner$xtsqldatabase.db"SELECT * FROM db.events;"# retrive a model which was saved into a files as files using cloudpickle$xtgetdatabase.dbmodelhashoutput# If you have two databases, and you want to merge one to the other$xtcopysource.dbtarget.db