Package: ml-logger
ML-Logger, A Simple and Scalable Logging Utility With a Beautiful Visualization Dashboard That Is Super FastML-Logger makes it easy to:save data locally and remotely, asbinary, in a transparentpicklefile, with the same API and zero
configuration.write from 500+ worker containers to a single instrumentation servervisualizematplotlib.pyplotfigures from a remote server locally withlogger.savefig('my_figure.png')And ml-logger does all of these withminimal configuration— you can use the same logging
code both locally and remotely with no code change.ML-logger is highly performant -- the remote writes are asynchronous. For this reason it doesn't slow down your training
even with 100+ metric keys.Why did we built this, you might ask? Because we want to make it easy for people in ML to
use the same logging code in all of they projects, so that it is easy to get started with
someone else's baseline.Getting Started!Toinstallml_logger, do:pipinstallml-loggerThe landscape of python modules is a lot messier than that of javascript. The most up-to-dategraphenerequires the following versions:yes | pip install graphene==2.1.3
yes | pip install graphql-core==2.1
yes | pip install graphql-relay==0.4.5
yes | pip install graphql-server-core==1.1.1Add the following to your~/.zshrcexportML_LOGGER_ROOT=<your-logging-server-address>exportML_LOGGER_USER=<your-handle># make sure it is the same on the cluster.Now you can rock!fromml_loggerimportloggerprint(logger)# ~> logging data to /tmp/ml-logger-debugLog key/value pairs, and metrics:foriinrange(1):logger.log(metrics={'some_val/smooth':10,'status':f"step ({i})"},reward=20,timestep=i)### flush the data, otherwise the value would be overwritten with new values in the next iteration.logger.flush()# outputs ~># ╒════════════════════╤════════════════════════════╕# │       reward       │             20             │# ├────────────────────┼────────────────────────────┤...Simple Logging ExampleIn your project files, do:fromparams_protoimportParamsProto,Proto,FlagclassArgs(ParamsProto):""" CLI Args for the programTry:python3 example.py --helpAnd it should print out the help strings"""seed=Proto(1,help="random seed")D_lr=5e-4G_lr=1e-4Q_lr=1e-4T_lr=1e-4plot_interval=10verbose=Flag("the verbose flag")if__name__=='__main__':importscipyimportnumpyasnpfromml_loggerimportlogger,LOGGER_USER# Put in your ~/.bashrc# export ML_LOGGER_ROOT = "http://<your-logging-server>:8081"# export ML_LOGGER_USER = $USERlogger.configure(prefix=f"{LOGGER_USER}/scratch/your-experiment-prefix!")logger.log_params(Args=vars(Args))logger.upload_file(__file__)forepochinrange(10):logger.log(step=epoch,D_loss=0.2,G_loss=0.1,mutual_information=0.01)logger.log_key_value(epoch,'some string key',0.0012)# when the step index updates, logger flushes all of the key-value pairs to file system/logging serverlogger.flush()# Imagesface=scipy.misc.face()face_bw=scipy.misc.face(gray=True)logger.save_image(face,"figures/face_rgb.png")logger.save_image(face_bw,"figures/face_bw.png")image_bw=np.zeros((64,64,1))image_bw_2=scipy.misc.face(gray=True)[::4,::4]logger.save_video([face]*5,"videos/face.mp4")And it should print out✓createdanewloggingclient
Dashboard:http://app.dash.ml/user-name/your-experiment-prefix!
Log_directory:http://<your-server-ip>:8080
══════════════════════════════════════════Args────────────────────┬─────────────────────seed│1D_lr│0.0005G_lr│0.0001Q_lr│0.0001T_lr│0.0001plot_interval│10════════════════════╧═════════════════════
╒════════════════════╤════════════════════╕
│step│9│
├────────────────────┼────────────────────┤
│Dloss│0.200│
├────────────────────┼────────────────────┤
│Gloss│0.100│
├────────────────────┼────────────────────┤
│mutualinformation│0.010│
╘════════════════════╧════════════════════╛


Processfinishedwithexitcode0To View the ResultsNote, thejwtbased access control has not been implemented yet. So you should leave the token field empty.Logging to a ServerSkip this if you just want to log locally.When training in parallel, you want to kickstart an logging server (Instrument Server). To do so, run:python-mml_logger.serverUse ssh tunnel if you are running on a managed cluster (with SLURM for instance).Important: to set allow remote logging, you need to pass in--host=0.0.0.0so that the server accepts non-localhost connections.python-mml_logger.server--host=0.0.0.0Asynchronously log the summary of LOTs of training metricsA common scenario is you only want to upload averaged statistics of your metrics. A pattern
that @jachiam uses is the following:store_metrics(),peak_stored_metrics(), andlog_metrics_summary()# You log lots of metrics during training.foriinrange(100):logger.store_metrics(metrics={'some_val/smooth':10},some=20,timestep=i)# you can peak what's inside the cache and print out a table like this:logger.peek_stored_metrics(len=4)# outputs ~>#      some      |   timestep    |some_val/smooth# ━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━#       20       |       0       |      10#       20       |       1       |      10#       20       |       2       |      10#       20       |       3       |      10# The metrics are stored in-memory. Now we need to actually log the summaries:logger.log_metrics_summary(silent=True)# outputs ~> . (data is now logged to the server)Table of Contentsloggingmatplotlib.pyplotfigures on an headless server[documentation under construction]How to DevelopFirst clone repo, install dev dependencies, and install the module under evaluation mode.gitclonehttps://github.com/episodeyang/ml_logger.gitcdml_logger&&cdml_logger&&pipinstall-rrequirements-dev.txt
pipinstall-e.Testing local-mode (without a server)You should be inside ml_logger/ml_logger folderpwd# ~> ml_logger/ml_loggermaketestTesting with a server (You need to do both for an PR)To test with a live server, first run (in a separate console)python -m ml_logger.server --data-dir /tmp/ml-logger-debugor do:makestart-test-serverThen run this test script with the option:python-mpytesttests--capture=no--data-dirhttp://0.0.0.0:8081or domaketest-with-serverYour PR should have both of these two tests working. ToDo: add CI to this repo.To PublishYou needtwine,rst-lintetc, which are included in therequirements-dev.txtfile.Logging Matplotlib pyplotsConfiguring The Experiment Folderfromml_loggerimportlogger,ML_Loggerfromdatetimeimportdatetimenow=datetime.now()logger.configure("/tmp/ml-logger-demo","deep_Q_learning",f"{now:%Y%m%d-%H%M%S}")This is a singleton pattern similar tomatplotlib.pyplot. However, you could also use the logger constructorlogger=ML_Logger(root_dir="/tmp/ml-logger-demo",prefix=f"deep_Q_learning/{now:%Y%m%d-%H%M%S}")Logging Text, and Metricslogger.log({"some_var/smooth":10},some=Color(0.85,'yellow',percent),step=3)colored output: (where the values are yellow)╒════════════════════╤════════════════════╕
│  some var/smooth   │         10         │
├────────────────────┼────────────────────┤
│        some        │       85.0%        │
╘════════════════════╧════════════════════╛Logging Matplotlib FiguresWe have optimized ML-Logger, so it supports any format thatpyplotsupports. To save a figure locally or remotely,importmatplotlib.pyplotaspltimportnumpyasnpxs=np.linspace(-5,5)plt.plot(xs,np.cos(xs),label='Cosine Func')logger.savefig('cosine_function.pdf')Logging VideosIt is especially hard to visualize RL training sessions on a remote computer. With ML-Logger this is easy, and
super fast. We optimized the serialization and transport process, so that a large stack of video tensor gets
first compressed byffmepgbefore getting sent over the wire.The compression rate (and speed boost) can be 2000:1.importnumpyasnpdefim(x,y):canvas=np.zeros((200,200))foriinrange(200):forjinrange(200):ifx-5<i<x+5andy-5<j<y+5:canvas[i,j]=1returncanvasframes=[im(100+i,80)foriinrange(20)]logger.log_video(frames,"test_video.mp4")Saving PyTorch ModulesPyTorch has a very nice module saving and loading API that has inspired the one inKeras. We make it easy to save
this state dictionary (state_dict) to a server, and load it. This way you can load from 100+ of your previous
experiments, without having to download those weights to your code repository.# save a modulelogger.save_module(cnn,"FastCNN.pkl")# load a modulelogger.load_module(cnn,f"FastCNN.pkl")Saving Tensorflow ModelsThe format tensorflow uses to save the models is opaque. I prefer to save model weights inpickleas a dictionary.
This way the weight files are transparent. ML_Logger offers easy helper functions to save and load from checkpoints
saved in this format:## To save checkpointfromml_loggerimportloggerimporttensorflowastflogger.configure(log_directory="/tmp/ml-logger-demos")x=tf.get_variable('x',shape=[],initializer=tf.constant_initializer(0.0))y=tf.get_variable('y',shape=[],initializer=tf.constant_initializer(10.0))c=tf.Variable(1000)sess=tf.InteractiveSession()sess.run(tf.global_variables_initializer())trainables=tf.trainable_variables()logger.save_variables(trainables,path="variables.pkl",namespace="checkpoints")which creates a filecheckpoints/variables.pklunder/tmp/ml-logger-demos.VisualizationAn idea visualization dashboard would beFast, instantaneous.On an AWS headless server? View the plots as if they are on your local computer.Searchable, performantly.So that you don't have to remember where an experiment is from last week.Answer Questions, from 100+ Experiments.We make available Google's internal hyperparameter visualization tool,
on your own computer.Searching for Hyper ParametersExperiments are identified by themetrics.pklfile. You can log multiple times to the samemetrics.pklfile,
and the later parameter values overwrites earlier ones with the same key. We enforce namespace in this file, so each
key/value argument you pass into thelogger.log_parametersfunction call has to be a dictionary.Args=dict(learning_rate=10,hidden_size=200)logger.log_parameters(Args=Args)How to launch the Vis AppThis requires node.js and yarn dev environmentat the moment. We will streamline this process without these
requirements soon.download this repositorygo toml-vis-appfolderInstall the dev dependenciesinstall node:Installationinstall yarn:Installationinstall the dependencies of this visualization app:yarn installin that folder, runyarn.The IP address of the server is currently hard codedhere.To use this with your own
instrumentation server, over-write this line. I'm planning on making this configuration more accessible.Full Logging APIfromml_loggerimportlogger,Color,percentlogger.log_params(G=dict(some_config="hey"))logger.log(some=Color(0.1,'yellow'),step=0)logger.log(some=Color(0.28571,'yellow',lambdav:"{:.5f}%".format(v*100)),step=1)logger.log(some=Color(0.85,'yellow',percent),step=2)logger.log({"some_var/smooth":10},some=Color(0.85,'yellow',percent),step=3)logger.log(some=Color(10,'yellow'),step=4)colored output: (where the values are yellow)╒════════════════════╤════════════════════╕
│        some        │        0.1         │
╘════════════════════╧════════════════════╛
╒════════════════════╤════════════════════╕
│        some        │     28.57100%      │
╘════════════════════╧════════════════════╛
╒════════════════════╤════════════════════╕
│        some        │       85.0%        │
╘════════════════════╧════════════════════╛
╒════════════════════╤════════════════════╕
│  some var/smooth   │         10         │
├────────────────────┼────────────────────┤
│        some        │       85.0%        │
╘════════════════════╧════════════════════╛This version of logger also prints out a tabular printout of the data you are logging to yourstdout.can silencestdoutper key (perlogger.logcall)can print with color:logger.log(timestep, some_key=green(some_data))can print with custom formatting:logger.log(timestep, some_key=green(some_data, percent))wherepercentuses the correctunixtable characters (please stop using|and+.Use│,┼instead)A typical print out of this logger look like the following:fromml_loggerimportML_Loggerlogger=ML_Logger(root_dir=f"/mnt/bucket/deep_Q_learning/{datetime.now(%Y%m%d-%H%M%S.%f):}")logger.log_params(G=vars(G),RUN=vars(RUN),Reporting=vars(Reporting))outputs the following═════════════════════════════════════════════════════
              G               
───────────────────────────────┬─────────────────────
           env_name            │ MountainCar-v0      
             seed              │ None                
      stochastic_action        │ True                
         conv_params           │ None                
         value_params          │ (64,)               
        use_layer_norm         │ True                
         buffer_size           │ 50000               
      replay_batch_size        │ 32                  
      prioritized_replay       │ True                
            alpha              │ 0.6                 
          beta_start           │ 0.4                 
           beta_end            │ 1.0                 
    prioritized_replay_eps     │ 1e-06               
      grad_norm_clipping       │ 10                  
           double_q            │ True                
         use_dueling           │ False               
     exploration_fraction      │ 0.1                 
          final_eps            │ 0.1                 
         n_timesteps           │ 100000              
        learning_rate          │ 0.001               
            gamma              │ 1.0                 
        learning_start         │ 1000                
        learn_interval         │ 1                   
target_network_update_interval │ 500                 
═══════════════════════════════╧═════════════════════
             RUN              
───────────────────────────────┬─────────────────────
        log_directory          │ /mnt/slab/krypton/machine_learning/ge_dqn/2017-11-20/162048.353909-MountainCar-v0-prioritized_replay(True)
          checkpoint           │ checkpoint.cp       
           log_file            │ output.log          
═══════════════════════════════╧═════════════════════
          Reporting           
───────────────────────────────┬─────────────────────
     checkpoint_interval       │ 10000               
        reward_average         │ 100                 
        print_interval         │ 10                  
═══════════════════════════════╧═════════════════════
╒════════════════════╤════════════════════╕
│      timestep      │        1999        │
├────────────────────┼────────────────────┤
│      episode       │         10         │
├────────────────────┼────────────────────┤
│    total reward    │       -200.0       │
├────────────────────┼────────────────────┤
│ total reward/mean  │       -200.0       │
├────────────────────┼────────────────────┤
│  total reward/max  │       -200.0       │
├────────────────────┼────────────────────┤
│time spent exploring│       82.0%        │
├────────────────────┼────────────────────┤
│    replay beta     │        0.41        │
╘════════════════════╧════════════════════╛TODO:Visualization (Preview):boom:In addition, ml-logger also comes with a powerful visualization dashboard that beats tensorboard in every aspect.An Example Log from ML-LoggerA common pain that comes after getting to launch ML training jobs on AWS
is a lack of a good way to manage and visualize your data. So far, a common
practice is to upload your experiment data to aws s3 or google cloud buckets.
Then one quickly realizes that downloading data from s3 can be slow. s3 doesnot offer diffsync like gcloud-cli'sg rsync. This makes it hard to sync a
large collection of data that is constantly appended to.So far the best way we have found for organizing experimental data is to
have a centralized instrumentation server. Compared with managing your data
on S3, a centralized instrumentation server makes it much easier to move
experiments around, run analysis that is co-located with your data, and
hosting visualization dashboards on the same machine. To download data
locally, you can usesshfs,smba,rsyncor a variety of remote disks. All
faster than s3.ML-Logger is the logging utility that allows you to do this. To make ML_logger
easy to use, we made it so that you can use ml-logger with zero configuration,
logging to your local hard-drive by default. When the logging directory fieldlogger.configure(log_directory= <your directory>)is an http end point,
the logger will instantiate a fast, future based logging client that launches
http requests in a separate thread. We optimized the client so that it won't
slow down your training code.API wise, ML-logger makes it easy for you to log textual printouts, simple
scalars, numpy tensors, image tensors, andpyplotfigures. Because you might
also want to read data from the instrumentation server, we also made it possible to
load numpy, pickle, text and binary files remotely.In the future, we will start building an integrated dashboard with fast search,
live figure update and markdown-based reporting/dashboarding to go with ml-logger.Now give this a try, and profit!