Package: tensorflow-tracer
TensorFlow Runtime TracerThis project is a web application to monitor and trace TensorFlow scripts in the runtime on theoplevel.It starts a web server upon the execution of the script. The web interface keeps track of all the session runs and can trace the execution on demand.The goal of this tool is to facilitate the process of performance tuning with minimal code changes and insignificant runtime overhead. Both Higher-level (tf.estimator.Estimator) and Low-level (tf.train.MonitoredTrainingSessionand co) APIs are supported. It also supportshorovodandIBM Distributed Deep Learning (DDL).
The tracing session can be saved, reloaded, and distributed effortlessly.Some screenshotshere.InstallationUsepipto install:pipinstalltensorflow-tracerQuick StartInstalltensorflow-tracerand run an example:$ pip3 install tensorflow-tracer
$ git clone https://github.com/xldrx/tensorflow-tracer.git
$ python3 ./tensorflow-tracer/examples/estimator-example.pyBrowse to:http://0.0.0.0:9999How to UseAddtftracerto your code:Estimator API:fromtftracerimportTracingServer...tracing_server=TracingServer()estimator.train(input_fn,hooks=[tracing_server.hook])Low-Level API:fromtftracerimportTracingServer...tracing_server=TracingServer()withtf.train.MonitoredTrainingSession(hooks=[tracing_server.hook]):...[More examples here]Run your code and browse to:http://0.0.0.0:9999How to Trace an Existing CodeIf you want to trace an existing script without any modification usetftracer.hook_injectPlease note that this is experimental and may cause unexpected errors:Add the following to the beggining of the main script:
.. code-block:: pythonimport tftracer
tftracer.hook_inject()
...Run your code and browse tohttp://0.0.0.0:9999Command lineTracing sessions can be stored either through the web interface or by callingtracing_server.save_session(filename).To reload a session, run this in the terminal:tftracerfilenameThen browse to:http://0.0.0.0:9999APIFull Documentation ishere.Known Bugs/LimitationsOnly Python3 is supported.The web interface loads javascript/css libraries remotely (e.g.vue.js,ui-kit,jquery,jquery-ui,Google Roboto,awesome-icons, ... ). Therefore an active internet connection is needed to properly render the interface. The tracing server does not require any remote connection.All traces are kept in the memory while tracing server is running.Tracing usestf.train.SessionRunHookand is unable to trace auxiliary runs such asinit_op.The tracing capability is limited to whattf.RunMetadataoffers. For example, CUPTI events are missing when tracing a distributed job.HTTPS is not supported.Frequently Asked QuestionsHow to trace/visualize just one session run?Usetftracer.Timeline. for example:fromtftracerimportTimeline...withtf.train.MonitoredTrainingSession()assess:withTimeline()astl:sess.run(fetches,**tl.kwargs)...tl.visualize(filename)Comparision to TensorBoard?The nature of this project is a short-lived light-weight interactive tracing interface to monitor and trace execution on theop-level. In comparisonTensorBoardis a full-featured tool to inspect the application on many levels:tftracerdoes not make any assumption about the dataflow DAG. There is no need to add any additionalopto the data flow dag (i.e.tf.summary) or having aglobal step.tftracerruns as a thread and lives from the start of the execution and lasts until the end of it.TensorBoardruns as a separate process and can outlive the main script.Cite this tool@misc{hashemi-tftracer-2018,
  author ={Sayed Hadi Hashemi},
  title ={TensorFlow Runtime Tracer},
  year ={2018},
  publisher ={GitHub},
  journal ={GitHub repository},
  howpublished ={\url{https://github.com/xldrx/tensorflow-tracer}},}