Faster ML Developmentwith TensorFlow
Shanqing Cai (cais@google.com)
Senior Software Engineer, Google Brain (Cambridge, MA)

Guest Lecture @ MIT 6.S191
February, 2018

How are machine learning models represented?

Model is a Data Structure

Model is a Program

e.g. A Graphe.g. Python Code

aka

aka

“Symbolic” | “Deferred Execution” |“Define-and-run”“Imperative” | “Eager Execution” |“Define-by-run”

TensorFlow: Symbolic Mode
By default, TensorFlow is a symbolic engine.
import tensorflow as tfaddx = tf.constant(10.0)
w = tf.constant(4.0)

multiplyb = tf.constant(2.0)
y = tf.multiply(x, w)
print(y)# You get: Tensor("Mul:0",shape=(), dtype=float32)

xwb

Model as a Data Structurez = tf.add(y, b)
print(z)# You get: Tensor("Add:0",shape=(), dtype=float32)# You need to create a “session” to perform the# actual computation.
sess = tf.Session()tf.Session

print(sess.run(z))# You get: 42.0.

Output and/or model updatesSymbolic Execution in TensorFlow
Pros:+ makes (de)serialization easier+ deployment on devices(e.g., mobile, TPU, XLA)
Model as a Data Structure

XLA:
Optimized binary forCPUs and accelerators
TF Session

TF Session

Symbolic Execution in TensorFlow
Pros:+ makes (de)serialization easier+ deployment on devices(e.g., mobile, TPU, XLA)
Model as a Data Structure+ interoperability between languages

Java...

Symbolic Execution in TensorFlow
Pros:+ makes (de)serialization easier+ deployment on devices

Shared states:model weights andupdates(e.g., mobile, TPU, XLA)
+ interoperability between languages

ParameterServer

Model as a Data Structure+ distributed training...
...
Worker 1

Worker 2

Worker 3

Worker nSymbolic Execution in TensorFlow
Pros:+ makes (de)serialization easier+ deployment on devices

Model is a Data Structure(e.g., mobile, TPU, XLA)

e.g. A Graph+ interoperability between languages+ distributed training+ speed and concurrency not limited by language(e.g., Python global interpreter lock)aka“Symbolic” | “Deferred Execution”

Symbolic Execution in TensorFlow
Pros:+ makes (de)serialization easier+ deployment on devices

Model is a Data Structure(e.g., mobile, TPU, XLA)

e.g. A Graph+ interoperability between languages+ distributed training+ speed and concurrency not limited by language(e.g., Python global interpreter lock)aka“Symbolic” | “Deferred Execution”

Symbolic Execution in TensorFlow
Pros:

Cons:

+ makes (de)serialization easier- less intuitive+ deployment on devices- harder to debug (*but see later slides)(e.g., mobile, TPU, XLA)

- harder to write control flow structures+ interoperability between languages+ distributed training+ speed and concurrency not limited by language(e.g., Python global interpreter lock)- harder to write dynamic models

Eager Execution in TensorFlow+ easier to learn (“Pythonic”)+ easier to debug+ makes dynamic (data-dependent)
neural structures easier to write

Model is a Program
e.g. Python Codeaka“Imperative” | “Eager Execution”

Eager Execution in TensorFlow
By default, TensorFlow is a symbolic engine.
import tensorflow as tf

But since version1.5, you can switch tothe imperative (eager) mode.
import tensorflow as tf
import tensorflow.contrib.eager as tfetfe.enable_eager_execution()x = tf.constant(10.0)

x = tf.constant(10.0)

w = tf.constant(4.0)

w = tf.constant(4.0)

b = tf.constant(2.0)

b = tf.constant(2.0)

y = tf.multiply(x, w)

y = tf.multiply(x, w)

print(y)print(y)# You get: Tensor("Mul:0",shape=(), dtype=float32)# You get: tf.Tensor(40.0,shape=(), dtype=float32)z = tf.add(y, b)z = tf.add(y, b)

print(z)print(z)# You get: Tensor("Add:0",shape=(), dtype=float32)# You get: tf.Tensor(42.0,shape=(), dtype=float32)

See eager-mode examples and notebooks.

Symbolic vs. Eager Mode+ easier to learn (“Pythonic”)+ easier to debug+ makes dynamic (data-dependent)
neural structures easier to write

Model is a Program
e.g. Python Codeaka

“Imperative” | “Eager Execution”

TensorFlow: Control Flow in Symbolic vs. Eager
Writing a basic RNN:

http://colah.github.io/posts/2015-08Understanding-LSTMs/

Symbolic

Eager

dense1 = tf.layers.Dense(state_size, activation='tanh')dense1 = tf.layers.Dense(state_size, activation='tanh')dense2 = tf.layers.Dense(state_size)

dense2 = tf.layers.Dense(state_size)

def loop_cond(i, state, output):

for i in xrange(max_sequence_len):return i < max_sequence_len

input_slice = input_array.read(i)combined = tf.concat([input_slice, state], axis=1)

def loop_body(i, state, output):state_updated = dense1(combined)

input_slice = input_array.read(i)state = tf.where(i >= sequence_lengths, state, state_updated)combined = tf.concat([input_slice, state], axis=1)

output_updated = dense2(state)state_updated = dense1(combined)

output = tf.where(state = tf.where(i >= sequence_lengths, state, state_updated)i >= sequence_lengths, output, output_updated)

output_updated = dense2(state)
output = tf.where(i >= sequence_lengths, output, output_updated)
return i + 1, state, output_, final_state, final_output = tf.while_loop(loop_cond, loop_body,[i, initial_state, dummy_initial_output])sess.run([final_state, final_output])

final_state, final_output = state, output

Model Structures: Static vs. Dynamic
Static models

Source: Inception model in TensorFlow＋ Model structure is fixed regardless of input data.
＋ The majority of DL models for image, audio and numerical data.

Model Structures: Static vs. Dynamic
Traditional RNN

Dynamic Models, e.g., Tree RNN
SNP
I

eat

kale

IVPVNPIeat

kaleeat＋ Models whose structure cannot be easily described as a graph, i.e., changes a lot with input data.
＋ Used by some state-of-the-art models that deal with hierarchical structures in natural language.
＋ Difficult to write in the symbolic way (using tf.cond and tf.while_loop)＋ Straightforward with Eager: using the native Python control flow. See the SPINN example.

kaleWhat if you want to debug symbolic execution?
TensorFlow Debugger (tfdbg):
Command Line Interface
import tensorflow as tf
from tensorflow.python import debug as tfdbg

tfdbg

a = tf.constant(10.0)
b = tf.Variable(4.0)
c = tf.Variable(2.0)
x = tf.multiply(a, b)
y = tf.add(c, x)sess = tf.Session()sess = tfdbg.LocalCLIDebugWrapperSession(sess)
sess.run(tf.global_variables_initializer())
sess.run(y)tf.Session

What if you want to debug symbolic execution?
import tensorflow as tf
from tensorflow.python import debug as tfdbg
a = tf.constant(10.0)
b = tf.Variable(4.0)
c = tf.Variable(2.0)
x = tf.multiply(a, b)
y = tf.add(c, x)
sess = tf.Session()sess = tfdbg.LocalCLIDebugWrapperSession(sess)
sess.run(tf.global_variables_initializer())
sess.run(y)● Presents after each Session.run:○ All tensor values in the computation graph○ Graph structure… in an interactive, mouse-clickable CLI.

TensorFlow: Debugging Numerical Instability(NaNs and Infinities)
Common causes of NaNs and infinities
in DL models:○ underflow followed by:■ division by zero
■ logarithm of zerotfdbg> run -f has_inf_or_nan
See walkthrough at
https://www.tensorflow.org/programmers_guide/debugger○ overflow caused by:■ learning rate too high■ bad training examples

New Tool: Graphical Debugger for TensorFlow(TensorBoard Debugger Plugin)# Do the following in a terminal.

import tensorflow as tf
from tensorflow.python import debug as tf_debug# Install nightly builds.
pip install --upgrade --force-reinstall \tf-nightly tb-nightly grpcio

a = tf.random_normal([10, 1])
b = tf.random_normal([10, 10])c = tf.random_normal([10, 1])# Start tensorboard with debugger enabled.
tensorboard \--logdir /tmp/logdir \x = tf.matmul(b, a)
y = tf.add(c, x)--port 6006 \--debugger_port 7007

sess = tf.Session()sess = tf_debug.TensorBoardDebugWrapperSession(# Open a browser and navigate to:#http://localhost:6006/#debugger

sess, 'localhost:7007')
for _ in xrange(100):
sess.run(y)# Then save the code in a file and run it. -->● Not publicly announced yet (coming in TensorFlow 1.6)
● But available for preview in nightly builds of tensorflow
and tensorboard

Try it yourself!

New Tool: Visual Debugger for TensorFlow

A tree view of allgraph nodes.
Checkbox = watch.

Detailed view ofwatched tensor
values.

Right-click nodes andselect “expand andhighlight” to go to thecorresponding line in thesource code.

View the runtime
graph structure.
Tying graph nodes
back to the Pythonlines that createdthem.

Step node by node(tensor by tensor).

Continue overSession.runs or to acertain tensor-value
condition.

View summaries ofwatched tensor
values.

Summary● ML/DL models can be represented in two ways:○ as a data structure → Symbolic Execution:
good for deployment, distribution, and optimization○ as a program → Eager Execution:
good for prototyping, debugging and dynamic models; easier to learn● TensorFlow supports both modes● TensorFlow Debugger (tfdbg) provides visibility into
symbolically-executing models and help you debug/understand them in:○ command line
○ browser

Acknowledgements
Google Brain Team in Mountain View, CA and Cambridge, MA.
Chi Zeng and Mahima Pushkarna: Collaborators on the visual tfdbg project.
Open-source contributors to TensorFlow.

Thank you!
For questions, email cais@google.com
For TensorFlow issues, go to https://github.com/tensorflow/tensorflow/issues
For TensorBoard issues, go to https://github.com/tensorflow/tensorboard/issues