TensorFlow AutoGraph for .NET. TensorFlow 2.x has three mode of graph…
By: Haiping Chen
Tags: Data Science, Machinelerning, TensorFlow, C Sharp Programming, Deep Learning

TensorFlow 2.x has three mode of graph computation, namely static graph construction (the main method used by TensorFlow 1.x), Eager mode and AutoGraph method. In TensorFlow 2.x, the official recommendation is using the Eager mode and the AutoGraph method.

The Eager mode is convenient for model development and debugging by data scientist, the code is readable and the coding efficiency is high for the machine learning engineers, but the computation performance is not as good as the static computation graph. The execution efficiency of the static graph is very high, but the code writing and readability are poor, and the program debugging is difficult.

In TensorFlow 2, eager execution is turned on by default. The user interface is intuitive and flexible (running one-off operations is much easier and faster), but this can come at the expense of performance and deployability.

AutoGraph takes into account the two major advantages of dynamic graph coding efficiency and static graph execution efficiency. Literally understood, the AutoGraph mechanism is an automatic calculation graph conversion function, which can write codes developed in accordance with dynamic graph writing rules. , Through the AutoGraph mechanism, it is converted into a static computation graph and executed in the same way as a static calculation graph, achieving the purpose of “quickly writing code and debugging code in a dynamic graph” + “efficient code execution in a static graph”.

Of course, the code and usage scenarios that the AutoGraph mechanism can convert are not unconstrained (or it will completely replace the static calculation graph and the dynamic calculation graph). The fitness of the AutoGraph mechanism requires certain coding rules and a deep understanding of AutoGraph. Perform the internal operation process of the calculation graph conversion, otherwise the conversion will fail, or an abnormal execution process that does not meet the expectations will occur.

In the next two sections, we will explain in detail the principles and coding specifications of AutoGraph’s internal operating mechanism. Next, we will use a simple example to understand how to use the AutoGraph mechanism.

This is the simplest Tensor numeric multiplication operation:

public void Run()

{

var a = tf.constant(2);

var b = tf.constant(3);

var output = Mul(a, b);

print(output);



Console.ReadKey();

} Tensor Mul(Tensor a, Tensor b)

{

return a * b;

}

After running, it will output the result of multiplying the values ​​2 * 3 = 6:

tf.Tensor: shape=(), dtype=int32, numpy=6

This process is carried out in dynamic computation graph mode, namely numerical computation in Eager Mode.

In TensorFlow for .NET, we implement the AutoGraph mechanism in two ways:

Method ① Manually run the tf.autograph.to_graph() method to convert the function into a static computation graph;

Method ② Add Attribute property label AutoGraph before the function body to modify the function to realize automatic static graph conversion.

We recommend ways ② [AutoGraph] way, more flexible and convenient, the code more readable, let these two ways illustrate, respectively.

① tf.autograph.to_graph() detailed description

We can use the tf.autograph.to_graph() method to convert the dynamic calculation graph to a static calculation graph. The usage is very simple. It can be completed by adding a conversion code directly. Let’s perform the AutoGraph conversion test on the above multiplication operation.

Inject tf.autograph.to_graph() for AutoGraph conversion:

public void Run()

{

var func = tf.autograph.to_graph(Mul);// AutoGraph

var a = tf.constant(2);

var b = tf.constant(3);

var output = func(a, b);

print(output);



Console.ReadKey();

} Tensor Mul(Tensor a , Tensor b)

{

return a * b;

}

By adding a line of code var func = tf.autograph.to_graph(Mul), we can realize the calculation graph conversion of the AutoGraph mechanism, and the result of the multiplication operation remains unchanged:

tf.Tensor: shape=(), dtype=int32, numpy=6

AutoGraph operating efficiency

Generally speaking, when the computational graph model is composed of a large number of small operations, the efficiency of AutoGraph is greatly improved. If the number of model operations is small, the efficiency of AutoGraph is not greatly improved, sometimes due to the time-consuming conversion of computational graphs , Resulting in longer running time.

Let’s make a simple modification to the previous example and add a loop test to test the running time under different conditions.

Case 1# We directly increase the loop, and run the multiplication loop one million times

code show as below:

public void Run()

{

Stopwatch sw = new Stopwatch();

sw.Restart();

foreach (var item in range(1000000))

{

var a = tf.constant(2);

var b = tf.constant(3);

var output = Mul(a, b);

}

sw.Stop();

print("Eager Mode：" + sw.Elapsed.TotalSeconds.ToString("0.0 s"));



sw.Restart();

var func = tf.autograph.to_graph(Mul);//AutoGraph

foreach (var item in range(1000000))

{

var a = tf.constant(2);

var b = tf.constant(3);

var output = func(a, b);

}

sw.Stop();

print("AutoGraph Mode：" + sw.Elapsed.TotalSeconds.ToString("0.0 s"));



Console.ReadKey();

} Tensor Mul(Tensor a, Tensor b)

{

return a * b;

}

The results are as follows:

Eager Mode：20.2 s

AutoGraph Mode：41.8 s

We can see that the Eager mode 20.2 s is faster than the AutoGraph mode 41.8 s (different PC configurations may have slightly different calculation results).

Case 2# We add a little operand, and then run the multiplication operation in the same loop one million times

The operand of the multiplication operation has been added and modified to 4 consecutive multiplications a * b * a * b * a * b * a * b, the code is as follows:

public void Run()

{

Stopwatch sw = new Stopwatch();

sw.Restart();

foreach (var item in range(1000000))

{

var a = tf.constant(2);

var b = tf.constant(3);

var output = Mul(a, b);

}

sw.Stop();

print("Eager Mode：" + sw.Elapsed.TotalSeconds.ToString("0.0 s"));



sw.Restart();

var func = tf.autograph.to_graph(Mul);//AutoGraph

foreach (var item in range(1000000))

{

var a = tf.constant(2);

var b = tf.constant(3);

var output = func(a, b);

}

sw.Stop();

print("AutoGraph Mode：" + sw.Elapsed.TotalSeconds.ToString("0.0 s"));



Console.ReadKey();

} Tensor Mul(Tensor a , Tensor b)

{

return a * b * a * b * a * b * a * b;

}

The results are as follows:

Eager Mode：78.3 s

AutoGraph Mode：47.1 s

This time, we can see that Eager mode 78.3 s is much slower than AutoGraph mode 47.1 s (different PC configurations may have slightly different calculation results), and the efficiency advantage of AutoGraph has been reflected.

Next, let’s look at other simple code examples of AutoGraph.

Example 1: Simple conditional comparison operation

A simple way to compare the size of 2 Tensor constant values ​​and output the smaller value:

public void Run ()

{

var func = tf.autograph.to_graph(Condition_Min);

var a = tf.constant(3);

var b = tf.constant(2);

var output = func(a, b);

print(output);



Console.ReadKey();

} // TwoInputs_OneOutput_Condition

Tensor Condition_Min(Tensor a, Tensor b)

{

return tf.cond(a < b, a, b);

}

Run the code, the smaller value 2 is correctly output:

tf.Tensor: shape=(), dtype=int32, numpy=2

Example 2: AutoGraph conversion of Lambda expression

The AutoGraph mechanism also supports the conversion of static calculation graphs of Lambda expressions. A simple sample code is as follows:

public void Run ()

{

var func = tf.autograph.to_graph((x, y) => x * y);

var output = func(tf.constant(3), tf.constant(2));

print(output);



Console.ReadKey();

}

Running the code, the operation result of the Lambda multiplication expression is correctly output 6:

tf.Tensor: shape=(), dtype=int32, numpy=6

② [AutoGraph] detailed description

Another recommended AutoGraph conversion method is to add the Attribute property label [AutoGraph] in front of the function body to modify the function to realize automatic static graph conversion. This method is more convenient, the amount of code changes is less, and the program can be modified and more readable.

Let’s take a look through a few different examples.

Example 1# Floating point addition operation

This time we do an addition operation of floating-point decimals, the code is as follows:

public void Run ()

{

var a = tf.constant(2.0);

var b = tf.constant(1.5);

var output = Add(a, b);

print(output);



Console.ReadKey();

} Tensor Add(Tensor a, Tensor b)

{

var c = a + b;

return c;

}

After running, the program outputs the result of the addition operation:

tf.Tensor: shape=(), dtype=TF_DOUBLE, numpy=3.5

We can implement AutoGraph conversion by adding the feature tag [AutoGraph] in front of the Add() method. The code is as follows:

public void Run ()

{

var a = tf.constant(2.0);

var b = tf.constant(1.5);

var output = Add(a, b);

print(output); Console.ReadKey();

}

[AutoGraph]

Tensor Add(Tensor a, Tensor b)

{

var c = a + b;

return c;

}

After running, the program outputs the same result of the addition operation:

tf.Tensor: shape=(), dtype=TF_DOUBLE, numpy=3.5

Example 2# MNIST logistic regression test

Next, we will test a slightly more complicated case-MNIST logistic regression. For the specific content of MNIST logistic regression, you can refer to the example Handwritten Character Classification Logistic Regression.

At the same time, we added a timer to test whether there is a difference in time between Eager mode and AutoGraph mode.

The complete console code of MNIST logistic regression in Eager mode is as follows:

using static Tensorflow.KerasApi;

using Tensorflow;

using static Tensorflow.Binding;

using Tensorflow.Keras.Optimizers;

using System.Diagnostics; namespace Test_Core

{

class Program

{

static void Main(string[] args)

{

Stopwatch sw = new Stopwatch();

sw.Restart(); TFNET tfnet = new TFNET ();

tfnet . Run (); sw.Stop();

print("Time Last：" + sw.Elapsed.TotalSeconds.ToString("0.0 s"));

}

} class TFNET

{

ResourceVariable W = tf.Variable(tf.ones((784, 10)), name: "weight");

ResourceVariable b = tf.Variable(tf.zeros(10), name: "bias");

SGD optimizer = keras.optimizers.SGD(0.01f);

public void Run ()

{

int training_epochs = 20000;

int batch_size = 256;

int num_features = 784; // 28*28

int display_step = 1000;

float accuracy = 0f; var ((x_train, y_train), (x_test, y_test)) = keras.datasets.mnist.load_data();

(x_train, x_test) = (x_train.reshape((-1, num_features)), x_test.reshape((-1, num_features)));

(x_train, x_test) = (x_train / 255f, x_test / 255f); var train_data = tf.data.Dataset.from_tensor_slices(x_train, y_train);

train_data = train_data.repeat().shuffle(5000).batch(batch_size).prefetch(1);

train_data = train_data.take(training_epochs); foreach (var (step, (batch_x, batch_y)) in enumerate(train_data, 1))

{

run_optimization(batch_x, batch_y); if (step % display_step == 0)

{

var pred = logistic_regression(batch_x);

var loss = cross_entropy(pred, batch_y);

var acc = Accuracy(pred, batch_y);

print($"step: {step}, loss: {(float)loss}, accuracy: {(float)acc}");

accuracy = acc.numpy();

}

} {

var pred = logistic_regression(x_test);

print($"Test Accuracy: {(float)Accuracy(pred, y_test)}");

}

}

Tensor Accuracy(Tensor y_pred, Tensor y_true)

{

var correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64));

return tf.reduce_mean(tf.cast(correct_prediction, tf.float32));

}

Tensor cross_entropy(Tensor y_pred, Tensor y_true)

{

y_true = tf.cast(y_true, TF_DataType.TF_UINT8);

y_true = tf.one_hot(y_true, depth: 10);

y_pred = tf.clip_by_value(y_pred, 1e-9f, 1.0f);

return tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred), 1));

}

Tensor logistic_regression(Tensor x)

{

return tf.nn.softmax(tf.matmul(x, W) + b);

}

private void run_optimization(Tensor x, Tensor y)

{

using var g = tf.GradientTape();

var pred = logistic_regression(x);

var loss = cross_entropy(pred, y);

var gradients = g.gradient(loss, (W, b));

optimizer.apply_gradients(zip(gradients, (W, b)));

} }

}

The program execution took 47.2 seconds, and the running result output is as follows (different PCs may have slightly different execution time and calculation results):

step: 1000, loss: 0.60991216, accuracy: 0.859375

step: 2000, loss: 0.52773017, accuracy: 0.8515625

step: 3000, loss: 0.49317402, accuracy: 0.86328125

step: 4000, loss: 0.3669852, accuracy: 0.8984375

step: 5000, loss: 0.34490967, accuracy: 0.91015625

step: 6000, loss: 0.36495364, accuracy: 0.90625

step: 7000, loss: 0.3448297, accuracy: 0.91796875

step: 8000, loss: 0.3595497, accuracy: 0.890625

step: 9000, loss: 0.4222083, accuracy: 0.87109375

step: 10000, loss: 0.3518588, accuracy: 0.90625

step: 11000, loss: 0.3254419, accuracy: 0.91015625

step: 12000, loss: 0.3375517, accuracy: 0.90234375

step: 13000, loss: 0.47696534, accuracy: 0.875

step: 14000, loss: 0.38535655, accuracy: 0.890625

step: 15000, loss: 0.4106849, accuracy: 0.875

step: 16000, loss: 0.33494166, accuracy: 0.90625

step: 17000, loss: 0.41341114, accuracy: 0.91796875

step: 18000, loss: 0.33995813, accuracy: 0.90625

step: 19000, loss: 0.32934952, accuracy: 0.9140625

Test Accuracy: 0.9166

Time Last：47.2 s

We can implement AutoGraph conversion by adding the feature tag [AutoGraph] in front of the optimizer run_optimization() method. The complete console code is as follows:

using static Tensorflow.KerasApi;

using Tensorflow;

using static Tensorflow.Binding;

using Tensorflow.Keras.Optimizers;

using System.Diagnostics;

using Tensorflow.Graphs; namespace Test_Core

{

class Program

{

static void Main(string[] args)

{

Stopwatch sw = new Stopwatch();

sw.Restart(); TFNET tfnet = new TFNET ();

tfnet . Run (); sw.Stop();

print("Time Last：" + sw.Elapsed.TotalSeconds.ToString("0.0 s"));

}

} class TFNET

{

ResourceVariable W = tf.Variable(tf.ones((784, 10)), name: "weight");

ResourceVariable b = tf.Variable(tf.zeros(10), name: "bias");

SGD optimizer = keras.optimizers.SGD(0.01f);

public void Run ()

{

int training_epochs = 20000;

int batch_size = 256;

int num_features = 784; // 28*28

int display_step = 1000;

float accuracy = 0f; var ((x_train, y_train), (x_test, y_test)) = keras.datasets.mnist.load_data();

(x_train, x_test) = (x_train.reshape((-1, num_features)), x_test.reshape((-1, num_features)));

(x_train, x_test) = (x_train / 255f, x_test / 255f); var train_data = tf.data.Dataset.from_tensor_slices(x_train, y_train);

train_data = train_data.repeat().shuffle(5000).batch(batch_size).prefetch(1);

train_data = train_data.take(training_epochs); foreach (var (step, (batch_x, batch_y)) in enumerate(train_data, 1))

{

run_optimization(batch_x, batch_y); if (step % display_step == 0)

{

var pred = logistic_regression(batch_x);

var loss = cross_entropy(pred, batch_y);

var acc = Accuracy(pred, batch_y);

print($"step: {step}, loss: {(float)loss}, accuracy: {(float)acc}");

accuracy = acc.numpy();

}

} {

var pred = logistic_regression(x_test);

print($"Test Accuracy: {(float)Accuracy(pred, y_test)}");

}

}

Tensor Accuracy(Tensor y_pred, Tensor y_true)

{

var correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64));

return tf.reduce_mean(tf.cast(correct_prediction, tf.float32));

}

Tensor cross_entropy(Tensor y_pred, Tensor y_true)

{

y_true = tf.cast(y_true, TF_DataType.TF_UINT8);

y_true = tf.one_hot(y_true, depth: 10);

y_pred = tf.clip_by_value(y_pred, 1e-9f, 1.0f);

return tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred), 1));

}

Tensor logistic_regression(Tensor x)

{

return tf.nn.softmax(tf.matmul(x, W) + b);

}

[AutoGraph]

private void run_optimization(Tensor x, Tensor y)

{

using var g = tf.GradientTape();

var pred = logistic_regression(x);

var loss = cross_entropy(pred, y);

var gradients = g.gradient(loss, (W, b));

optimizer.apply_gradients(zip(gradients, (W, b)));

} }

}

This time the program execution took 45.3 seconds, and the running result output is as follows (different PCs may have slightly different execution time and calculation results):

step: 1000, loss: 0.5845629, accuracy: 0.88671875

step: 2000, loss: 0.42607152, accuracy: 0.8984375

step: 3000, loss: 0.5029169, accuracy: 0.85546875

step: 4000, loss: 0.33637348, accuracy: 0.90625

step: 5000, loss: 0.4658397, accuracy: 0.875

step: 6000, loss: 0.3632791, accuracy: 0.89453125

step: 7000, loss: 0.38538253, accuracy: 0.90234375

step: 8000, loss: 0.3392726, accuracy: 0.90234375

step: 9000, loss: 0.30917627, accuracy: 0.90234375

step: 10000, loss: 0.3470266, accuracy: 0.9140625

step: 11000, loss: 0.33611432, accuracy: 0.921875

step: 12000, loss: 0.37535334, accuracy: 0.89453125

step: 13000, loss: 0.35773978, accuracy: 0.890625

step: 14000, loss: 0.35660252, accuracy: 0.8984375

step: 15000, loss: 0.23310906, accuracy: 0.9453125

step: 16000, loss: 0.3695503, accuracy: 0.88671875

step: 17000, loss: 0.31830546, accuracy: 0.90625

step: 18000, loss: 0.2850374, accuracy: 0.91796875

step: 19000, loss: 0.26065654, accuracy: 0.9375

Test Accuracy: 0.917

Time Last：45.3 s

Comparing the results of the second time, after adding the AutoGraph mechanism, the program operation efficiency is not significantly improved, and the execution time is not much different.

Through the above two examples of “MNIST Logistic Regression Example” and “One Million Multiplication Operations”, we can see that for a single operation that is time-consuming, the performance improvement brought by the AutoGraph mechanism will not be much , And when the model consists of many small operations, the efficiency of the AutoGraph mechanism is greatly improved. In the actual application process, you can comprehensively consider specific application scenarios and selectively apply the AutoGraph mechanism.

Principle of AutoGraph Mechanism

When we use the tf.autograph.to_graph() or [AutoGraph] method to convert a function into a static calculation graph, what happens inside the program?

For example, we test the following code:

public void Run ()

{

var func = tf.autograph.to_graph(Add);//AutoGraph

print("First Run:Initialization"); var a = tf.constant(2);

var b = tf.constant(3);

var output = func(a, b);

print($"Second Run:{output.numpy()}"); output = func(output, b);

print($"Third Run:{output.numpy()}"); Console.ReadKey();

} Tensor Add(Tensor a, Tensor b)

{

foreach (var i in range(3))

print(i);

var c = a + b;

print(c);

print("tracing"); return c;

}

After running, we will see the following results:

0

1

2

tf.Tensor 'add:0' shape=<unknown> dtype=int32

tracing

First Run:Initialization

Second Run:5

Third Run:8

Let’s analyze this process step by step. In general, two things have happened.

When we wrote the function body of Add(), nothing happened, only the signature of such a function was recorded in the C# memory stack.

When we call this function for the first time through the tf.autograph.to_graph() method, the first thing happens. The program creates a static calculation graph by analyzing the content of the Add() function body. The parsing process will track the code that executes the weight of the function, determine the Tensor type of each variable, and add the OP operator to the calculation graph according to the execution order to generate a fixed static calculation graph.

Therefore, we see that the terminal sequentially outputs the value of the loop variable i = 0 1 2, the initial content of the Tensor add operator c inside the function, and the print test string tracing at the end of the function body. Here we can also see from the content of the tensor c inside the function body that c is just an operation operator in the static calculation graph, and the initial calculation result 0 is not obtained according to the Eager mode.

When we call the function for the second time and pass in the parameter Tensor to be calculated, the second thing happens. The program automatically calls the static calculation graph just created, passes in the Tensor to be calculated, executes the calculation graph, and outputs the calculated result. In this process, the program no longer runs the internal code of the function body, and directly calls the calculation graph created in the previous step for calculations, so the print() method of many tests inside the function body will not be executed again.

Therefore, the program only outputs the second result “Second Run: 5”.

This operation is no longer in the Eager mode method, but similar to the following statement executed in TensorFlow 1.x:

using (var sess = tf.Session(tf.get_default_graph()))

{

var result = sess.run(tf.constant(2),tf.constant(3));

}

The process of calling the function for the third time is exactly the same as that for the second time. The program still calls the original calculation graph for repeated calculations and outputs the result “Third Run: 8”.

We can see that through the AutoGraph mechanism, the execution efficiency of repeated operations can be improved dramatically.

Here I simply demonstrate the process by way of diagrams, as shown in the following figure:

Thanks for the SciSharp STACK again, bring fancy Machine Learning technology to .NET community.