[TensorFlow] Python ⇔ Protocol Buffers ⇔ GPU/分散コンピューティング

# はじめに
ディープラーニングのフレームワークTensorFlowは各種処理をProtocol Buffers経由で外だしすることで、PythonとGPU／分散コンピュータとのスイッチングを減らし、計算を効率化しています。

[Protocol Buffers](https://developers.google.com/protocol-buffers/)は、Googleの分散コンピューティングを支える技術で、言語非依存、プラットフォーム非依存にデータ構造をシリアライズする仕組みです。現在、C++, C#, GO, Java, Pythonがサポートされています。

![ProtocolBuffers.png](https://qiita-image-store.s3.amazonaws.com/0/107056/18601620-88c4-829e-b880-a70c7ebed757.png)


TensorFlowは、処理をノードとしてグラフを構築し、一気に計算する仕組みになっています。
[グラフの例](https://qiita-image-store.s3.amazonaws.com/0/107056/16f4c460-0906-b983-64e8-b19289e5551b.png)

以下TensorFlowのグラフがProtocol Buffers形式にシリアライズされる様子を見てみたいと思います。

# サンプルプログラム
TensorFlowで足し算をしてみます。
初期値０で加算１を３回実行します。

```add.py
import tensorflow as tf

state = tf.Variable(0, name="counter")

one = tf.constant(1)
new_value = tf.add(state, one)
update = tf.assign(state, new_value)

init_op = tf.initialize_all_variables()

with tf.Session() as sess:
  sess.run(init_op)
  print(sess.run(state))
  for _ in range(3):
    sess.run(update)
    print(sess.run(state))
```

# TensorBoardのグラフ
**TensorBoardでグラフを表示**
![add.png](https://qiita-image-store.s3.amazonaws.com/0/107056/06f8d233-0c02-1016-4762-f964fcb60fe4.png)

# Protocol Buffers
以下、TensorFlowを使ったPythonのプログラムが、Protocol Buffersのノードに変換されると、どのようになるのかを見ていきます。

## 変数の定義
+ Python

```
state = tf.Variable(0, name="counter")
```

+ Protocol Buffers：counterという名前の変数

```
name: "counter"
op: "Variable"
attr {
  key: "container"
  value {
    s: ""
  }
}
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "shape"
  value {
    shape {
    }
  }
}
attr {
  key: "shared_name"
  value {
    s: ""
  }
}
```

+ Protocol Buffers：初期値０（定数）の定義

```
name: "counter/initial_value"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 0
    }
  }
}
```

+ Protocol Buffers：変数counterに初期値０をアサイン

```
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}
```

+ Protocol Buffers：変数counterを読みだす

```
name: "counter/read"
op: "Identity"
input: "counter"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
```

## 定数（加算値）の定義
+ Python

```
one = tf.constant(1)
```

+ Protocol Buffers：定数１の定義

```
name: "Const"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 1
    }
  }
}
```

## 加算オペレーションの定義
+ Python

```
one = tf.constant(1)
```

+ Protocol Buffers：加算オペレーションを定義

```
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
```

## 加算結果を変数に値を代入
+ Python

```
update = tf.assign(state, new_value)
```

+ Protocol Buffers：加算結果を変数に値を代入

```
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}
```

# おわりに
Pythonプログラムの裏でTensorFlowがProtocol Buffersを利用していることを理解していると、なぜその場所にTensorFlowの処理が差し込まれているのか、分かってくるのではないでしょうか。

TensorFlowには、ベースとなる各種処理が用意されています（[API](https://www.tensorflow.org/versions/master/api_docs/python/framework.html)）。

+ グラフ
+ 変数
+ テンソル
+ 数学
+ 文字列
+ 制御構文
+ アサーション
+ イメージ
+ 動画
+ 入出力
+ ニューラルネットワーク
+ オペレーションのサマリー
+ ユニットテスト

TensorFlowをディープラーニングのフレームワークとしてではなく、GPUコンピューティング/分散コンピューティングのフレームワークとして利用してみるのも面白いですね。
# 　　　　　　
　
　
　
　
　
　
　
　
　
　
　
　
# （参考）
### グラフ表示プログラム

```
graph = tf.get_default_graph()
summary_writer = tf.train.SummaryWriter('log_valiable', graph)
operations =  graph.get_operations()
for operation in operations:
    print("======================")
    print("=== name ===")
    print(operation.name)
    print("=== type ===")
    print(operation.type)
    print("=== inputs ===")
    for input in operation.inputs:
        print(input)
    print("=== control_inputs ===")
    for control_input in operation.control_inputs:
        print(control_input)
    print("=== outputs ===")
    for output in operation.outputs:
        print(output)
    print("=== node_def ===")
    print(operation.node_def)
    print("=== op_def ===")
    print(operation.op_def)
    print("=== traceback ===")
    print(operation.traceback)
    print("")
```

### コンソール出力

```
0
1
2
3
======================
=== name ===
counter/initial_value
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/initial_value"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 0
    }
  }
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 293, '_init_from_args', 'initial_value, name="initial_value", dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 657, 'convert_to_tensor', 'ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 180, '_constant_tensor_conversion_function', 'return constant(v, dtype=dtype, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter
=== type ===
Variable
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter"
op: "Variable"
attr {
  key: "container"
  value {
    s: ""
  }
}
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "shape"
  value {
    shape {
    }
  }
}
attr {
  key: "shared_name"
  value {
    s: ""
  }
}

=== op_def ===
name: "Variable"
output_arg {
  name: "ref"
  type_attr: "dtype"
  is_ref: true
}
attr {
  name: "shape"
  type: "shape"
}
attr {
  name: "dtype"
  type: "type"
}
attr {
  name: "container"
  type: "string"
  default_value {
    s: ""
  }
}
attr {
  name: "shared_name"
  type: "string"
  default_value {
    s: ""
  }
}
is_stateful: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 300, '_init_from_args', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/state_ops.py', 146, 'variable_op', 'container=container, shared_name=shared_name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 490, '_variable', 'name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("counter/initial_value:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("counter/Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== op_def ===
name: "Assign"
input_arg {
  name: "ref"
  type_attr: "T"
  is_ref: true
}
input_arg {
  name: "value"
  type_attr: "T"
}
output_arg {
  name: "output_ref"
  type_attr: "T"
  is_ref: true
}
attr {
  name: "T"
  type: "type"
}
attr {
  name: "validate_shape"
  type: "bool"
  default_value {
    b: true
  }
}
attr {
  name: "use_locking"
  type: "bool"
  default_value {
    b: true
  }
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 317, '_init_from_args', 'validate_shape=validate_shape).op'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
counter/read
=== type ===
Identity
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
=== control_inputs ===
=== outputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
=== node_def ===
name: "counter/read"
op: "Identity"
input: "counter"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}

=== op_def ===
name: "Identity"
input_arg {
  name: "input"
  type_attr: "T"
}
output_arg {
  name: "output"
  type_attr: "T"
}
attr {
  name: "T"
  type: "type"
}

=== traceback ===
[('./valiable.py', 5, '<module>', 'state = tf.Variable(0, name="counter")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 215, '__init__', 'dtype=dtype)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 327, '_init_from_args', 'self._snapshot = array_ops.identity(self._variable, name="read")'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_array_ops.py', 1128, 'identity', 'result = _op_def_lib.apply_op("Identity", input=input, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Const
=== type ===
Const
=== inputs ===
=== control_inputs ===
=== outputs ===
Tensor("Const:0", shape=(), dtype=int32)
=== node_def ===
name: "Const"
op: "Const"
attr {
  key: "dtype"
  value {
    type: DT_INT32
  }
}
attr {
  key: "value"
  value {
    tensor {
      dtype: DT_INT32
      tensor_shape {
      }
      int_val: 1
    }
  }
}

=== op_def ===
None
=== traceback ===
[('./valiable.py', 7, '<module>', 'one = tf.constant(1)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/constant_op.py', 167, 'constant', 'attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Add
=== type ===
Add
=== inputs ===
Tensor("counter/read:0", shape=(), dtype=int32)
Tensor("Const:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Add:0", shape=(), dtype=int32)
=== node_def ===
name: "Add"
op: "Add"
input: "counter/read"
input: "Const"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}

=== op_def ===
name: "Add"
input_arg {
  name: "x"
  type_attr: "T"
}
input_arg {
  name: "y"
  type_attr: "T"
}
output_arg {
  name: "z"
  type_attr: "T"
}
attr {
  name: "T"
  type: "type"
  allowed_values {
    list {
      type: DT_HALF
      type: DT_FLOAT
      type: DT_DOUBLE
      type: DT_UINT8
      type: DT_INT8
      type: DT_INT16
      type: DT_INT32
      type: DT_INT64
      type: DT_COMPLEX64
      type: DT_COMPLEX128
      type: DT_STRING
    }
  }
}

=== traceback ===
[('./valiable.py', 8, '<module>', 'new_value = tf.add(state, one)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_math_ops.py', 71, 'add', 'result = _op_def_lib.apply_op("Add", x=x, y=y, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
Assign
=== type ===
Assign
=== inputs ===
Tensor("counter:0", shape=(), dtype=int32_ref)
Tensor("Add:0", shape=(), dtype=int32)
=== control_inputs ===
=== outputs ===
Tensor("Assign:0", shape=(), dtype=int32_ref)
=== node_def ===
name: "Assign"
op: "Assign"
input: "counter"
input: "Add"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== op_def ===
name: "Assign"
input_arg {
  name: "ref"
  type_attr: "T"
  is_ref: true
}
input_arg {
  name: "value"
  type_attr: "T"
}
output_arg {
  name: "output_ref"
  type_attr: "T"
  is_ref: true
}
attr {
  name: "T"
  type: "type"
}
attr {
  name: "validate_shape"
  type: "bool"
  default_value {
    b: true
  }
}
attr {
  name: "use_locking"
  type: "bool"
  default_value {
    b: true
  }
}
allows_uninitialized_input: true

=== traceback ===
[('./valiable.py', 9, '<module>', 'update = tf.assign(state, new_value)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_state_ops.py', 45, 'assign', 'use_locking=use_locking, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 749, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]

======================
=== name ===
init
=== type ===
NoOp
=== inputs ===
=== control_inputs ===
name: "counter/Assign"
op: "Assign"
input: "counter"
input: "counter/initial_value"
attr {
  key: "T"
  value {
    type: DT_INT32
  }
}
attr {
  key: "_class"
  value {
    list {
      s: "loc:@counter"
    }
  }
}
attr {
  key: "use_locking"
  value {
    b: true
  }
}
attr {
  key: "validate_shape"
  value {
    b: true
  }
}

=== outputs ===
=== node_def ===
name: "init"
op: "NoOp"
input: "^counter/Assign"

=== op_def ===
name: "NoOp"

=== traceback ===
[('./valiable.py', 11, '<module>', 'init_op = tf.initialize_all_variables()'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1063, 'initialize_all_variables', 'return initialize_variables(all_variables())'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/variables.py', 1051, 'initialize_variables', 'return control_flow_ops.group(*[v.initializer for v in var_list], name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2645, 'group', 'return _GroupControlDeps(dev, deps, name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/control_flow_ops.py', 2603, '_GroupControlDeps', 'return no_op(name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/ops/gen_control_flow_ops.py', 184, 'no_op', 'result = _op_def_lib.apply_op("NoOp", name=name)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py', 756, 'apply_op', 'op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 2380, 'create_op', 'original_op=self._default_original_op, op_def=op_def)'), ('/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/ops.py', 1298, '__init__', 'self._traceback = _extract_stack()')]
```

### Protocol Buffersのベンチマーク
[Protocol Buffers が本当に遅いのか実際に確かめてみた](http://nojima.hatenablog.com/entry/2016/08/24/215002) @2016/8/24