Repository: optuna/optuna

Issue:
Modify the simple multi-backend Keras example to use tf.keras
## Motivation

- Multi-backend Keras will be deprecated and it is recommended to use tf.keras
  - https://github.com/keras-team/keras#multi-backend-keras-and-tfkeras
- But, [`examples/keras_simple.py`](https://github.com/optuna/optuna/blob/82d23924e02a8a5fc82fac802544170ff04226d2/examples/keras_simple.py) currently use multi-backend Keras

## Description

I think we have two options:

1. Modify `examples/keras_simple.py`
2. Add a new example `examples/tfkeras_simple.py`

If we choose option 1, we have to additionally discuss which is better:
- Only edits `examples/keras_simple.py`
- Edits `examples/keras_simple.py` and rename it to `examples/tfkeras_simple.py`
  - [Pruning examples](https://github.com/optuna/optuna/tree/82d23924e02a8a5fc82fac802544170ff04226d2/examples/pruning) uses names `keras_integration.py` and `tfkeras_integration`
  - That is, for pruning examples, `keras` means multi-backend Keras and `tfkeras` means tf.keras



Discussion:

Resolving patch:
diff --git a/examples/tfkeras_simple.py b/examples/tfkeras_simple.py
new file mode 100644
--- /dev/null
+++ b/examples/tfkeras_simple.py
@@ -0,0 +1,87 @@
+"""
+Optuna example that optimizes a neural network classifier configuration for the
+MNIST dataset using tf.keras.
+
+In this example, we optimize the validation accuracy of MNIST classification using
+tf.keras. We optimize the filter and kernel size, kernel stride and layer activation.
+
+"""
+
+from tensorflow.keras.backend import clear_session
+from tensorflow.keras.datasets import mnist
+from tensorflow.keras.layers import Conv2D
+from tensorflow.keras.layers import Dense
+from tensorflow.keras.layers import Flatten
+from tensorflow.keras.models import Sequential
+from tensorflow.keras.optimizers import RMSprop
+
+import optuna
+
+
+N_TRAIN_EXAMPLES = 3000
+N_VALID_EXAMPLES = 1000
+BATCHSIZE = 128
+CLASSES = 10
+EPOCHS = 10
+
+
+def objective(trial):
+    # Clear clutter from previous tf.keras session graphs.
+    clear_session()
+
+    (x_train, y_train), (x_valid, y_valid) = mnist.load_data()
+    img_x, img_y = x_train.shape[1], x_train.shape[2]
+    x_train = x_train.reshape(-1, img_x, img_y, 1)[:N_TRAIN_EXAMPLES].astype("float32") / 255
+    x_valid = x_valid.reshape(-1, img_x, img_y, 1)[:N_VALID_EXAMPLES].astype("float32") / 255
+    y_train = y_train[:N_TRAIN_EXAMPLES]
+    y_valid = y_valid[:N_VALID_EXAMPLES]
+    input_shape = (img_x, img_y, 1)
+
+    model = Sequential()
+    model.add(
+        Conv2D(
+            filters=trial.suggest_categorical("filters", [32, 64]),
+            kernel_size=trial.suggest_categorical("kernel_size", [3, 5]),
+            strides=trial.suggest_categorical("strides", [1, 2]),
+            activation=trial.suggest_categorical("activation", ["relu", "linear"]),
+            input_shape=input_shape,
+        )
+    )
+    model.add(Flatten())
+    model.add(Dense(CLASSES, activation="softmax"))
+
+    # We compile our model with a sampled learning rate.
+    lr = trial.suggest_float("lr", 1e-5, 1e-1, log=True)
+    model.compile(
+        loss="sparse_categorical_crossentropy", optimizer=RMSprop(lr=lr), metrics=["accuracy"]
+    )
+
+    model.fit(
+        x_train,
+        y_train,
+        validation_data=(x_valid, y_valid),
+        shuffle=True,
+        batch_size=BATCHSIZE,
+        epochs=EPOCHS,
+        verbose=False,
+    )
+
+    # Evaluate the model accuracy on the validation set.
+    score = model.evaluate(x_valid, y_valid, verbose=0)
+    return score[1]
+
+
+if __name__ == "__main__":
+    study = optuna.create_study(direction="maximize")
+    study.optimize(objective, n_trials=100, timeout=600)
+
+    print("Number of finished trials: {}".format(len(study.trials)))
+
+    print("Best trial:")
+    trial = study.best_trial
+
+    print("  Value: {}".format(trial.value))
+
+    print("  Params: ")
+    for key, value in trial.params.items():
+        print("    {}: {}".format(key, value))


Test patch: