Repository: keras-team/autokeras

Issue:
"checkpoint not found" error in the "structured_data_classification" example
### Bug Description

I was trying to run the "structured_data_classification" example with the newly released version 1.0.3, but encountered this "checkpoint not found" error 

### Bug Reproduction
Code for reproducing the bug:

train_file_path = "data/train.csv"
test_file_path = "data/eval.csv"

x_train_df = pd.read_csv(train_file_path)
print(type(x_train_df)) # pandas.DataFrame
y_train_df = x_train_df.pop('survived')
print(type(y_train_df)) # pandas.Series

\# Preparing testing data.
x_test_df = pd.read_csv(test_file_path)
y_test_df = x_test_df.pop('survived')

\# It tries 3 different models.
clf = ak.StructuredDataClassifier(
    overwrite=True,
    max_trials=3,
    seed=66,
    project_name='song4',
    directory='akeras_models/'
)
\# Feed the structured data classifier with training data.
clf.fit(x_train_df, y_train_df) # , epochs=10)

\# Predict with the best model.
predicted_y = clf.predict(x_test_df)
\# Evaluate the best model with testing data.
print(clf.evaluate(x_test_df, y_test_df))

Data used by the code:

### Expected Behavior
2020-06-26 11:26:29.720935: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-06-26 11:26:29.720955: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
2020-06-26 11:26:29.720970: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (Super-AI): /proc/driver/nvidia/version does not exist
2020-06-26 11:26:29.721125: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-06-26 11:26:29.752555: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2900230000 Hz
2020-06-26 11:26:29.758760: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f82d4000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-26 11:26:29.758785: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
[Starting new trial]
Epoch 1/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7875 - accuracy: 0.5016/16 [==============================] - 0s 11ms/step - loss: 0.7533 - accuracy: 0.5977 - val_loss: 0.6331 - val_accuracy: 0.7043
Epoch 2/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7675 - accuracy: 0.6516/16 [==============================] - 0s 3ms/step - loss: 0.6475 - accuracy: 0.6855 - val_loss: 0.7478 - val_accuracy: 0.4261
Epoch 3/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.8791 - accuracy: 0.5016/16 [==============================] - 0s 3ms/step - loss: 0.6380 - accuracy: 0.6914 - val_loss: 0.8299 - val_accuracy: 0.3304
Epoch 4/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7965 - accuracy: 0.6816/16 [==============================] - 0s 3ms/step - loss: 0.6334 - accuracy: 0.7285 - val_loss: 0.8162 - val_accuracy: 0.3565
Epoch 5/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6657 - accuracy: 0.6516/16 [==============================] - 0s 3ms/step - loss: 0.5991 - accuracy: 0.7344 - val_loss: 0.7410 - val_accuracy: 0.3913
Epoch 6/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6547 - accuracy: 0.6516/16 [==============================] - 0s 3ms/step - loss: 0.5796 - accuracy: 0.7285 - val_loss: 0.7873 - val_accuracy: 0.3826
Epoch 7/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5260 - accuracy: 0.7516/16 [==============================] - 0s 3ms/step - loss: 0.5761 - accuracy: 0.7266 - val_loss: 0.8332 - val_accuracy: 0.3652
Epoch 8/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7437 - accuracy: 0.6816/16 [==============================] - 0s 3ms/step - loss: 0.5543 - accuracy: 0.7500 - val_loss: 0.7986 - val_accuracy: 0.4000
Epoch 9/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4914 - accuracy: 0.7516/16 [==============================] - 0s 3ms/step - loss: 0.5277 - accuracy: 0.7734 - val_loss: 0.7369 - val_accuracy: 0.4087
Epoch 10/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6530 - accuracy: 0.7116/16 [==============================] - 0s 3ms/step - loss: 0.5629 - accuracy: 0.7520 - val_loss: 0.6682 - val_accuracy: 0.5391
Epoch 11/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5999 - accuracy: 0.7816/16 [==============================] - 0s 3ms/step - loss: 0.5369 - accuracy: 0.7676 - val_loss: 0.7082 - val_accuracy: 0.5478
[Trial complete]
[Trial summary]
 |-Trial ID: f3e9fec44a0956276a01421772bcb618
 |-Score: 0.7043478488922119
 |-Best step: 0
 > Hyperparameters:
 |-classification_head_1/dropout_rate: 0
 |-optimizer: adam
 |-structured_data_block_1/dense_block_1/dropout_rate: 0.5
 |-structured_data_block_1/dense_block_1/num_layers: 1
 |-structured_data_block_1/dense_block_1/units_0: 512
 |-structured_data_block_1/dense_block_1/units_1: 16
 |-structured_data_block_1/dense_block_1/use_batchnorm: True
[Starting new trial]
Epoch 1/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.9774 - accuracy: 0.5016/16 [==============================] - 0s 10ms/step - loss: 0.8073 - accuracy: 0.5391 - val_loss: 1.0624 - val_accuracy: 0.4870
Epoch 2/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.9896 - accuracy: 0.4316/16 [==============================] - 0s 5ms/step - loss: 0.6816 - accuracy: 0.6426 - val_loss: 0.6217 - val_accuracy: 0.7130
Epoch 3/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6186 - accuracy: 0.6816/16 [==============================] - 0s 5ms/step - loss: 0.5884 - accuracy: 0.7207 - val_loss: 0.6258 - val_accuracy: 0.7043
Epoch 4/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6960 - accuracy: 0.5616/16 [==============================] - 0s 4ms/step - loss: 0.5688 - accuracy: 0.7266 - val_loss: 0.6487 - val_accuracy: 0.6522
Epoch 5/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4751 - accuracy: 0.7516/16 [==============================] - 0s 4ms/step - loss: 0.5353 - accuracy: 0.7422 - val_loss: 0.7280 - val_accuracy: 0.4087
Epoch 6/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6062 - accuracy: 0.5616/16 [==============================] - 0s 5ms/step - loss: 0.5268 - accuracy: 0.7461 - val_loss: 0.7232 - val_accuracy: 0.6348
Epoch 7/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4782 - accuracy: 0.6516/16 [==============================] - 0s 5ms/step - loss: 0.5273 - accuracy: 0.7559 - val_loss: 0.6015 - val_accuracy: 0.7130
Epoch 8/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4447 - accuracy: 0.8116/16 [==============================] - 0s 4ms/step - loss: 0.5104 - accuracy: 0.7676 - val_loss: 0.6554 - val_accuracy: 0.6435
Epoch 9/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4685 - accuracy: 0.8716/16 [==============================] - 0s 4ms/step - loss: 0.5135 - accuracy: 0.7812 - val_loss: 0.5953 - val_accuracy: 0.7391
Epoch 10/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4235 - accuracy: 0.8716/16 [==============================] - 0s 4ms/step - loss: 0.4754 - accuracy: 0.7871 - val_loss: 0.6090 - val_accuracy: 0.7130
Epoch 11/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7253 - accuracy: 0.7816/16 [==============================] - 0s 4ms/step - loss: 0.4961 - accuracy: 0.7734 - val_loss: 0.5601 - val_accuracy: 0.7391
Epoch 12/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4596 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.4459 - accuracy: 0.8008 - val_loss: 0.5370 - val_accuracy: 0.7826
Epoch 13/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5129 - accuracy: 0.7816/16 [==============================] - 0s 5ms/step - loss: 0.4602 - accuracy: 0.8066 - val_loss: 0.4664 - val_accuracy: 0.8087
Epoch 14/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4598 - accuracy: 0.8116/16 [==============================] - 0s 4ms/step - loss: 0.4366 - accuracy: 0.8281 - val_loss: 0.4810 - val_accuracy: 0.7913
Epoch 15/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6860 - accuracy: 0.7816/16 [==============================] - 0s 4ms/step - loss: 0.4841 - accuracy: 0.8027 - val_loss: 0.4353 - val_accuracy: 0.8174
Epoch 16/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4164 - accuracy: 0.7816/16 [==============================] - 0s 5ms/step - loss: 0.4539 - accuracy: 0.8086 - val_loss: 0.4668 - val_accuracy: 0.7913
Epoch 17/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5218 - accuracy: 0.6816/16 [==============================] - 0s 5ms/step - loss: 0.4484 - accuracy: 0.7910 - val_loss: 0.4026 - val_accuracy: 0.8174
Epoch 18/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4353 - accuracy: 0.8716/16 [==============================] - 0s 4ms/step - loss: 0.4409 - accuracy: 0.7969 - val_loss: 0.4444 - val_accuracy: 0.8087
Epoch 19/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5144 - accuracy: 0.6516/16 [==============================] - 0s 5ms/step - loss: 0.4730 - accuracy: 0.7754 - val_loss: 0.4585 - val_accuracy: 0.8087
Epoch 20/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4510 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.4625 - accuracy: 0.7969 - val_loss: 0.4528 - val_accuracy: 0.8435
Epoch 21/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5078 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.4495 - accuracy: 0.7910 - val_loss: 0.4037 - val_accuracy: 0.8348
Epoch 22/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4735 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.4015 - accuracy: 0.8242 - val_loss: 0.4102 - val_accuracy: 0.8087
Epoch 23/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4540 - accuracy: 0.7816/16 [==============================] - 0s 5ms/step - loss: 0.4340 - accuracy: 0.8047 - val_loss: 0.3913 - val_accuracy: 0.8348
Epoch 24/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5882 - accuracy: 0.7116/16 [==============================] - 0s 4ms/step - loss: 0.4257 - accuracy: 0.8125 - val_loss: 0.3854 - val_accuracy: 0.8435
Epoch 25/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4903 - accuracy: 0.7816/16 [==============================] - 0s 5ms/step - loss: 0.4263 - accuracy: 0.8242 - val_loss: 0.4097 - val_accuracy: 0.8261
Epoch 26/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5059 - accuracy: 0.8416/16 [==============================] - 0s 4ms/step - loss: 0.4068 - accuracy: 0.8203 - val_loss: 0.4205 - val_accuracy: 0.8174
Epoch 27/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.6119 - accuracy: 0.7516/16 [==============================] - 0s 4ms/step - loss: 0.4279 - accuracy: 0.8086 - val_loss: 0.3904 - val_accuracy: 0.8348
Epoch 28/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4844 - accuracy: 0.7816/16 [==============================] - 0s 4ms/step - loss: 0.4052 - accuracy: 0.8398 - val_loss: 0.4189 - val_accuracy: 0.8174
Epoch 29/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5712 - accuracy: 0.6516/16 [==============================] - 0s 4ms/step - loss: 0.4195 - accuracy: 0.8203 - val_loss: 0.4423 - val_accuracy: 0.8348
Epoch 30/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.3971 - accuracy: 0.8716/16 [==============================] - 0s 4ms/step - loss: 0.4081 - accuracy: 0.8340 - val_loss: 0.3681 - val_accuracy: 0.8522
Epoch 31/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4551 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.4357 - accuracy: 0.8145 - val_loss: 0.4294 - val_accuracy: 0.8696
Epoch 32/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4730 - accuracy: 0.7816/16 [==============================] - 0s 4ms/step - loss: 0.4254 - accuracy: 0.8184 - val_loss: 0.4023 - val_accuracy: 0.8174
Epoch 33/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4941 - accuracy: 0.7816/16 [==============================] - 0s 4ms/step - loss: 0.4452 - accuracy: 0.8145 - val_loss: 0.3756 - val_accuracy: 0.8261
Epoch 34/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5311 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.4210 - accuracy: 0.8164 - val_loss: 0.4062 - val_accuracy: 0.8087
Epoch 35/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4841 - accuracy: 0.7116/16 [==============================] - 0s 5ms/step - loss: 0.4034 - accuracy: 0.8105 - val_loss: 0.3638 - val_accuracy: 0.8348
Epoch 36/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4511 - accuracy: 0.8416/16 [==============================] - 0s 4ms/step - loss: 0.4177 - accuracy: 0.8242 - val_loss: 0.4077 - val_accuracy: 0.8522
Epoch 37/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5379 - accuracy: 0.8116/16 [==============================] - 0s 4ms/step - loss: 0.4063 - accuracy: 0.8262 - val_loss: 0.3705 - val_accuracy: 0.8348
Epoch 38/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4644 - accuracy: 0.8416/16 [==============================] - 0s 4ms/step - loss: 0.4079 - accuracy: 0.8359 - val_loss: 0.3600 - val_accuracy: 0.8696
Epoch 39/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4944 - accuracy: 0.7516/16 [==============================] - 0s 4ms/step - loss: 0.4129 - accuracy: 0.8457 - val_loss: 0.3446 - val_accuracy: 0.8609
Epoch 40/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4056 - accuracy: 0.8116/16 [==============================] - 0s 4ms/step - loss: 0.3861 - accuracy: 0.8262 - val_loss: 0.3640 - val_accuracy: 0.8696
Epoch 41/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4578 - accuracy: 0.8716/16 [==============================] - 0s 5ms/step - loss: 0.4050 - accuracy: 0.8262 - val_loss: 0.3439 - val_accuracy: 0.8696
Epoch 42/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.5131 - accuracy: 0.8116/16 [==============================] - 0s 4ms/step - loss: 0.3956 - accuracy: 0.8281 - val_loss: 0.3349 - val_accuracy: 0.8609
Epoch 43/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4367 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.3686 - accuracy: 0.8340 - val_loss: 0.3352 - val_accuracy: 0.8435
Epoch 44/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4487 - accuracy: 0.8716/16 [==============================] - 0s 4ms/step - loss: 0.3879 - accuracy: 0.8281 - val_loss: 0.3221 - val_accuracy: 0.8783
Epoch 45/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4602 - accuracy: 0.8416/16 [==============================] - 0s 5ms/step - loss: 0.3942 - accuracy: 0.8418 - val_loss: 0.3461 - val_accuracy: 0.8609
Epoch 46/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4294 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.3849 - accuracy: 0.8320 - val_loss: 0.3527 - val_accuracy: 0.8609
Epoch 47/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4323 - accuracy: 0.8416/16 [==============================] - 0s 5ms/step - loss: 0.3939 - accuracy: 0.8203 - val_loss: 0.3732 - val_accuracy: 0.8261
Epoch 48/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4246 - accuracy: 0.8416/16 [==============================] - 0s 4ms/step - loss: 0.3832 - accuracy: 0.8496 - val_loss: 0.4042 - val_accuracy: 0.8696
Epoch 49/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.3794 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.3627 - accuracy: 0.8398 - val_loss: 0.3698 - val_accuracy: 0.8609
Epoch 50/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4711 - accuracy: 0.7816/16 [==============================] - 0s 4ms/step - loss: 0.3887 - accuracy: 0.8242 - val_loss: 0.3443 - val_accuracy: 0.8609
Epoch 51/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.3824 - accuracy: 0.8116/16 [==============================] - 0s 4ms/step - loss: 0.3925 - accuracy: 0.8340 - val_loss: 0.4368 - val_accuracy: 0.8261
Epoch 52/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4646 - accuracy: 0.8116/16 [==============================] - 0s 5ms/step - loss: 0.3852 - accuracy: 0.8438 - val_loss: 0.4202 - val_accuracy: 0.8087
Epoch 53/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4349 - accuracy: 0.8716/16 [==============================] - 0s 5ms/step - loss: 0.3697 - accuracy: 0.8398 - val_loss: 0.4056 - val_accuracy: 0.8261
Epoch 54/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.4676 - accuracy: 0.8416/16 [==============================] - 0s 5ms/step - loss: 0.3724 - accuracy: 0.8379 - val_loss: 0.3753 - val_accuracy: 0.8261
[Trial complete]
[Trial summary]
 |-Trial ID: 05b23cac10d5f471c6d77d73f3c7000e
 |-Score: 0.8782608509063721
 |-Best step: 43
 > Hyperparameters:
 |-classification_head_1/dropout_rate: 0
 |-optimizer: adam
 |-structured_data_block_1/dense_block_1/dropout_rate: 0.25
 |-structured_data_block_1/dense_block_1/num_layers: 2
 |-structured_data_block_1/dense_block_1/units_0: 1024
 |-structured_data_block_1/dense_block_1/units_1: 128
 |-structured_data_block_1/dense_block_1/use_batchnorm: True
[Starting new trial]
Epoch 1/1000
 1/16 [>.............................] - ETA: 0s - loss: 2.1611 - accuracy: 0.5316/16 [==============================] - 0s 6ms/step - loss: 0.9218 - accuracy: 0.6270 - val_loss: 0.5265 - val_accuracy: 0.7043
Epoch 2/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.8769 - accuracy: 0.4616/16 [==============================] - 0s 2ms/step - loss: 0.7397 - accuracy: 0.6289 - val_loss: 0.6304 - val_accuracy: 0.7043
Epoch 3/1000
 1/16 [>.............................] - ETA: 0s - loss: 1.0053 - accuracy: 0.4616/16 [==============================] - 0s 2ms/step - loss: 0.7013 - accuracy: 0.6152 - val_loss: 0.5581 - val_accuracy: 0.7304
Epoch 4/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7856 - accuracy: 0.5916/16 [==============================] - 0s 2ms/step - loss: 0.6441 - accuracy: 0.6621 - val_loss: 0.5394 - val_accuracy: 0.7478
Epoch 5/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7105 - accuracy: 0.5316/16 [==============================] - 0s 2ms/step - loss: 0.6350 - accuracy: 0.6758 - val_loss: 0.5434 - val_accuracy: 0.7304
Epoch 6/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7370 - accuracy: 0.5616/16 [==============================] - 0s 2ms/step - loss: 0.6494 - accuracy: 0.6699 - val_loss: 0.5633 - val_accuracy: 0.7130
Epoch 7/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7826 - accuracy: 0.5916/16 [==============================] - 0s 2ms/step - loss: 0.6480 - accuracy: 0.6797 - val_loss: 0.5581 - val_accuracy: 0.7130
Epoch 8/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7585 - accuracy: 0.5616/16 [==============================] - 0s 2ms/step - loss: 0.6350 - accuracy: 0.6797 - val_loss: 0.5479 - val_accuracy: 0.7217
Epoch 9/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7414 - accuracy: 0.5616/16 [==============================] - 0s 2ms/step - loss: 0.6270 - accuracy: 0.6816 - val_loss: 0.5461 - val_accuracy: 0.7130
Epoch 10/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7368 - accuracy: 0.5616/16 [==============================] - 0s 2ms/step - loss: 0.6234 - accuracy: 0.6895 - val_loss: 0.5405 - val_accuracy: 0.7652
Epoch 11/1000
 1/16 [>.............................] - ETA: 0s - loss: 0.7234 - accuracy: 0.6216/16 [==============================] - 0s 2ms/step - loss: 0.6178 - accuracy: 0.6934 - val_loss: 0.5374 - val_accuracy: 0.7652
[Trial complete]
[Trial summary]
 |-Trial ID: c0ef66df3ffa96c10e2e2bae85fc9e26
 |-Score: 0.7652173638343811
 |-Best step: 9
 > Hyperparameters:
 |-classification_head_1/dropout_rate: 0
 |-optimizer: adam
 |-structured_data_block_1/dense_block_1/dropout_rate: 0.0
 |-structured_data_block_1/dense_block_1/num_layers: 2
 |-structured_data_block_1/dense_block_1/units_0: 32
 |-structured_data_block_1/dense_block_1/units_1: 64
 |-structured_data_block_1/dense_block_1/use_batchnorm: False
Epoch 1/54
 1/20 [>.............................] - ETA: 0s - loss: 0.7088 - accuracy: 0.5317/20 [========================>.....] - ETA: 0s - loss: 0.6610 - accuracy: 0.6420/20 [==============================] - 0s 3ms/step - loss: 0.6477 - accuracy: 0.6539
Epoch 2/54
 1/20 [>.............................] - ETA: 0s - loss: 0.7955 - accuracy: 0.5918/20 [==========================>...] - ETA: 0s - loss: 0.6055 - accuracy: 0.6920/20 [==============================] - 0s 3ms/step - loss: 0.5937 - accuracy: 0.7033
Epoch 3/54
 1/20 [>.............................] - ETA: 0s - loss: 0.7455 - accuracy: 0.6217/20 [========================>.....] - ETA: 0s - loss: 0.5864 - accuracy: 0.7220/20 [==============================] - 0s 3ms/step - loss: 0.5717 - accuracy: 0.7352
Epoch 4/54
 1/20 [>.............................] - ETA: 0s - loss: 0.7916 - accuracy: 0.5618/20 [==========================>...] - ETA: 0s - loss: 0.5190 - accuracy: 0.7220/20 [==============================] - 0s 3ms/step - loss: 0.5068 - accuracy: 0.7384
Epoch 5/54
 1/20 [>.............................] - ETA: 0s - loss: 0.7673 - accuracy: 0.6818/20 [==========================>...] - ETA: 0s - loss: 0.5247 - accuracy: 0.7420/20 [==============================] - 0s 3ms/step - loss: 0.5110 - accuracy: 0.7544
Epoch 6/54
 1/20 [>.............................] - ETA: 0s - loss: 0.6668 - accuracy: 0.6818/20 [==========================>...] - ETA: 0s - loss: 0.5063 - accuracy: 0.7620/20 [==============================] - 0s 3ms/step - loss: 0.4923 - accuracy: 0.7719
Epoch 7/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5890 - accuracy: 0.6818/20 [==========================>...] - ETA: 0s - loss: 0.5177 - accuracy: 0.7620/20 [==============================] - 0s 3ms/step - loss: 0.5031 - accuracy: 0.7767
Epoch 8/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4941 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4566 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4423 - accuracy: 0.8054
Epoch 9/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5542 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4922 - accuracy: 0.7820/20 [==============================] - 0s 3ms/step - loss: 0.4780 - accuracy: 0.7879
Epoch 10/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5366 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4773 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4616 - accuracy: 0.8054
Epoch 11/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5197 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4724 - accuracy: 0.8020/20 [==============================] - 0s 3ms/step - loss: 0.4572 - accuracy: 0.8150
Epoch 12/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5889 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4865 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4692 - accuracy: 0.8006
Epoch 13/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4531 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4594 - accuracy: 0.7720/20 [==============================] - 0s 3ms/step - loss: 0.4458 - accuracy: 0.7847
Epoch 14/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4767 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4649 - accuracy: 0.7820/20 [==============================] - 0s 3ms/step - loss: 0.4464 - accuracy: 0.7959
Epoch 15/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5535 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4507 - accuracy: 0.8020/20 [==============================] - 0s 3ms/step - loss: 0.4358 - accuracy: 0.8102
Epoch 16/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5714 - accuracy: 0.7819/20 [===========================>..] - ETA: 0s - loss: 0.4497 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4420 - accuracy: 0.8022
Epoch 17/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5527 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4730 - accuracy: 0.8020/20 [==============================] - 0s 3ms/step - loss: 0.4567 - accuracy: 0.8134
Epoch 18/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4463 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4503 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4341 - accuracy: 0.8054
Epoch 19/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5689 - accuracy: 0.7519/20 [===========================>..] - ETA: 0s - loss: 0.4518 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4424 - accuracy: 0.7959
Epoch 20/54
 1/20 [>.............................] - ETA: 0s - loss: 0.6748 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4583 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4414 - accuracy: 0.8022
Epoch 21/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5214 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4366 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4210 - accuracy: 0.8214
Epoch 22/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5192 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4446 - accuracy: 0.8020/20 [==============================] - 0s 3ms/step - loss: 0.4345 - accuracy: 0.8102
Epoch 23/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4957 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4292 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4153 - accuracy: 0.8214
Epoch 24/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5285 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4391 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4238 - accuracy: 0.8262
Epoch 25/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5750 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4353 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4227 - accuracy: 0.8214
Epoch 26/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5240 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4439 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4325 - accuracy: 0.8214
Epoch 27/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5631 - accuracy: 0.7118/20 [==========================>...] - ETA: 0s - loss: 0.4357 - accuracy: 0.7920/20 [==============================] - 0s 3ms/step - loss: 0.4200 - accuracy: 0.8054
Epoch 28/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5509 - accuracy: 0.8418/20 [==========================>...] - ETA: 0s - loss: 0.4116 - accuracy: 0.8220/20 [==============================] - 0s 3ms/step - loss: 0.3992 - accuracy: 0.8309
Epoch 29/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5300 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4295 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4145 - accuracy: 0.8246
Epoch 30/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5644 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4213 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4067 - accuracy: 0.8214
Epoch 31/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4828 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4098 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.3952 - accuracy: 0.8230
Epoch 32/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4392 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4129 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.3986 - accuracy: 0.8198
Epoch 33/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4751 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.4084 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.3929 - accuracy: 0.8262
Epoch 34/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5702 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4053 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3921 - accuracy: 0.8405
Epoch 35/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5798 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.4133 - accuracy: 0.8220/20 [==============================] - 0s 3ms/step - loss: 0.3983 - accuracy: 0.8325
Epoch 36/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5477 - accuracy: 0.7519/20 [===========================>..] - ETA: 0s - loss: 0.4206 - accuracy: 0.8120/20 [==============================] - 0s 3ms/step - loss: 0.4111 - accuracy: 0.8214
Epoch 37/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4746 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.3898 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3768 - accuracy: 0.8389
Epoch 38/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4018 - accuracy: 0.8418/20 [==========================>...] - ETA: 0s - loss: 0.3853 - accuracy: 0.8220/20 [==============================] - 0s 3ms/step - loss: 0.3733 - accuracy: 0.8357
Epoch 39/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4808 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4058 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3945 - accuracy: 0.8373
Epoch 40/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5848 - accuracy: 0.7518/20 [==========================>...] - ETA: 0s - loss: 0.3917 - accuracy: 0.8220/20 [==============================] - 0s 3ms/step - loss: 0.3813 - accuracy: 0.8309
Epoch 41/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4457 - accuracy: 0.7820/20 [==============================] - ETA: 0s - loss: 0.3719 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3719 - accuracy: 0.8373
Epoch 42/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4841 - accuracy: 0.7819/20 [===========================>..] - ETA: 0s - loss: 0.3947 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3856 - accuracy: 0.8405
Epoch 43/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4550 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.4005 - accuracy: 0.8220/20 [==============================] - 0s 3ms/step - loss: 0.3844 - accuracy: 0.8373
Epoch 44/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4141 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.3850 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3735 - accuracy: 0.8437
Epoch 45/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5307 - accuracy: 0.7819/20 [===========================>..] - ETA: 0s - loss: 0.3839 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3754 - accuracy: 0.8357
Epoch 46/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4849 - accuracy: 0.7519/20 [===========================>..] - ETA: 0s - loss: 0.3710 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3630 - accuracy: 0.8437
Epoch 47/54
 1/20 [>.............................] - ETA: 0s - loss: 0.3873 - accuracy: 0.8419/20 [===========================>..] - ETA: 0s - loss: 0.3604 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3521 - accuracy: 0.8437
Epoch 48/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4314 - accuracy: 0.7819/20 [===========================>..] - ETA: 0s - loss: 0.3736 - accuracy: 0.8520/20 [==============================] - 0s 3ms/step - loss: 0.3650 - accuracy: 0.8565
Epoch 49/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4029 - accuracy: 0.8718/20 [==========================>...] - ETA: 0s - loss: 0.3775 - accuracy: 0.8220/20 [==============================] - 0s 3ms/step - loss: 0.3652 - accuracy: 0.8357
Epoch 50/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4561 - accuracy: 0.8118/20 [==========================>...] - ETA: 0s - loss: 0.3712 - accuracy: 0.8520/20 [==============================] - 0s 3ms/step - loss: 0.3590 - accuracy: 0.8581
Epoch 51/54
 1/20 [>.............................] - ETA: 0s - loss: 0.4574 - accuracy: 0.7118/20 [==========================>...] - ETA: 0s - loss: 0.3812 - accuracy: 0.8220/20 [==============================] - 0s 3ms/step - loss: 0.3678 - accuracy: 0.8357
Epoch 52/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5078 - accuracy: 0.8718/20 [==========================>...] - ETA: 0s - loss: 0.4007 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3848 - accuracy: 0.8437
Epoch 53/54
 1/20 [>.............................] - ETA: 0s - loss: 0.5683 - accuracy: 0.7818/20 [==========================>...] - ETA: 0s - loss: 0.3683 - accuracy: 0.8320/20 [==============================] - 0s 3ms/step - loss: 0.3588 - accuracy: 0.8437
Epoch 54/54
 1/20 [>.............................] - ETA: 0s - loss: 0.3632 - accuracy: 0.8418/20 [==========================>...] - ETA: 0s - loss: 0.3854 - accuracy: 0.8420/20 [==============================] - 0s 3ms/step - loss: 0.3714 - accuracy: 0.8517
Traceback (most recent call last):
  File "structured_data_classification.py", line 37, in <module>
    predicted_y = clf.predict(x_test_df)
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/autokeras/tasks/structured_data.py", line 114, in predict
    **kwargs)
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/autokeras/auto_model.py", line 414, in predict
    model = self.tuner.get_best_model()
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/autokeras/engine/tuner.py", line 49, in get_best_model
    model = super().get_best_models()[0]
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/kerastuner/engine/tuner.py", line 258, in get_best_models
    return super(Tuner, self).get_best_models(num_models)
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/kerastuner/engine/base_tuner.py", line 241, in get_best_models
    models = [self.load_model(trial) for trial in best_trials]
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/kerastuner/engine/base_tuner.py", line 241, in <listcomp>
    models = [self.load_model(trial) for trial in best_trials]
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/kerastuner/engine/tuner.py", line 184, in load_model
    trial.trial_id, best_epoch))
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
    return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow/python/keras/engine/network.py", line 1231, in load_weights
    py_checkpoint_reader.NewCheckpointReader(filepath)
  File "/home/gsong/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow/python/training/py_checkpoint_reader.py", line 95, in NewCheckpointReader
    return CheckpointReader(compat.as_bytes(filepattern))
ValueError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on akeras_models/song4/trial_05b23cac10d5f471c6d77d73f3c7000e/checkpoints/epoch_43/checkpoint: Not found: akeras_models/song4/trial_05b23cac10d5f471c6d77d73f3c7000e/checkpoints/epoch_43; No such file or directory


### Setup Details
Include the details about the versions of:
 - OS type and version: Unbuntu 18.04
 - Python: 3.6
 - autokeras: 1.0.3
 - keras-tuner: 1.0.2rc0
 - scikit-learn:
 - numpy:
 - pandas:
 - tensorflow: 2.2.0 no GPU used

### Additional context
<!---
If applicable, add any other context about the problem.
-->



Discussion:
In trial_05b23cac10d5f471c6d77d73f3c7000e/checkpoints/, there are the following checkpoints:
epoch_0
epoch_44
epoch_45
...
epoch_53

BUT, no epoch43
Same with me. On MacOS.
Currently, I can only reproduce it by not setting `overwrite` which by default is set to False.
It is caused by not properly saving the oracle and tuner.
The linked PR solves this problem.
Will be merged soon.
Hi there,
I installed the current version of the master with `pip install git+https://github.com/keras-team/autokeras.git@master`. However, i still get the same error.


    OS type and version: Windows 10
    Python: 3.7
    autokeras: 1.0.4.dev0
    keras-tuner: 1.0.2rc0
    tensorflow: 2.2.0 no GPU used


> Hi there,
> I installed the current version of the master with `pip install git+https://github.com/keras-team/autokeras.git@master`. However, i still get the same error.
> 
> ```
> OS type and version: Windows 10
> Python: 3.7
> autokeras: 1.0.4.dev0
> keras-tuner: 1.0.2rc0
> tensorflow: 2.2.0 no GPU used
> ```

I have the same issue as well!
I really cannot reproduce the error. It would be great if you can share a colab notebook. Thank you!
> I really cannot reproduce the error. It would be great if you can share a colab notebook. Thank you!

https://colab.research.google.com/drive/1H-i0Y6QZRGyvNaprHHSUoeHRDO6z9O59?usp=sharing
@mreismendes Thank you so much! will get to it right away.

Resolving patch:
diff --git a/autokeras/engine/tuner.py b/autokeras/engine/tuner.py
--- a/autokeras/engine/tuner.py
+++ b/autokeras/engine/tuner.py
@@ -47,7 +47,7 @@ def _populate_initial_space(self):
         pass
 
     def get_best_model(self):
-        model = super().get_best_models()[0]
+        model = self._build_best_model()
         model.load_weights(self.best_model_path)
         return model
 
@@ -174,10 +174,13 @@ def _get_best_trial_epochs(self):
         best_trial = self.oracle.get_best_trials(1)[0]
         return len(best_trial.metrics.metrics['val_loss']._observations)
 
-    def final_fit(self, x=None, **fit_kwargs):
+    def _build_best_model(self):
         best_trial = self.oracle.get_best_trials(1)[0]
         best_hp = best_trial.hyperparameters
-        model = self.hypermodel.build(best_hp)
+        return self.hypermodel.build(best_hp)
+
+    def final_fit(self, x=None, **fit_kwargs):
+        model = self._build_best_model()
         self.adapt(model, x)
         model.fit(x, **fit_kwargs)
         return model


Test patch: