Question: I have observed an issue while using the Hyperband algorithm in Optuna. According to the Hyperband algorithm, when min_resources = 5, max_resources = 20, and reduction_factor = 2, the search should start with an initial space of 4 models for bracket 1, with each model receiving 5 epochs in the first round. Subsequently, the number of models is reduced by a factor of 2 in each round and search space should also reduced by factor of 2 for next brackets i.e bracket 2 will have initial search space of 2 models, and the number of epochs for the remaining models is doubled in each subsequent round. so total models should be 11 is expected but it is training lot's of models. link of the article:- https://arxiv.org/pdf/1603.06560.pdf import optuna import numpy as np import pandas as pd from tensorflow.keras.layers import Dense,Flatten,Dropout import tensorflow as tf from tensorflow.keras.models import Sequential # Toy dataset generation def generate_toy_dataset(): np.random.seed(0) X_train = np.random.rand(100, 10) y_train = np.random.randint(0, 2, size=(100,)) X_val = np.random.rand(20, 10) y_val = np.random.randint(0, 2, size=(20,)) return X_train, y_train, X_val, y_val X_train, y_train, X_val, y_val = generate_toy_dataset() # Model building function def build_model(trial): model = Sequential() model.add(Dense(units=trial.suggest_int('unit_input', 20, 30), activation='selu', input_shape=(X_train.shape[1],))) num_layers = trial.suggest_int('num_layers', 2, 3) for i in range(num_layers): units = trial.suggest_int(f'num_layer_{i}', 20, 30) activation = trial.suggest_categorical(f'activation_layer_{i}', ['relu', 'selu', 'tanh']) model.add(Dense(units=units, activation=activation)) if trial.suggest_categorical(f'dropout_layer_{i}', [True, False]): model.add(Dropout(rate=0.5)) model.add(Dense(1, activation='sigmoid')) optimizer_name = trial.suggest_categorical('optimizer', ['adam', 'rmsprop']) if optimizer_name == 'adam': optimizer = tf.keras.optimizers.Adam() else: optimizer = tf.keras.optimizers.RMSprop() model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy', tf.keras.metrics.AUC(name='val_auc')]) return model def objective(trial): model = build_model(trial) # Assuming you have your data prepared # Modify the fit method to include AUC metric history = model.fit(X_train, y_train, validation_data=(X_val, y_val), verbose=1) # Check if 'val_auc' is recorded auc_key = None for key in history.history.keys(): if key.startswith('val_auc'): auc_key = key print(f"auc_key is {auc_key}") break if auc_key is None: raise ValueError("AUC metric not found in history. Make sure it's being recorded during training.") # Report validation AUC for each model if auc_key =="val_auc": step=0 else: step = int(auc_key.split('_')[-1]) auc_value=history.history[auc_key][0] trial.report(auc_value, step=step) print(f"prune or not:-{trial.should_prune()}") if trial.should_prune(): raise optuna.TrialPruned() return history.history[auc_key] # Optuna study creation study = optuna.create_study( direction='maximize', pruner=optuna.pruners.HyperbandPruner( min_resource=5, max_resource=20, reduction_factor=2 ) ) # Start optimization study.optimize(objective) 

Answer: You are using the default value of the parameter n_trials in the study.optimize function, which is None. According to the documentation, that means that it will stop evaluating configurations when it "times out". Optuna's Hyperband implementation is not identical to what was described in the original article. It has some tweaks to make the algorithm compatible with Optuna's inner workings. You can check the number of successive halving brackets like this: study.pruner._n_brackets. And you can check the allocated budget to each bracket like this: study.pruner._trial_allocation_budgets. What I am still trying to figure out is how the n_trials plays into defining the number of configurations that will be examined at each bracket.