Repository: keras-team/autokeras

Issue:
Specify hyperparameter ranges for blocks
### Feature Description
We want to enable the users to specify the value ranges for any argument in the blocks.
The following code example shows a typical use case.
The users can specify the number of units in a DenseBlock to be either 10 or 20.

### Code Example
<!---
Please provide a code example for using that feature
given the proposed feature is implemented.
-->
```python
import autokeras as ak
from kerastuner.engine.hyperparameters import Choice

input_node = ak.ImageInput()
output_node = ak.DenseBlock(num_units=Choice("num_units", [10, 20]))(input_node)
output_node = ak.ClassificationHead()(output_node)
model = ak.AutoModel(input_node, output_node)
```

### Note
<!---
Why do we need the feature?
-->
Each pull request should only change one hyperparameter in one of the blocks.

### Solution
<!---
Please tell us how to implement the feature,
if you have one in mind.
-->
Example pull requests are #1419 #1425 .
Here are the steps to follow:
1. You can just change any other argument in any other block supported by AutoKeras, as shown [here](https://github.com/keras-team/autokeras/blob/master/autokeras/__init__.py#L16-L38).
2. Change the docstring. [example](https://github.com/keras-team/autokeras/pull/1419/files#diff-7b757ddf51e45c0e02ad8de148d02fd2abd59d670f91d35cac71157e6684c13cL60-R61)
3. Make sure you imported the module. `from kerastuner.engine import hyperparameters`.
4. Change the typing of the argument. [example](https://github.com/keras-team/autokeras/pull/1419/files#diff-7b757ddf51e45c0e02ad8de148d02fd2abd59d670f91d35cac71157e6684c13cL73-R74)
5. Change the saving mechanism to serialized objects. [example](https://github.com/keras-team/autokeras/pull/1419/files#diff-7b757ddf51e45c0e02ad8de148d02fd2abd59d670f91d35cac71157e6684c13cL95-R100)
6. Change the loading mechanism to deserialized objects. [example](https://github.com/keras-team/autokeras/pull/1419/files#diff-7b757ddf51e45c0e02ad8de148d02fd2abd59d670f91d35cac71157e6684c13cR110)
7. Change how we initialize the hyperparameter to self. [example](https://github.com/keras-team/autokeras/pull/1419/files#diff-7b757ddf51e45c0e02ad8de148d02fd2abd59d670f91d35cac71157e6684c13cL80-R85) Copy from where it is originally defined. [example](https://github.com/keras-team/autokeras/pull/1419/files#diff-7b757ddf51e45c0e02ad8de148d02fd2abd59d670f91d35cac71157e6684c13cL115)
8. Change how we use it. [example](https://github.com/keras-team/autokeras/pull/1419/files#diff-7b757ddf51e45c0e02ad8de148d02fd2abd59d670f91d35cac71157e6684c13cL124-R129)


Discussion:
@haifeng-jin I would be happy to take this one
@lc0 Great! Thank you. Let me know if you have any questions.

Resolving patch:
diff --git a/autokeras/blocks/basic.py b/autokeras/blocks/basic.py
--- a/autokeras/blocks/basic.py
+++ b/autokeras/blocks/basic.py
@@ -65,7 +65,8 @@ class DenseBlock(block_module.Block):
             If left unspecified, it will be tuned automatically.
         use_bn: Boolean. Whether to use BatchNormalization layers.
             If left unspecified, it will be tuned automatically.
-        dropout: Float. The dropout rate for the layers.
+        dropout: Float or kerastuner.engine.hyperparameters.Choice.
+            The dropout rate for the layers.
             If left unspecified, it will be tuned automatically.
     """
 
@@ -74,7 +75,7 @@ def __init__(
         num_layers: Optional[Union[int, hyperparameters.Choice]] = None,
         num_units: Optional[Union[int, hyperparameters.Choice]] = None,
         use_batchnorm: Optional[bool] = None,
-        dropout: Optional[float] = None,
+        dropout: Optional[Union[float, hyperparameters.Choice]] = None,
         **kwargs,
     ):
         super().__init__(**kwargs)
@@ -91,7 +92,11 @@ def __init__(
             int,
         )
         self.use_batchnorm = use_batchnorm
-        self.dropout = dropout
+        self.dropout = utils.get_hyperparameter(
+            dropout,
+            hyperparameters.Choice("dropout", [0.0, 0.25, 0.5], default=0.0),
+            float,
+        )
 
     def get_config(self):
         config = super().get_config()
@@ -100,7 +105,7 @@ def get_config(self):
                 "num_layers": hyperparameters.serialize(self.num_layers),
                 "num_units": hyperparameters.serialize(self.num_units),
                 "use_batchnorm": self.use_batchnorm,
-                "dropout": self.dropout,
+                "dropout": hyperparameters.serialize(self.dropout),
             }
         )
         return config
@@ -109,6 +114,7 @@ def get_config(self):
     def from_config(cls, config):
         config["num_layers"] = hyperparameters.deserialize(config["num_layers"])
         config["num_units"] = hyperparameters.deserialize(config["num_units"])
+        config["dropout"] = hyperparameters.deserialize(config["dropout"])
         return cls(**config)
 
     def build(self, hp, inputs=None):
@@ -121,10 +127,6 @@ def build(self, hp, inputs=None):
         use_batchnorm = self.use_batchnorm
         if use_batchnorm is None:
             use_batchnorm = hp.Boolean("use_batchnorm", default=False)
-        if self.dropout is not None:
-            dropout = self.dropout
-        else:
-            dropout = hp.Choice("dropout", [0.0, 0.25, 0.5], default=0)
 
         for i in range(utils.add_to_hp(self.num_layers, hp)):
             units = utils.add_to_hp(self.num_units, hp, "units_{i}".format(i=i))
@@ -132,8 +134,10 @@ def build(self, hp, inputs=None):
             if use_batchnorm:
                 output_node = layers.BatchNormalization()(output_node)
             output_node = layers.ReLU()(output_node)
-            if dropout > 0:
-                output_node = layers.Dropout(dropout)(output_node)
+            if utils.add_to_hp(self.dropout, hp) > 0:
+                output_node = layers.Dropout(utils.add_to_hp(self.dropout, hp))(
+                    output_node
+                )
         return output_node
 
 


Test patch: