Repository: keras-team/keras

Issue:
ThresholdedReLU crashes when the input is a list
**System information**.
- Have I written custom code (as opposed to using a stock example script provided in Keras): Yes
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu
- TensorFlow installed from (source or binary): binary
- TensorFlow version (use command below): 2.8.0
- Python version: 3.7.12
- Bazel version (if compiling from source): N/A
- GPU model and memory: N/A
- Exact command to reproduce: https://colab.research.google.com/drive/144FOk8RO-Ew_eBtGZlCmsUL6k0cEAlUO?usp=sharing

**Describe the problem**.
`keras.layers.ThresholdedReLU` fails to accept a list input by reporting the following error:

```
[/usr/local/lib/python3.7/dist-packages/keras/layers/advanced_activations.py](https://localhost:8080/#) in call(self, inputs)
    262 
    263   def call(self, inputs):
--> 264     theta = tf.cast(self.theta, inputs.dtype)
    265     return inputs * tf.cast(tf.greater(inputs, theta), inputs.dtype)
    266 

AttributeError: Exception encountered when calling layer "thresholded_re_lu_1" (type ThresholdedReLU).

'list' object has no attribute 'dtype'

Call arguments received:
  • inputs=['tf.Tensor(shape=(None, 1, 10), dtype=float32)', 'tf.Tensor(shape=(None, 1, 10), dtype=float32)', 'tf.Tensor(shape=(None, 1, 10), dtype=float32)']
```
In contrast, `keras.layers.ReLU` and `keras.layers.LeakyReLU` can accept the list input.

**Describe the current behavior**.
`keras.layers.ThresholdedReLU` crashes when the input is a list

**Describe the expected behavior**.
ThresholdedReLU can accept the list input.

**[Contributing](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md)**.

- Do you want to contribute a PR? (yes/no):
- If yes, please read [this page](https://github.com/keras-team/keras/blob/master/CONTRIBUTING.md) for instructions
- Briefly describe your candidate solution(if contributing):

After comparing the code between `ThresholedReLU` and `ReLU`, I think the reason is that `ReLU` directly use the backend implementation: [keras/layers/activation/relu.py#L96](https://github.com/keras-team/keras/blob/90aa76f6c48ec5252b2db7926ff060e262dfc1cc/keras/layers/activation/relu.py#L96) while ThresholdedReLU implements by itself: [keras/layers/activation/thresholded_relu.py#L63](https://github.com/keras-team/keras/blob/90aa76f6c48ec5252b2db7926ff060e262dfc1cc/keras/layers/activation/thresholded_relu.py#L63). Not sure why does such an implementation inconsistency exist, but I think we can do something similar in the thresholded_relu.py#L61-63 like [backend.relu](https://github.com/keras-team/keras/blob/90aa76f6c48ec5252b2db7926ff060e262dfc1cc/keras/backend.py#L4963) does:

```
def call(self, inputs):
    dtype = getattr(inputs, 'dtype', floatx())
    theta = tf.cast(self.theta, dtype)
    return inputs * tf.cast(tf.greater(inputs, theta), dtype)
```

Of course, we can also directly use the `backend.relu` for the implementation of `ThresholdedReLU` like `ReLU` and `LeakyReLU` do.

**Standalone code to reproduce the issue**.
You can access this [link](https://colab.research.google.com/drive/144FOk8RO-Ew_eBtGZlCmsUL6k0cEAlUO?usp=sharing) or run the following code:

```
import keras
x = keras.layers.Input(shape=(1,10))
y = keras.layers.ThresholdedReLU()([x,x,x])
model = keras.models.Model(x,y)
model.summary()
```




Discussion:
@maybeLee,
`keras.layers.ThresholdedReLU` accepts input_shape as tuple of integers.

```
import keras
x = keras.layers.Input(shape=(1,10))
y = keras.layers.ThresholdedReLU()((x))
model = keras.models.Model(x,y)
model.summary()
```

**Output**
```
Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_10 (InputLayer)       [(None, 1, 10)]           0         
                                                                 
 thresholded_re_lu_11 (Thres  (None, 1, 10)            0         
 holdedReLU)                                                     
                                                                 
=================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0
```
@gadagashwini,
Thanks for your reply! I think the problem in this issue is that: `ThresholdedReLU` fails to _**accept a list of tensor**_ as input while similar layers such as `ReLU` and `LeakyReLU` can accept a list of tensor as input: 

Sending a list of tensor to `ReLU`:
```
import keras
x = keras.layers.Input(shape=(1,10))
y = keras.layers.ReLU()([x,x,x])
model = keras.models.Model(x,y)
model.summary()
```

```
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_17 (InputLayer)          [(None, 1, 10)]      0           []                               
                                                                                                  
 re_lu_7 (ReLU)                 (3, None, 1, 10)     0           ['input_17[0][0]',               
                                                                  'input_17[0][0]',               
                                                                  'input_17[0][0]']               
                                                                                                  
==================================================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0
__________________________________________________________________________________________________
```

Sending a list of tensor to `ThresholdedReLU`:
```
import keras
x = keras.layers.Input(shape=(1,10))
y = keras.layers.ThresholdedReLU()([x,x,x])
model = keras.models.Model(x,y)
model.summary()
```

```
[/usr/local/lib/python3.7/dist-packages/keras/layers/advanced_activations.py](https://localhost:8080/#) in call(self, inputs)
    262 
    263   def call(self, inputs):
--> 264     theta = tf.cast(self.theta, inputs.dtype)
    265     return inputs * tf.cast(tf.greater(inputs, theta), inputs.dtype)
    266 

AttributeError: Exception encountered when calling layer "thresholded_re_lu_7" (type ThresholdedReLU).

'list' object has no attribute 'dtype'

Call arguments received:
  • inputs=['tf.Tensor(shape=(None, 1, 10), dtype=float32)', 'tf.Tensor(shape=(None, 1, 10), dtype=float32)', 'tf.Tensor(shape=(None, 1, 10), dtype=float32)']
```

Resolving patch:
diff --git a/keras/layers/activation/thresholded_relu.py b/keras/layers/activation/thresholded_relu.py
--- a/keras/layers/activation/thresholded_relu.py
+++ b/keras/layers/activation/thresholded_relu.py
@@ -59,8 +59,8 @@ def __init__(self, theta=1.0, **kwargs):
     self.theta = backend.cast_to_floatx(theta)
 
   def call(self, inputs):
-    theta = tf.cast(self.theta, inputs.dtype)
-    return inputs * tf.cast(tf.greater(inputs, theta), inputs.dtype)
+    dtype = self.compute_dtype
+    return inputs * tf.cast(tf.greater(inputs, self.theta), dtype)
 
   def get_config(self):
     config = {'theta': float(self.theta)}


Test patch: