Repository: Theano/Theano

Issue:
MaxAndArgmax ValueError
I am using my own softmax function, which sometimes fails:

```
def SoftMax(x):
    x = T.exp(x - T.max(x, axis=x.ndim-1, keepdims=True))
    return x / T.sum(x, axis=x.ndim-1, keepdims=True)
```

My graph compiles fine, but when I execute it it sometimes fails with this error (mode=FAST_COMPILE):

```
Traceback (most recent call last):
  File "train.py", line 505, in <module>
    main(args)
  File "train.py", line 307, in main
    samples, costs = random_sampler.sample([sample_context], n_samples=1, n_turns=1)
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 55, in sample_apply
    samples, costs = sample_logic(sampler, joined_context, **kwargs) 
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 385, in sample
    next_probs, new_hd = self.next_probs_predictor(combined_hidden_states, prev_hd, prev_words, context)
  File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/compile/function_module.py", line 866, in __call__
    self.fn() if output_subset is None else\
  File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/vm.py", line 304, in __call__
    link.raise_with_op(node, thunk)
  File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/vm.py", line 300, in __call__
    thunk()
  File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/op.py", line 866, in rval
    r = p(n, [x[0] for x in i], o)
  File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/tensor/basic.py", line 1425, in perform
    (-1,))
ValueError: total size of new array must be unchanged
Apply node that caused the error: MaxAndArgmax(Elemwise{add,no_inplace}.0, TensorConstant{(1,) of 1})
Toposort index: 69
Inputs types: [TensorType(float32, matrix), TensorType(int64, (True,))]
Inputs shapes: [(0, 23), (1,)]
Inputs strides: [(92, 4), (8,)]
Inputs values: [array([], shape=(0, 23), dtype=float32), array([1])]
Outputs clients: [[InplaceDimShuffle{0,x}(max)], []]

Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 119, in sample
    self.compile()
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 84, in compile
    self.next_probs_predictor = self.model.build_next_probs_function()
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 2143, in build_next_probs_function
    outputs, hd = self.utterance_decoder.build_next_probs_predictor(self.beam_hd_input, self.beam_source, self.beam_hd)
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1369, in build_next_probs_predictor
    return self.build_decoder(inp, x, mode=UtteranceDecoder.BEAM_SEARCH, prev_state=prev_state)
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1438, in build_decoder
    _res = f_dec(x, xmask, decoder_inp, hd_init)
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1695, in LSTM_step
    softmax_outputs = self.output_softmax(pre_activ)
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1377, in output_softmax
    return SoftMax(T.dot(pre_activ, self.Wd_emb.T) + self.bd_out)
  File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/utils.py", line 221, in SoftMax
    x = T.exp(x - T.max(x, axis=x.ndim-1, keepdims=True))
```

BUT, if I instead use the nnet version, i.e.:

```
def SoftMax(x):
    return T.nnet.softmax(x)
```

then everything works fine. My graph is complicated, but could this be a bug in the T.max function?

Note, on my experiments, x is a TensorType(float32, matrix) with shape (1, 23) and I am using the bleeding edge Theano version: 146ef971fae05dc14ea10282b79f83e6796e94a0.



Discussion:
It seem the MaxAndArgmax python code don't like it when the inputs shapes
is (0, 23). Is it normal that you have a tensor with 0 elements?

I don't know why using the nnet version don't cause this problem. Why you
don't use it? It is a stable version like your implementation and work on
the GPU. So I don't see a reason not to use it.

You could use the flag optimizer=fast_compile instead of the mode flags.
This will make the graph optimization phase as fast, but still use C code
(the first time it compile could be slower to compile the c code we cache).

On Wed, Oct 12, 2016 at 1:15 PM, Iulian Vlad Serban <
notifications@github.com> wrote:

> I am using my own softmax function, which sometimes fails:
> 
> def SoftMax(x):
>     x = T.exp(x - T.max(x, axis=x.ndim-1, keepdims=True))
>     return x / T.sum(x, axis=x.ndim-1, keepdims=True)
> 
> My graph compiles fine, but when I execute it it sometimes fails with this
> error (mode=FAST_COMPILE):
> 
> Traceback (most recent call last):
>   File "train.py", line 505, in <module>
>     main(args)
>   File "train.py", line 307, in main
>     samples, costs = random_sampler.sample([sample_context], n_samples=1, n_turns=1)
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 55, in sample_apply
>     samples, costs = sample_logic(sampler, joined_context, **kwargs)
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 385, in sample
>     next_probs, new_hd = self.next_probs_predictor(combined_hidden_states, prev_hd, prev_words, context)
>   File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/compile/function_module.py", line 866, in **call**
>     self.fn() if output_subset is None else\
>   File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/vm.py", line 304, in **call**
>     link.raise_with_op(node, thunk)
>   File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/link.py", line 325, in raise_with_op
>     reraise(exc_type, exc_value, exc_trace)
>   File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/vm.py", line 300, in **call**
>     thunk()
>   File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/gof/op.py", line 866, in rval
>     r = p(n, [x[0] for x in i], o)
>   File "/u/serbaniv/THEANO_BLEEDING_EDGE/Theano/theano/tensor/basic.py", line 1425, in perform
>     (-1,))
> ValueError: total size of new array must be unchanged
> Apply node that caused the error: MaxAndArgmax(Elemwise{add,no_inplace}.0, TensorConstant{(1,) of 1})
> Toposort index: 69
> Inputs types: [TensorType(float32, matrix), TensorType(int64, (True,))]
> Inputs shapes: [(0, 23), (1,)]
> Inputs strides: [(92, 4), (8,)]
> Inputs values: [array([], shape=(0, 23), dtype=float32), array([1])]
> Outputs clients: [[InplaceDimShuffle{0,x}(max)], []]
> 
> Backtrace when the node is created(use Theano flag traceback.limit=N to make it longer):
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 119, in sample
>     self.compile()
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/search.py", line 84, in compile
>     self.next_probs_predictor = self.model.build_next_probs_function()
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 2143, in build_next_probs_function
>     outputs, hd = self.utterance_decoder.build_next_probs_predictor(self.beam_hd_input, self.beam_source, self.beam_hd)
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1369, in build_next_probs_predictor
>     return self.build_decoder(inp, x, mode=UtteranceDecoder.BEAM_SEARCH, prev_state=prev_state)
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1438, in build_decoder
>     _res = f_dec(x, xmask, decoder_inp, hd_init)
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1695, in LSTM_step
>     softmax_outputs = self.output_softmax(pre_activ)
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/dialog_encdec.py", line 1377, in output_softmax
>     return SoftMax(T.dot(pre_activ, self.Wd_emb.T) + self.bd_out)
>   File "/data/lisatmp4/serbaniv/Dialog/MrRNN-latent-norm/utils.py", line 221, in SoftMax
>     x = T.exp(x - T.max(x, axis=x.ndim-1, keepdims=True))
> 
> BUT, if I instead use the nnet version, i.e.:
> 
> def SoftMax(x):
>     return T.nnet.softmax(x)
> 
> then everything works fine. My graph is complicated, but could this be a
> bug in the T.max function?
> 
> Note, on my experiments, x is a TensorType(float32, matrix) with shape (1,
> 23) and I am using the bleeding edge Theano version: 146ef97
> https://github.com/Theano/Theano/commit/146ef971fae05dc14ea10282b79f83e6796e94a0
> .
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/Theano/Theano/issues/5081, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AALC-6ztokz6ugDndl31lGopdopupbtfks5qzRWXgaJpZM4KU_0q
> .

Yes, the issue is that `T.max(x, axis=x.ndim-1, keepdims=True)` does not work when `x.shape == (0, 23)`, because there is no maximum over axis 0.
I think `T.nnet.softmax` gets around that because it just returns an empty array without trying to do any computation on it.

Wait, `x.ndim-1` is 1, not 0. So it should work, and indeed it works in numpy:

``` python
>>> z = np.zeros((0, 23))
>>> z - z.max(axis=1, keepdims=True)
array([], shape=(0, 23), dtype=float64)
```

I'll have a look.


Resolving patch:
diff --git a/theano/tensor/basic.py b/theano/tensor/basic.py
--- a/theano/tensor/basic.py
+++ b/theano/tensor/basic.py
@@ -1421,8 +1421,10 @@ def perform(self, node, inp, outs):
                                 dtype='int64')
         # Not-reduced axes in front
         transposed_x = numpy.transpose(x, numpy.concatenate((keep_axes, axes)))
-        reshaped_x = transposed_x.reshape(transposed_x.shape[:len(keep_axes)] +
-                                          (-1,))
+        kept_shape = transposed_x.shape[:len(keep_axes)]
+        reduced_shape = transposed_x.shape[len(keep_axes):]
+        new_shape = kept_shape + (numpy.prod(reduced_shape),)
+        reshaped_x = transposed_x.reshape(new_shape)
 
         max_idx[0] = theano._asarray(numpy.argmax(reshaped_x, axis=-1),
                                      dtype='int64')


Test patch:
diff --git a/theano/tensor/tests/test_basic.py b/theano/tensor/tests/test_basic.py
--- a/theano/tensor/tests/test_basic.py
+++ b/theano/tensor/tests/test_basic.py
@@ -3083,6 +3083,15 @@ def test_multiple_axes(self):
         v = eval_outputs(max_and_argmax(x, [1, -1])[0].shape)
         assert tuple(v) == numpy.max(data, (1, -1)).shape
 
+    def test_zero_shape(self):
+        x = tensor.matrix()
+        m, i = max_and_argmax(x, axis=1)
+        f = theano.function([x], [m, i])
+        xv = numpy.zeros((0, 4), dtype=floatX)
+        mv, iv = f(xv)
+        assert mv.shape == (0,)
+        assert iv.shape == (0,)
+
 
 class T_argmin_argmax(unittest.TestCase):
     def setUp(self):