Repository: Theano/Theano

Issue:
Warn if taking softmax over broadcastable dimension
At least one user inadvertently had a softmax applied to a network layer with a single output unit. The softmax Op could issue a warning if the axis to be softmaxed is broadcastable (i.e., guaranteed to only have a single element), because usually you wouldn't want this. (Otherwise, Theano could also use an optimizer replacing softmax over a broadcastable dimension with `T.ones_like()` ;) ).


Discussion:
Can I work on this? I'm new so will take some time. 

So this is basically about changing theano.tensor.nnet.nnet.softmax to issue a warning when softmax is fixed to a single output unit layer, isn't it? 

Also I was thinking if adding the optimizer to replace softmax for broadcastable dimensions can be done alongside the warning without using only one. 
Nobody work on it to my knowledge. When we build the graph, we could put a
one like as Jan said, but I think it would be better done as an
optimisation. I would recommend that you do a PR with the warning first.
The optimisation isn't really useful add this is a useless use case.

Le sam. 18 févr. 2017 11:01, Kanchana Ranasinghe <notifications@github.com>
a écrit :

> Can I work on this? I'm new so will take some time.
>
> So this is basically about changing theano.tensor.nnet.nnet.softmax to
> issue a warning when softmax is fixed to a single output unit layer, isn't
> it?
>
> Also I was thinking if adding the optimizer to replace softmax for
> broadcastable dimensions can be done alongside the warning without using
> only one.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/Theano/Theano/issues/5501#issuecomment-280854595>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AALC-yXSsGg5LfF0iSFDwKfxYpiOJrT2ks5rdxXSgaJpZM4L4gZl>
> .
>

> So this is basically about changing theano.tensor.nnet.nnet.softmax to issue a warning when softmax is fixed to a single output unit layer, isn't it?

Exactly. As the softmax currently only supports 2D input, you'd just need to check if `input.broadcastable[1]` is True and issue a warning if it is.

> The optimisation isn't really useful add this is a useless use case.

Yep, this was meant as a joke, sorry if it wasn't explicit enough!

Resolving patch:
diff --git a/theano/tensor/nnet/nnet.py b/theano/tensor/nnet/nnet.py
--- a/theano/tensor/nnet/nnet.py
+++ b/theano/tensor/nnet/nnet.py
@@ -14,13 +14,14 @@
 """
 from __future__ import absolute_import, print_function, division
 import logging
+import warnings
 import numpy
 from six.moves import xrange
 
 import theano
 from theano import gof
 from theano import scalar
-from theano.tensor import extra_ops
+from theano.tensor import extra_ops, as_tensor_variable
 from theano.gof.opt import copy_stack_trace
 from theano.tensor import basic as tensor, subtensor, opt, elemwise
 from theano.tensor.type import (values_eq_approx_remove_inf,
@@ -802,6 +803,9 @@ def softmax_graph(c):
 
 
 def softmax(c):
+    c = as_tensor_variable(c)
+    if c.broadcastable[-1]:
+        warnings.warn("The softmax is applied on a dimension of shape 1, which does not have a semantic meaning.")
     return softmax_op(c)
 
 


Test patch: