Repository: lisa-lab/pylearn2

Issue:
init_bias_target_marginals is broken due to deprecation of one-hot format
Copied from: https://groups.google.com/d/topic/pylearn-users/IGTc4740p8s/discussion

When I add "init_bias_target_marginals" to my SoftMax layer:

```
                 }, !obj:pylearn2.models.mlp.Softmax {
                     max_col_norm: 1.9365,
                     layer_name: 'y',
                     n_classes: 2,
                     irange: .0,
                     init_bias_target_marginals: *train,
                 }
```

I get the following exception:

```
ValueError: number of columns in x (2) does not match length of b (1)
Apply node that caused the error: GpuSoftmaxWithBias(GpuDot22.0, softmax_b)
Inputs types: [CudaNdarrayType(float32, matrix), CudaNdarrayType(float32, vector)]
Inputs shapes: [(25, 2), (1,)]
Inputs strides: [(2, 1), (0,)]
Inputs scalar values: ['not scalar', <CudaNdarray object at 0x1a4ca930>]
```

From Pascal:
It looks like the code computing the marginals expects the targets to be 
in one-hot format, even though it has been deprecated. 

The solution would be to update the computation done at mlp.py:1213, 
and add a test to make sure it does not get broken again. 

CC @madisonmay
CC @lamblin 



Discussion:
I'll take a look at this one tonight and see what I can do.

Correct me if I'm wrong, but it's a problem of calculating class frequencies from the vector of classes rather than the one hot vector, correct? 

i.e. 

```
n = float(init_bias_target_marginals.y.shape[0])
marginals = np.bincount(init_bias_target_marginals.y[:,0])/n
```

rather than

```
marginals = init_bias_target_marginals.y.mean(axis=0)
```

Yes, pretty much.
I think `init_bias_target_marginals.y` should be a vector in that case, though, so the indexing by `[:,0]` would not work.
In fact, maybe we should keep the previous code if `len(y.shape) == 2`, I'm not sure.
Edit: or, rather, depending on `self._has_binary_target`.

In practice I found `init_bias_target_marginals.y` to be a 2d array rather than a vector, but we could use `init_bias_target_marginals.y.flat` to handle both cases cleanly.

Docs for `np.ndarray.flat` are [here](http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.flat.html).

That sounds like a problem. At least if `init_bias_target_marginals` is a `DenseDesignMatrix`, if `y` is 2d of width 1, it should mean that the target is a scalar regression target, and in that case we should average it, not bin it (and it would not really do anything as the softmax of 1 element would always be 1 anyway).
Do you have an example of that?

Running [this yaml file](https://gist.github.com/madisonmay/a016d6c305fb4e018198) and printing the shape of `init_bias_target_marginals.y` should reproduce the described behavior.  

The shape of `init_bias_target_marginals.y` is reported to be (50000, 1).

Thanks for the example! It was pointed in https://groups.google.com/forum/#!topic/pylearn-dev/9CBeU72y6cc that I was mistaken about the dimensionality of y.
So, it looks like we should check `init_bias_target_marginals.y_labels`, apply the binning if it is not None, and apply the previous code if it is None.
Also, if the labels are not None, but there is more than one label per example, that should be an error, as it does not make sense to use Softmax for that.
Sorry about the confusion.


Resolving patch:
diff --git a/pylearn2/models/mlp.py b/pylearn2/models/mlp.py
--- a/pylearn2/models/mlp.py
+++ b/pylearn2/models/mlp.py
@@ -518,7 +518,7 @@ def get_output_space(self):
 
     @wraps(Layer.get_target_space)
     def get_target_space(self):
-        
+
         return self.layers[-1].get_target_space()
 
     @wraps(Layer.set_input_space)
@@ -1152,7 +1152,7 @@ class Softmax(Layer):
     layer_name : string
         Name of Softmax layers.
     irange : float
-        If specified, initialized each weight randomly in 
+        If specified, initialized each weight randomly in
         U(-irange, irange).
     istdev : float
         If specified, initialize each weight randomly from
@@ -1171,7 +1171,9 @@ class Softmax(Layer):
         inputs.
     max_col_norm : float
         Maximum norm for a column of the weight matrix.
-    init_bias_target_marginals : WRITEME
+    init_bias_target_marginals : dataset
+        Take the probability distribution of the targets into account to
+        intelligently initialize biases.
     binary_target_dim : int, optional
         If your targets are class labels (i.e. a binary vector) then set the
         number of targets here so that an IndexSpace of the proper dimension
@@ -1202,16 +1204,28 @@ def __init__(self, n_classes, layer_name, irange=None,
         if binary_target_dim is not None:
             assert isinstance(binary_target_dim, py_integer_types)
             self._has_binary_target = True
-            self._target_space = IndexSpace(dim=binary_target_dim, 
+            self._target_space = IndexSpace(dim=binary_target_dim,
                                             max_labels=n_classes)
         else:
             self._has_binary_target = False
-    
+
         self.output_space = VectorSpace(n_classes)
         if not no_affine:
             self.b = sharedX(np.zeros((n_classes,)), name='softmax_b')
             if init_bias_target_marginals:
-                marginals = init_bias_target_marginals.y.mean(axis=0)
+
+                y = init_bias_target_marginals.y
+                if init_bias_target_marginals.y_labels is None:
+                    marginals = y.mean(axis=0)
+                else:
+                    # compute class frequencies
+                    if np.max(y.shape) != np.prod(y.shape):
+                        raise AssertionError("Use of "
+                                             "`init_bias_target_marginals` "
+                                             "requires that each example has "
+                                             "a single label.")
+                    marginals = np.bincount(y.flat)/float(y.shape[0])
+
                 assert marginals.ndim == 1
                 b = pseudoinverse_softmax_numpy(marginals).astype(self.b.dtype)
                 assert b.ndim == 1
@@ -1492,12 +1506,12 @@ def _cost(self, Y, Y_hat):
         z = z - z.max(axis=1).dimshuffle(0, 'x')
         log_prob = z - T.log(T.exp(z).sum(axis=1).dimshuffle(0, 'x'))
         # we use sum and not mean because this is really one variable per row
-        
+
         if self._has_binary_target:
             # The following code is the equivalent of accessing log_prob by the
-            # indices in Y, but it is written such that the computation can 
+            # indices in Y, but it is written such that the computation can
             # happen on the GPU rather than CPU.
-            
+
             flat_Y = Y.flatten()
             flat_log_prob = log_prob.flatten()
             flat_indices = flat_Y + T.arange(Y.shape[0])*self.n_classes
@@ -1507,7 +1521,7 @@ def _cost(self, Y, Y_hat):
             log_prob_of = (Y * log_prob)
 
         return log_prob_of
-        
+
 
     @wraps(Layer.cost)
     def cost(self, Y, Y_hat):
@@ -4434,7 +4448,7 @@ def set_input_space(self, space):
                                                  for layer in self.layers))
         self._target_space = CompositeSpace(tuple(layer.get_target_space()
                                                   for layer in self.layers))
-        
+
     @wraps(Layer.get_params)
     def get_params(self):
         rval = []


Test patch:
diff --git a/pylearn2/models/tests/test_mlp.py b/pylearn2/models/tests/test_mlp.py
--- a/pylearn2/models/tests/test_mlp.py
+++ b/pylearn2/models/tests/test_mlp.py
@@ -3,8 +3,10 @@
 import numpy as np
 import theano
 from theano import tensor, config
+from nose.tools import assert_raises
 
 from pylearn2.datasets.vector_spaces_dataset import VectorSpacesDataset
+from pylearn2.datasets.dense_design_matrix import DenseDesignMatrix
 from pylearn2.termination_criteria import EpochCounter
 from pylearn2.training_algorithms.sgd import SGD
 from pylearn2.train import Train
@@ -13,6 +15,7 @@
                                  sampled_dropout_average, CompositeLayer)
 from pylearn2.space import VectorSpace, CompositeSpace, Conv2DSpace
 from pylearn2.utils import is_iterable, sharedX
+from pylearn2.expr.nnet import pseudoinverse_softmax_numpy
 
 
 class IdentityLayer(Linear):
@@ -302,7 +305,7 @@ def test_multiple_inputs():
             VectorSpace(20),
             VectorSpace(15),
             VectorSpace(5)]),
-         ('features1', 'features0', 'targets'))
+        ('features1', 'features0', 'targets'))
     )
     train = Train(dataset, mlp, SGD(0.1, batch_size=5))
     train.algorithm.termination_criterion = EpochCounter(1)
@@ -346,7 +349,7 @@ def test_softmax_binary_targets():
 
     y_hat_bin = mlp_bin.fprop(X)
     y_hat_vec = mlp_vec.fprop(X)
-    cost_bin = theano.function([X, y_bin], mlp_bin.cost(y_bin, y_hat_bin), 
+    cost_bin = theano.function([X, y_bin], mlp_bin.cost(y_bin, y_hat_bin),
                                allow_input_downcast=True)
     cost_vec = theano.function([X, y_vec], mlp_vec.cost(y_vec, y_hat_vec),
                                allow_input_downcast=True)
@@ -355,7 +358,8 @@ def test_softmax_binary_targets():
     y_bin_data = np.random.randint(low=0, high=10, size=(batch_size, 1))
     y_vec_data = np.zeros((batch_size, num_classes))
     y_vec_data[np.arange(batch_size),y_bin_data.flatten()] = 1
-    np.testing.assert_allclose(cost_bin(X_data, y_bin_data), cost_vec(X_data, y_vec_data))
+    np.testing.assert_allclose(cost_bin(X_data, y_bin_data),
+                               cost_vec(X_data, y_vec_data))
 
 def test_set_get_weights_Softmax():
     """
@@ -367,8 +371,7 @@ def test_set_get_weights_Softmax():
 
     # VectorSpace input space
     layer = Softmax(num_classes, 's', irange=.1)
-    softmax_mlp = MLP(layers=[layer],
-            input_space=VectorSpace(dim=dim))
+    softmax_mlp = MLP(layers=[layer], input_space=VectorSpace(dim=dim))
     vec_weights = np.random.randn(dim, num_classes).astype(config.floatX)
     layer.set_weights(vec_weights)
     assert np.allclose(layer.W.get_value(), vec_weights)
@@ -378,11 +381,72 @@ def test_set_get_weights_Softmax():
     # Conv2DSpace input space
     layer = Softmax(num_classes, 's', irange=.1)
     softmax_mlp = MLP(layers=[layer],
-            input_space=Conv2DSpace(shape=(conv_dim[0], conv_dim[1]), 
+            input_space=Conv2DSpace(shape=(conv_dim[0], conv_dim[1]),
                                     num_channels=conv_dim[2]))
-    conv_weights = np.random.randn(conv_dim[0], conv_dim[1], conv_dim[2], 
+    conv_weights = np.random.randn(conv_dim[0], conv_dim[1], conv_dim[2],
                                    num_classes).astype(config.floatX)
     layer.set_weights(conv_weights.reshape(np.prod(conv_dim), num_classes))
-    assert np.allclose(layer.W.get_value(), conv_weights.reshape(np.prod(conv_dim), num_classes))
+    assert np.allclose(layer.W.get_value(),
+                       conv_weights.reshape(np.prod(conv_dim), num_classes))
     layer.W.set_value(conv_weights.reshape(np.prod(conv_dim), num_classes))
-    assert np.allclose(layer.get_weights_topo(), np.transpose(conv_weights, axes=(3, 0, 1, 2)))
+    assert np.allclose(layer.get_weights_topo(),
+                       np.transpose(conv_weights, axes=(3, 0, 1, 2)))
+
+def test_init_bias_target_marginals():
+    """
+    Test `Softmax` layer instantiation with `init_bias_target_marginals`.
+    """
+    batch_size = 5
+    n_features = 5
+    n_classes = 3
+    n_targets = 3
+    irange = 0.1
+    learning_rate = 0.1
+
+    X_data = np.random.random(size=(batch_size, n_features))
+
+    Y_categorical = np.asarray([[0],[1],[1],[2],[2]])
+    class_frequencies = np.asarray([.2, .4, .4])
+    categorical_dataset = DenseDesignMatrix(X_data,
+                                            y=Y_categorical,
+                                            y_labels=n_classes)
+
+    Y_continuous = np.random.random(size=(batch_size, n_targets))
+    Y_means = np.mean(Y_continuous, axis=0)
+    continuous_dataset = DenseDesignMatrix(X_data,
+                                           y=Y_continuous)
+
+    Y_multiclass = np.random.randint(n_classes,
+                                     size=(batch_size, n_targets))
+    multiclass_dataset = DenseDesignMatrix(X_data,
+                                           y=Y_multiclass,
+                                           y_labels=n_classes)
+
+    def softmax_layer(dataset):
+        return Softmax(n_classes, 'h0', irange=irange,
+                       init_bias_target_marginals=dataset)
+
+    valid_categorical_mlp = MLP(
+        layers=[softmax_layer(categorical_dataset)],
+        nvis=n_features
+    )
+
+    actual = valid_categorical_mlp.layers[0].b.get_value()
+    expected = pseudoinverse_softmax_numpy(class_frequencies)
+    assert np.allclose(actual, expected)
+
+    valid_continuous_mlp = MLP(
+        layers=[softmax_layer(continuous_dataset)],
+        nvis=n_features
+    )
+
+    actual = valid_continuous_mlp.layers[0].b.get_value()
+    expected = pseudoinverse_softmax_numpy(Y_means)
+    assert np.allclose(actual, expected)
+
+    def invalid_multiclass_mlp():
+        return MLP(
+            layers=[softmax_layer(multiclass_dataset)],
+            nvis=n_features
+        )
+    assert_raises(AssertionError, invalid_multiclass_mlp)