Repository: keras-team/keras

Issue:
is the Sequence to Sequence learning right?
Assume we are trying to learn a sequence to sequence map. For this we can use Recurrent and TimeDistributedDense layers. Now assume that the sequences have different lengths. We should pad both input and desired sequences with zeros, right? But how will the objective function handle the  padded values? There is no choice to pass a mask to the objective function. Won't this bias the cost function?



Discussion:
> We should pad both input and desired sequences with zeros, right? But how will the objective function handle the padded values?

You could use a mask the hide your padded values from the network. Then you can discard the masked values in your sequence output. Currently masking is only supported via an initial Embedding layer, though. See: http://keras.io/layers/recurrent/

I'm a little new to recurrent network. When Eder talked about the sequence to sequence map, it only reminds me of the char-level LSTM (http://karpathy.github.io/2015/05/21/rnn-effectiveness/). In this case, even we can discard the masked values in your sequence output, the padding values still have effects on the hyper-parameters of the model itself. So is it enough to just discard the masked values? Again, as Eder has asked, won't this bias the cost function?

Maybe this issue is of your interest #382

This worked for me. Padding the inputs and then the outputs, and adding special sequence start and sequence stop symbols to book-end each sequence, then the following model structure:

``` python
embedding_size = 64
hidden_size = 512

print('Build model...')
model = Sequential()
model.add(Embedding(max_features, embedding_size))
model.add(JZS1(embedding_size, hidden_size)) # try using a GRU instead, for fun
model.add(Dense(hidden_size, hidden_size))
model.add(Activation('relu'))
model.add(RepeatVector(MAX_LEN))
model.add(JZS1(hidden_size, hidden_size, return_sequences=True))
model.add(TimeDistributedDense(hidden_size, max_features, activation="softmax"))

model.compile(loss='mse', optimizer='adam')
```

If you have a sequence stop symbol, it should learn when to stop outputing non-zero values, and will output zero's thereafter. May not be ideal, but works within the current framework. I also tried replicating the output to the maxlen width (including the stop symbol) during training, and then just took the first valid sequence at test time.

btw, JZS1 is an RNN like a GRU or an LSTM, each of which could be used here instead in both the encoder and decoder.

Sounds like a good idea, but see that you are forcing your model to learn something else instead of your original problem.
I think I know how to solve this problem with a masking layer after each regular layer and if we allow loss to be a custom function from. This way, instead of averaging the cost with .mean(), we divide it by the number of non zero elements on each time series. This non discriminant averaging is where a lot of bias come from as well.

@fchollet is there any chance that we can get "loss" to check if its input is callable? I really didn't want to have to add new stuff to my code repo every time I needed something custom made. I can write a PR for that if there is a general interest. Let me know.

PR #446 is relevant here. Now all that we need is that the cost functions ask for that mask in their calculations.


Resolving patch:
diff --git a/keras/models.py b/keras/models.py
--- a/keras/models.py
+++ b/keras/models.py
@@ -63,13 +63,20 @@ def slice_X(X, start=None, stop=None):
 
 
 def weighted_objective(fn):
-    def weighted(y_true, y_pred, weights):
+    def weighted(y_true, y_pred, weights, mask=None):
         # it's important that 0 * Inf == 0, not NaN, so I need to mask first
         masked_y_true = y_true[weights.nonzero()[:-1]]
         masked_y_pred = y_pred[weights.nonzero()[:-1]]
         masked_weights = weights[weights.nonzero()]
         obj_output = fn(masked_y_true, masked_y_pred)
-        return (masked_weights.flatten() * obj_output.flatten()).mean()
+        if mask is None:
+            return (masked_weights.flatten() * obj_output.flatten()).mean()
+        else:
+            # We assume the time index to be masked is axis=1
+            wc = masked_weights * obj_output
+            wc = wc.reshape(mask.shape)
+            wc = wc.sum(axis=1) / mask.sum(axis=1)
+            return wc.mean()
     return weighted
 
 
@@ -338,7 +345,8 @@ class Sequential(Model, containers.Sequential):
             - set_weights
     '''
 
-    def compile(self, optimizer, loss, class_mode="categorical", theano_mode=None):
+    def compile(self, optimizer, loss, class_mode="categorical", theano_mode=None,
+                mask_cost=False):
         self.optimizer = optimizers.get(optimizer)
 
         self.loss = objectives.get(loss)
@@ -356,8 +364,12 @@ def compile(self, optimizer, loss, class_mode="categorical", theano_mode=None):
 
         self.weights = T.ones_like(self.y_train)
 
-        train_loss = weighted_loss(self.y, self.y_train, self.weights)
-        test_loss = weighted_loss(self.y, self.y_test, self.weights)
+        if mask_cost:
+            mask = self.layers[-1].get_output_mask()
+        else:
+            mask = None
+        train_loss = weighted_loss(self.y, self.y_train, self.weights, mask)
+        test_loss = weighted_loss(self.y, self.y_test, self.weights, mask)
 
         train_loss.name = 'train_loss'
         test_loss.name = 'test_loss'


Test patch:
diff --git a/tests/auto/test_loss_masking.py b/tests/auto/test_loss_masking.py
new file mode 100644
--- /dev/null
+++ b/tests/auto/test_loss_masking.py
@@ -0,0 +1,25 @@
+import numpy as np
+from keras.models import Sequential
+from keras.layers.core import TimeDistributedDense, Masking
+
+
+def test_cost_masking():
+    X = np.array(
+        [[[1, 1], [2, 1], [3, 1], [5, 5]],
+         [[1, 5], [5, 0], [0, 0], [0, 0]]], dtype=np.int32)
+
+    model = Sequential()
+    model.add(Masking(mask_value=0))
+    model.add(TimeDistributedDense(2, 1, init='one'))
+    model.compile(loss='mse', optimizer='sgd')
+    y = model.predict(X)
+
+    loss = model.fit(X, 4*y, nb_epoch=1, batch_size=2, verbose=1).history['loss'][0]
+    assert loss == 213.75
+
+    model = Sequential()
+    model.add(Masking(mask_value=0))
+    model.add(TimeDistributedDense(2, 1, init='one'))
+    model.compile(loss='mse', optimizer='sgd', mask_cost=True)
+    loss = model.fit(X, 4*y, nb_epoch=1, batch_size=2, verbose=1).history['loss'][0]
+    assert loss == 282.375