Repository: Theano/Theano

Issue:
Test_value 
I'm trying out the new GPU backend and I've encountered an error when I set `compute_test_value=warn`. The error doesn't happen with the old backend.

My theano version is: 0.9.0beta1.dev-e5d51daf5ac03bfd8bd076075ee587311dff6f48 and my `.theanorc` is empty. Also, I've installed libgpuarray this morning and I'm using CUDA 8.0 with cudnn v5.1 and numpy 1.11.3.

Here is a small example to reproduce the bug:

```python
import numpy as np
import theano
import theano.tensor as T

print("Theano v.: {}".format(theano.version.full_version))

init_state = theano.shared(np.zeros((4, 5), dtype=theano.config.floatX))
results, updates = \
    theano.scan(lambda x: x + 1,
                sequences=[],
                outputs_info=[init_state],
                n_steps=5)

print(results.tag.test_value)
```
Here is what I get when using the **old** backend:
```bash
THEANO_FLAGS="device=gpu1,compute_test_value=warn" python theano_test_value_issue.py
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release.  Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
 https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

/mnt/folder1/users/marc.cote/theano/theano/sandbox/cuda/__init__.py:556: UserWarning: Theano flag device=gpu* (old gpu back-end) only support floatX=float32. You have floatX=float64. Use the new gpu back-end with device=cuda* for that value of floatX.
  warnings.warn(msg)
Using gpu device 1: GeForce GTX TITAN X (CNMeM is disabled, cuDNN 5110)
Theano v.: 0.9.0beta1.dev-e5d51daf5ac03bfd8bd076075ee587311dff6f48
[[[ 1.  1.  1.  1.  1.]
  [ 1.  1.  1.  1.  1.]
  [ 1.  1.  1.  1.  1.]
  [ 1.  1.  1.  1.  1.]]

 [[ 2.  2.  2.  2.  2.]
  [ 2.  2.  2.  2.  2.]
  [ 2.  2.  2.  2.  2.]
  [ 2.  2.  2.  2.  2.]]

 [[ 3.  3.  3.  3.  3.]
  [ 3.  3.  3.  3.  3.]
  [ 3.  3.  3.  3.  3.]
  [ 3.  3.  3.  3.  3.]]

 [[ 4.  4.  4.  4.  4.]
  [ 4.  4.  4.  4.  4.]
  [ 4.  4.  4.  4.  4.]
  [ 4.  4.  4.  4.  4.]]

 [[ 5.  5.  5.  5.  5.]
  [ 5.  5.  5.  5.  5.]
  [ 5.  5.  5.  5.  5.]
  [ 5.  5.  5.  5.  5.]]]
```

Here is what I get when using the **new** backend:
```bash
THEANO_FLAGS="device=cuda1,compute_test_value=warn" python theano_test_value_issue.py
Using cuDNN version 5110 on context None
Mapped name None to device cuda1: GeForce GTX TITAN X (0000:01:00.0)
Theano v.: 0.9.0beta1.dev-e5d51daf5ac03bfd8bd076075ee587311dff6f48
Traceback (most recent call last):
  File "theano_test_value_issue.py", line 13, in <module>
    n_steps=5)
  File "/mnt/folder1/users/marc.cote/theano/theano/scan_module/scan.py", line 773, in scan
    condition, outputs, updates = scan_utils.get_updates_and_outputs(fn(*args))
  File "theano_test_value_issue.py", line 10, in <lambda>
    theano.scan(lambda x: x + 1,
  File "/mnt/folder1/users/marc.cote/theano/theano/tensor/var.py", line 128, in __add__
    return theano.tensor.basic.add(self, other)
  File "/mnt/folder1/users/marc.cote/theano/theano/gof/op.py", line 614, in __call__
    storage_map[ins] = [self._get_test_value(ins)]
  File "/mnt/folder1/users/marc.cote/theano/theano/gof/op.py", line 551, in _get_test_value
    ret = v.type.filter(v.tag.test_value)
  File "/mnt/folder1/users/marc.cote/theano/theano/tensor/type.py", line 150, in filter
    converted_data = theano._asarray(data, self.dtype)
  File "/mnt/folder1/users/marc.cote/theano/theano/misc/safe_asarray.py", line 34, in _asarray
    rval = np.asarray(a, dtype=dtype, order=order)
  File "/home/marc.cote/.local/env/gpu/local/lib/python2.7/site-packages/numpy/core/numeric.py", line 482, in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: For compute_test_value, one input test value does not have the requested type.
 
Backtrace when that variable is created:

  File "theano_test_value_issue.py", line 13, in <module>
    n_steps=5)

The error when converting the test value to that variable type:
__array__() takes no arguments (1 given)
```

I've investigated a bit and found out that the symbolic variable `x` in the lambda function got a `x.tag.test_value` which is a `gpuarray.array`. In my code, the error seems to be raised around here:
```bash
/mnt/folder1/users/marc.cote/theano/theano/gof/op.pyc in __call__(self, *inputs, **kwargs)
    612             for i, ins in enumerate(node.inputs):
    613                 try:
--> 614                     storage_map[ins] = [self._get_test_value(ins)]
    615                     compute_map[ins] = [True]
    616                 except AttributeError:
```
where `ins` would be `x` in the simple case.

Let me know how I can help.
Edit: It's probably not related but I found this while searching for the error message: https://github.com/Theano/Theano/blob/e345e095f94b0186b401977607eb009612ac821f/theano/scan_module/tests/test_scan.py#L1093


Discussion:
After a bit more of debugging:

```bash
> /mnt/folder1/users/marc.cote/theano/theano/misc/safe_asarray.py(34)_asarray()
     32         dtype = theano.config.floatX
     33     dtype = np.dtype(dtype)  # Convert into dtype object.
---> 34     rval = np.asarray(a, dtype=dtype, order=order)
     35     # Note that dtype comparison must be done by comparing their `num`
     36     # attribute. One cannot assume that two identical data types are pointers
```

For that particular debugging instance, `a` was a `gpuarray.array`
```bash
ipdb> print(repr(a))
gpuarray.array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       ..., 
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32)
```
which causes the following to raise an error:
```bash
ipdb> np.asarray(a, dtype=dtype, order=order)
*** TypeError: __array__() takes no arguments (1 given)
``` 

That said, not providing the `dtype` seems to "work":
```bash
ipdb> np.asarray(a, order=order)
array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       ..., 
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32)
```

Is it possible that `np.asarray` (I'm using Numpy v.11.3) has changed behavior and is now passing `dtype` to `__array__()` which doesn't expect that particular argument (https://github.com/Theano/libgpuarray/blob/master/pygpu/gpuarray.pyx#L1650)?
I would not be surprised of what you suggest. Numpy started to pass more
flag to downstream function. But it also raise the questions, what was
happening in the old backend before that made this work?

@abergeron, do you have time to check? He did a PR that also fix something
related to scan and the new backend. Can you try it? He have only one PR,
it is related to share variable the new backend.

Le lun. 6 févr. 2017 15:45, Marc-Alexandre Côté <notifications@github.com>
a écrit :

> After a bit more of debugging:
>
> > /mnt/folder1/users/marc.cote/theano/theano/misc/safe_asarray.py(34)_asarray()
>      32         dtype = theano.config.floatX
>      33     dtype = np.dtype(dtype)  # Convert into dtype object.
> ---> 34     rval = np.asarray(a, dtype=dtype, order=order)
>      35     # Note that dtype comparison must be done by comparing their `num`
>      36     # attribute. One cannot assume that two identical data types are pointers
>
> For that particular debugging instance, a was a gpuarray.array
>
> ipdb> print(repr(a))
> gpuarray.array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        ...,
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32)
>
> which causes the following to raise an error:
>
> ipdb> np.asarray(a, dtype=dtype, order=order)*** TypeError: __array__() takes no arguments (1 given)
>
> That said, not providing the dtype seems to "work":
>
> ipdb> np.asarray(a, order=order)
> array([[ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        ...,
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.],
>        [ 0.,  0.,  0., ...,  0.,  0.,  0.]], dtype=float32)
>
> Is it possible that np.asarray (I'm using Numpy v.11.3) has changed
> behavior and is now passing dtype to __array__() which doesn't expect
> that particular argument (
> https://github.com/Theano/libgpuarray/blob/master/pygpu/gpuarray.pyx#L1650
> )?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/Theano/Theano/issues/5503#issuecomment-277807774>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AALC-0VeO9roYbUYh3E3a6z_xup2p2xZks5rZ4Z8gaJpZM4L4pzd>
> .
>

I've looked at PR #5462 and it seems related. I'll try it tomorrow. Thanks.

Also, I've stumbled on another error with the new backend and scan. For some reason the `n_steps` argument is not an integer but a `gpuarray` object. It might be fixed by the aforementioned PR, though. I'll let you know.
So, I tried commit 943b347 (from PR #5462) and it is still giving me the error.

I have a **workaround** for now which consists in adding a `dtype` parameter to the `GpuArray.__array__` definition and just don't use it in the function.
For the scan n_steps error, it was reported by a student here. This PR fix it: https://github.com/Theano/Theano/pull/5508

But he have a later error. 
I merged gh-5462 and made a new PR that fix the n_steps error: https://github.com/Theano/Theano/pull/5508 that is based on the merged PR. So mostly, just test that new PR.

The n_steps error was reported by someone else, but he had a later error. If you still have problems, having a way to reproduce it would help us.
This PR should the dtype problem for array: https://github.com/Theano/libgpuarray/pull/347
@abergeron Great I'll try it out.
@nouiz regarding the scan a new error is raised here:
```bash
/mnt/folder1/users/marc.cote/theano/theano/scan_module/scan_perform.pyx in theano.scan_module.scan_perform.perform (/tmp/marc/.theano/scan_perform/mod.cpp:2803)()
    232     for idx in range(n_nit_sot):
    233         store_steps[<unsigned int>(idx + n_mit_mot + n_mit_sot + n_sit_sot)]=\
--> 234                 args[<unsigned int>(idx + n_mit_mot + n_mit_sot + n_sit_sot
    235                                     + n_shared_outs + n_seqs+1)]
    236 
TypeError: an integer is required
```
When I poke around:
```bash
    921             def p(node, args, outs):
    922                 return scan_perform_ext.perform(self.n_shared_outs,
    923                                                 self.n_mit_mot_outs,
    924                                                 self.n_seqs,
    925                                                 self.n_mit_mot,
    926                                                 self.n_mit_sot,
    927                                                 self.n_sit_sot,
    928                                                 self.n_nit_sot,
    929                                                 args[0],
    930                                                 self.as_while,
    931                                                 cython_mintaps,
    932                                                 cython_tap_array,
    933                                                 cython_tap_array_len,
    934                                                 cython_vector_seqs,
    935                                                 cython_vector_outs,
    936                                                 cython_mit_mot_out_slices,
    937                                                 cython_mit_mot_out_nslices,
    938                                                 cython_mitmots_preallocated,
    939                                                 cython_inps_is_tensor,
    940                                                 cython_outs_is_tensor,
    941                                                 self.fn.fn,
    942                                                 self.fn,
    943                                                 cython_destroy_map,
    944                                                 args,
    945                                                 outs,
--> 946                                                 self, node)
    947         except (ImportError, theano.gof.cmodule.MissingGXX):

ipdb> self.n_shared_outs
0
ipdb> self.n_mit_mot_outs
1
ipdb> self.n_seqs
3
ipdb> self.n_mit_mot
1
ipdb> self.n_mit_sot
0
ipdb> self.n_sit_sot
2
ipdb> self.n_nit_sot
3
ipdb> args[0]
array(20)
ipdb> self.as_while
False
ipdb> cython_mintaps
array([ 0, -1, -1,  0,  0,  0], dtype=int32)
ipdb> cython_tap_array
array([[ 0,  1],
       [-1,  0],
       [-1,  0]], dtype=int32)
ipdb> cython_tap_array_len
array([2, 1, 1], dtype=int32)
ipdb> cython_vector_seqs
array([0, 0, 0], dtype=int32)
ipdb> cython_vector_outs
array([0, 0, 0, 0, 0, 0], dtype=int32)
ipdb> cython_mit_mot_out_slices
array([[1]], dtype=int32)
ipdb> cython_mit_mot_out_nslices
array([1], dtype=int32)
ipdb> cython_mitmots_preallocated
array([1], dtype=int32)
ipdb> cython_inps_is_tensor
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)
ipdb> cython_outs_is_tensor
array([0, 0, 0, 0, 0, 0], dtype=int32)
ipdb> len(args)
```
Also,
```bash
ipdb> self.n_mit_mot + self.n_mit_sot + self.n_sit_sot + self.n_shared_outs + self.n_seqs + 1
7
ipdb> type(self.n_mit_mot + self.n_mit_sot + self.n_sit_sot + self.n_shared_outs + self.n_seqs + 1)
<type 'int'>
```
I think the following code:
```bash
/mnt/folder1/users/marc.cote/theano/theano/scan_module/scan_perform.pyx in theano.scan_module.scan_perform.perform (/tmp/marc/.theano/scan_perform/mod.cpp:2803)()
    232     for idx in range(n_nit_sot):
    233         store_steps[<unsigned int>(idx + n_mit_mot + n_mit_sot + n_sit_sot)]=\
--> 234                 args[<unsigned int>(idx + n_mit_mot + n_mit_sot + n_sit_sot
    235                                     + n_shared_outs + n_seqs+1)]
    236 
TypeError: an integer is required
```

is trying to get the address of a `gpuarray.array` object and store it in `store_steps` (which is an array of `int`)? Is it possible the necessary functions to cast a `gpuarray.array` to an `int` weren't implemented?
Oh no, that is not what it does. Looking at the `args` elements that are being indexed:
```bash
ipdb> args[7]
gpuarray.array(20)
ipdb> args[8]
gpuarray.array(20)
ipdb> args[9]
gpuarray.array(20)
```
Those are supposed to be castable as `int`. In this example, `20` is related to `n_steps` since it is the length of my input sequence(s).
See https://github.com/Theano/libgpuarray/pull/348. ~~Maybe #5508 is not needed anymore, I haven't checked yet.~~
Edit: I guessed it is too late but #5508 wasn't needed to make `theano.scan` work with the new GPU backend.
I think the args[x] should be an int, as we try to store it in an int array.

Do we miss a ".shape" as line 230 of file scan_perform.pyx?

On Tue, Feb 7, 2017 at 1:51 PM Marc-Alexandre Côté <notifications@github.com>
wrote:

> See Theano/libgpuarray#348
> <https://github.com/Theano/libgpuarray/pull/348>. Maybe #5508
> <https://github.com/Theano/Theano/pull/5508> is not needed anymore, I
> haven't checked yet.
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/Theano/Theano/issues/5503#issuecomment-278102631>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AALC-7ueM3mnRV_elpmbAjMh3YtwV7V-ks5raL0ogaJpZM4L4pzd>
> .
>

I'm still trying to pin down the cause of this issue. I think it is happening in the scan generated for the backpropagation. 
Alright, here a relatively small example reproducing the bug.

The command I ran (I'm using the master branch for both Theano and libgpuarray, numpy version is 1.11.3):
`THEANO_FLAGS="device=cuda0,floatX=float32,gpuarray.preallocate=0" python theano_nsteps_scan_issue.py`

Content of `theano_nsteps_scan_issue.py`.
```python
import numpy as np

import theano
import theano.tensor as T
from theano.ifelse import ifelse
from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams
import pygpu

print("Numpy v.: {}".format(np.version.full_version))
print("Theano v.: {}".format(theano.version.full_version))
print("libgpuarray v.: {}".format(pygpu.__version__))

t_rng = RandomStreams(42)
rng = np.random.RandomState(42)
tr_phase = T.fscalar('tr_phase')   # training phase switch

i_drop = 0.10       # input dropout
nc = 1234           # input channels
ns = 20             # sequence length
nbatch = 128        # batch size
state_chans = 512   # state channels


class Dropout(object):
    def __init__(self, dropout):
        self.dropout = dropout

    def __call__(self, X):
        if (self.dropout > 0.):
            retain_prob = 1 - self.dropout
            Xd = X * t_rng.binomial(X.shape, p=retain_prob, dtype=theano.config.floatX)
            Xd /= retain_prob
            X = ifelse(tr_phase > 0.5, Xd, X)
        return X

class BasicFCModule(object):
    def __init__(self, in_chans, out_chans):
        self.in_chans = in_chans
        self.out_chans = out_chans
        self.w1 = theano.shared(rng.rand(self.in_chans, self.out_chans).astype(theano.config.floatX))
        self.params = [self.w1]

    def apply(self, input):
        return self.w1[input] # Word embedding


class BasicLSTMModule(object):
    def __init__(self, state_chans):
        self.state_chans = state_chans
        self.apply_bn = False
        self.w1 = theano.shared(rng.rand(self.state_chans, 4*self.state_chans).astype(theano.config.floatX))
        self.params = [self.w1]

    def apply(self, input):
        def _step_func(input, state):
            s_vis = state[:, :self.state_chans]
            s_hid = state[:, self.state_chans:]
            h1 = T.dot(s_vis, self.w1) + input
            g_inp = T.nnet.sigmoid(h1[:, (0 * self.state_chans):(1 * self.state_chans)])
            g_for = T.nnet.sigmoid(h1[:, (1 * self.state_chans):(2 * self.state_chans)] + 1.)
            g_out = T.nnet.sigmoid(h1[:, (2 * self.state_chans):(3 * self.state_chans)])
            s_upd = T.tanh(h1[:, (3 * self.state_chans):])
            s_hid_new = g_for * s_hid + g_inp * s_upd
            s_vis_new = g_out * T.tanh(s_hid_new)
            return T.concatenate([s_vis_new, s_hid_new], axis=1)

        init_state = T.alloc(0., input.shape[0], 2 * self.state_chans)
        seq_input = input.dimshuffle(1, 0, 2)
        states, scan_updates = \
            theano.scan(_step_func,
                        outputs_info=[init_state],
                        sequences=[seq_input])

        states = states.dimshuffle(1, 0, 2)
        return states, scan_updates


inp_mod = BasicFCModule(in_chans=nc, out_chans=(4 * state_chans))
rec_mod = BasicLSTMModule(state_chans=state_chans)

all_params = inp_mod.params + rec_mod.params

# Setup symbolic vars for the model inputs, outputs, and costs
X_in = T.lmatrix()   # input sequences (nbatch, seq_len)
Y_in = T.lmatrix()   # input sequences (nbatch, seq_len)

inp_drop = Dropout(i_drop)
inp_x = inp_mod.apply(X_in)
inp_x = inp_drop(inp_x)  # Apply dropout

rnn_states, _ = rec_mod.apply(inp_x)
vis_states = rnn_states[:, :, :state_chans]
full_cost = vis_states.sum() * Y_in.sum()  + tr_phase  # Fake cost

grads = T.grad(full_cost, all_params)
params_updates = [(p, p - 0.001 * g) for p, g in zip(all_params, grads)]

print("Compiling training functions...")
g_trainunc = theano.function([X_in, Y_in], [full_cost],
                             givens={tr_phase: np.array(1., dtype=theano.config.floatX)},
                             updates=params_updates)

print("Running...")
xtr = rng.randint(0, nc, size=(nbatch, ns))
ytr = rng.randint(0, nc, size=(nbatch, ns))
t_result = g_trainunc(xtr, ytr)
```
Commenting the following 
`inp_x = inp_drop(inp_x)  # Apply dropout`
doesn't raise the error. Maybe it has to do with `ifelse` or the `binomial`

Resolving patch:
diff --git a/theano/gpuarray/opt.py b/theano/gpuarray/opt.py
--- a/theano/gpuarray/opt.py
+++ b/theano/gpuarray/opt.py
@@ -193,7 +193,8 @@ def local_opt(node):
                 context_name = None
                 # We replace if any input is a host_from_gpu
                 for i in node.inputs:
-                    if i.owner and i.owner.op == host_from_gpu:
+                    if (i.owner and i.owner.op == host_from_gpu and
+                            move_to_gpu(i)):
                         context_name = i.owner.inputs[0].type.context_name
                         replace = True
                         break
@@ -922,7 +923,7 @@ def local_gpua_lazy_ifelse(op, context_name, inputs, outputs):
     c = inputs[0]
     inps = []
     for v in inputs[1:]:
-        if isinstance(v.type, tensor.TensorType):
+        if isinstance(v.type, tensor.TensorType) and move_to_gpu(v):
             inps.append(as_gpuarray_variable(v, context_name))
         else:
             inps.append(v)
diff --git a/theano/scan_module/scan_op.py b/theano/scan_module/scan_op.py
--- a/theano/scan_module/scan_op.py
+++ b/theano/scan_module/scan_op.py
@@ -619,7 +619,8 @@ def format(var, as_var):
         # in this case is just a int saying how many steps of this output we
         # need to store. This input does not have the same dtype, nor is it the same
         # type of tensor as the output, it is always a scalar int.
-        new_inputs += self.outer_nitsot(inputs)
+        new_inputs += [as_tensor_variable(ons)
+                       for ons in self.outer_nitsot(inputs)]
         for inner_nonseq, _outer_nonseq in zip(self.inner_non_seqs(self.inputs),
                                                self.outer_non_seqs(inputs)):
             outer_nonseq = format(_outer_nonseq, as_var=inner_nonseq)


Test patch: