Repository: ray-project/ray

Issue:
[tune] Option --no-cuda is misleading in mnist_pytorch.py example
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Ubuntu 16.04
- **Ray installed from (source or binary)**: binary
- **Ray version**: 0.6.2
- **Python version**: 3.5
- **Exact command to reproduce**:

```
python3 mnist_pytorch.py
```

### Describe the problem

Argument parser's option `--no-cuda` which is False by default is misleading as `resources_per_trial` does not allocate a GPU.

https://github.com/ray-project/ray/blob/eddd60e14e95a2aadb06192bd141d06c68d5f082/python/ray/tune/examples/mnist_pytorch.py#L45-L49

https://github.com/ray-project/ray/blob/eddd60e14e95a2aadb06192bd141d06c68d5f082/python/ray/tune/examples/mnist_pytorch.py#L178-L180

Thanks



Discussion:
Nice catch! Would you be interested in pushing a fix?
@richardliaw yes, I can send a PR with a fix of both PyTorch examples : 
- mnist_pytorch_trainable.py
- mnist_pytorch.py.  

Idea is to add one GPU:
```python
"resources_per_trial": { 
     "cpu": 3,
     "gpu": 1
}
```
what do you think ?

PS: IMO examples are a little bit outdated as `Variable` is used which was deprecated since `0.4.0`.
Maybe we should update this too.

I think maybe do something like `"gpu": int(not args.no_cuda)`, since you don't want to set it like that when CUDA is disabled. 

Updating deprecated things would be great; thanks!
> I think maybe do something like "gpu": int(not args.no_cuda), since you don't want to set it like that when CUDA is disabled.

@richardliaw yes, but this is handled inside the trial code:
https://github.com/ray-project/ray/blob/eddd60e14e95a2aadb06192bd141d06c68d5f082/python/ray/tune/examples/mnist_pytorch.py#L62
so we can leave savely `"gpu": 1`.
If you leave “gpu”: 1, the example will not run if Ray does not detect a
GPU.

Thanks,
Richard

On Tue, Jan 29, 2019 at 2:37 AM vfdev <notifications@github.com> wrote:

> I think maybe do something like "gpu": int(not args.no_cuda), since you
> don't want to set it like that when CUDA is disabled.
>
> @richardliaw <https://github.com/richardliaw> yes, but this is handled
> inside the trial code:
>
> https://github.com/ray-project/ray/blob/eddd60e14e95a2aadb06192bd141d06c68d5f082/python/ray/tune/examples/mnist_pytorch.py#L62
> so we can leave savely "gpu": 1.
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/ray-project/ray/issues/3873#issuecomment-458491495>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AEUc5bFOAUQgKygCBnihLCLHdRu14Yvoks5vICRugaJpZM4aU24_>
> .
>


Resolving patch:
diff --git a/python/ray/tune/examples/mnist_pytorch.py b/python/ray/tune/examples/mnist_pytorch.py
--- a/python/ray/tune/examples/mnist_pytorch.py
+++ b/python/ray/tune/examples/mnist_pytorch.py
@@ -8,7 +8,6 @@
 import torch.nn.functional as F
 import torch.optim as optim
 from torchvision import datasets, transforms
-from torch.autograd import Variable
 
 # Training settings
 parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
@@ -120,7 +119,6 @@ def train(epoch):
         for batch_idx, (data, target) in enumerate(train_loader):
             if args.cuda:
                 data, target = data.cuda(), target.cuda()
-            data, target = Variable(data), Variable(target)
             optimizer.zero_grad()
             output = model(data)
             loss = F.nll_loss(output, target)
@@ -131,16 +129,17 @@ def test():
         model.eval()
         test_loss = 0
         correct = 0
-        for data, target in test_loader:
-            if args.cuda:
-                data, target = data.cuda(), target.cuda()
-            data, target = Variable(data, volatile=True), Variable(target)
-            output = model(data)
-            test_loss += F.nll_loss(
-                output, target, size_average=False).item()  # sum up batch loss
-            pred = output.data.max(
-                1, keepdim=True)[1]  # get the index of the max log-probability
-            correct += pred.eq(target.data.view_as(pred)).long().cpu().sum()
+        with torch.no_grad():
+            for data, target in test_loader:
+                if args.cuda:
+                    data, target = data.cuda(), target.cuda()
+                output = model(data)
+                # sum up batch loss
+                test_loss += F.nll_loss(output, target, reduction='sum').item()
+                # get the index of the max log-probability
+                pred = output.argmax(dim=1, keepdim=True)
+                correct += pred.eq(
+                    target.data.view_as(pred)).long().cpu().sum()
 
         test_loss = test_loss / len(test_loader.dataset)
         accuracy = correct.item() / len(test_loader.dataset)
@@ -176,7 +175,8 @@ def test():
                     "training_iteration": 1 if args.smoke_test else 20
                 },
                 "resources_per_trial": {
-                    "cpu": 3
+                    "cpu": 3,
+                    "gpu": int(not args.no_cuda)
                 },
                 "run": "train_mnist",
                 "num_samples": 1 if args.smoke_test else 10,
diff --git a/python/ray/tune/examples/mnist_pytorch_trainable.py b/python/ray/tune/examples/mnist_pytorch_trainable.py
--- a/python/ray/tune/examples/mnist_pytorch_trainable.py
+++ b/python/ray/tune/examples/mnist_pytorch_trainable.py
@@ -9,7 +9,6 @@
 import torch.nn.functional as F
 import torch.optim as optim
 from torchvision import datasets, transforms
-from torch.autograd import Variable
 
 from ray.tune import Trainable
 
@@ -127,7 +126,6 @@ def _train_iteration(self):
         for batch_idx, (data, target) in enumerate(self.train_loader):
             if self.args.cuda:
                 data, target = data.cuda(), target.cuda()
-            data, target = Variable(data), Variable(target)
             self.optimizer.zero_grad()
             output = self.model(data)
             loss = F.nll_loss(output, target)
@@ -138,18 +136,17 @@ def _test(self):
         self.model.eval()
         test_loss = 0
         correct = 0
-        for data, target in self.test_loader:
-            if self.args.cuda:
-                data, target = data.cuda(), target.cuda()
-            data, target = Variable(data, volatile=True), Variable(target)
-            output = self.model(data)
-
-            # sum up batch loss
-            test_loss += F.nll_loss(output, target, size_average=False).item()
-
-            # get the index of the max log-probability
-            pred = output.data.max(1, keepdim=True)[1]
-            correct += pred.eq(target.data.view_as(pred)).long().cpu().sum()
+        with torch.no_grad():
+            for data, target in self.test_loader:
+                if self.args.cuda:
+                    data, target = data.cuda(), target.cuda()
+                output = self.model(data)
+                # sum up batch loss
+                test_loss += F.nll_loss(output, target, reduction='sum').item()
+                # get the index of the max log-probability
+                pred = output.argmax(dim=1, keepdim=True)
+                correct += pred.eq(
+                    target.data.view_as(pred)).long().cpu().sum()
 
         test_loss = test_loss / len(self.test_loader.dataset)
         accuracy = correct.item() / len(self.test_loader.dataset)
@@ -188,7 +185,8 @@ def _restore(self, checkpoint_path):
                     "training_iteration": 1 if args.smoke_test else 20,
                 },
                 "resources_per_trial": {
-                    "cpu": 3
+                    "cpu": 3,
+                    "gpu": int(not args.no_cuda)
                 },
                 "run": TrainMNIST,
                 "num_samples": 1 if args.smoke_test else 20,


Test patch:
diff --git a/test/jenkins_tests/run_multi_node_tests.sh b/test/jenkins_tests/run_multi_node_tests.sh
--- a/test/jenkins_tests/run_multi_node_tests.sh
+++ b/test/jenkins_tests/run_multi_node_tests.sh
@@ -363,12 +363,11 @@ docker run --rm --shm-size=${SHM_SIZE} --memory=${MEMORY_SIZE} $DOCKER_SHA \
     --smoke-test
 
 docker run --rm --shm-size=${SHM_SIZE} --memory=${MEMORY_SIZE} $DOCKER_SHA \
-    python /ray/python/ray/tune/examples/mnist_pytorch.py \
-    --smoke-test
+    python /ray/python/ray/tune/examples/mnist_pytorch.py --smoke-test --no-cuda
 
 docker run --rm --shm-size=${SHM_SIZE} --memory=${MEMORY_SIZE} $DOCKER_SHA \
     python /ray/python/ray/tune/examples/mnist_pytorch_trainable.py \
-    --smoke-test
+    --smoke-test --no-cuda
 
 docker run --rm --shm-size=${SHM_SIZE} --memory=${MEMORY_SIZE} $DOCKER_SHA \
     python /ray/python/ray/tune/examples/genetic_example.py \