Repository: deeplearning4j/deeplearning4j
Issue: Poor GPU performance relative to CPU

This issue is similar to #2053, with a few key differences:
- I'm using a recurrent neural network.
- Performance is consistently and significantly _worse_ with the GPU backend (not merely comparable).
- I have a small dataset (16x9x308), which is probably at least partly to blame. However, the nsight profiling results with a larger dataset may suggest that DL4J can close the gap somewhat (see below).

I'm using the latest release of DeepLearning4J (0.5.0).

**Minimized Test Case**

``` java
package com.stottlerhenke.illuminate;

import java.util.List;

import com.stottlerhenke.illuminate.training.PerformanceListener;

import org.deeplearning4j.nn.api.OptimizationAlgorithm;
import org.deeplearning4j.nn.conf.*;
import org.deeplearning4j.nn.conf.layers.GravesLSTM;
import org.deeplearning4j.nn.conf.layers.RnnOutputLayer;
import org.deeplearning4j.nn.multilayer.MultiLayerNetwork;
import org.deeplearning4j.nn.weights.WeightInit;
import org.nd4j.linalg.api.ndarray.INDArray;
import org.nd4j.linalg.dataset.DataSet;
import org.nd4j.linalg.dataset.api.DataSetPreProcessor;
import org.nd4j.linalg.dataset.api.iterator.DataSetIterator;
import org.nd4j.linalg.factory.Nd4j;
import org.nd4j.linalg.lossfunctions.LossFunctions;

public class GpuProblem {

    public static int timeSeriesCount = 16;
    public static int inputCount = 9;
    public static int timeSteps = 308;
    public static int numHiddenNodes = 20;
    public static int truncatedBPTTLength = 100;
    public static int epochCount = 3;

    public static class MyDatasetIterator implements DataSetIterator {

        private DataSetPreProcessor preProcessor;

        int cursor = 0;

        @Override
        public boolean hasNext() {
            return cursor < epochCount;
        }

        @Override
        public DataSet next() {
            return next(1);
        }

        @Override
        public DataSet next(int num) {

            DataSet ds = createDataset();
            if (preProcessor != null)
                preProcessor.preProcess(ds);
            cursor += num;
            System.out.println("NEW CURSOR " + cursor);
            return ds;
        }

        private DataSet createDataset() {
            INDArray createDataSetinput = Nd4j.zeros(timeSeriesCount, inputCount, timeSteps);
            INDArray createDataSetlabels = Nd4j.zeros(timeSeriesCount, 2, timeSteps);
            return new DataSet(createDataSetinput, createDataSetlabels);
        }

        @Override
        public int totalExamples() {
            throw new UnsupportedOperationException();
        }

        @Override
        public int inputColumns() {
            return inputCount;
        }

        @Override
        public int totalOutcomes() {
            return 2;
        }

        @Override
        public boolean resetSupported() {
            return true;
        }

        @Override
        public void reset() {
            cursor = 0;
            System.out.println("RESET NEW CURSOR " + cursor);
        }

        @Override
        public int batch() {
            return timeSeriesCount;
        }

        @Override
        public int cursor() {
            throw new UnsupportedOperationException();
        }

        @Override
        public int numExamples() {
            throw new UnsupportedOperationException();
        }

        @Override
        public void setPreProcessor(DataSetPreProcessor preProcessor) {
            this.preProcessor = preProcessor;

        }

        @Override
        public DataSetPreProcessor getPreProcessor() {
            return this.preProcessor;
        }

        @Override
        public List<String> getLabels() {
            return null;
        }

    }

    public static void main(String[] args) {
        MultiLayerConfiguration.Builder builder =
                new NeuralNetConfiguration.Builder()
                        .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT).iterations(1)
                        .updater(Updater.ADAM).adamMeanDecay(0.9).adamVarDecay(0.999)
                        .regularization(true).l1(1e-4).dropOut(0.5)
                        .weightInit(WeightInit.XAVIER)
                        .learningRate(5e-4)
                        .list()
                        .layer(0, new GravesLSTM.Builder().nIn(inputCount).nOut(numHiddenNodes)
                                .activation("tanh").build())
                        .layer(1, new RnnOutputLayer.Builder().nIn(numHiddenNodes)
                                .activation("softmax")
                                .lossFunction(LossFunctions.LossFunction.MCXENT)
                                .nIn(numHiddenNodes).nOut(2).build())
                        .pretrain(false)
                        .backprop(true)
                        .backpropType(BackpropType.TruncatedBPTT)
                   .tBPTTBackwardLength(truncatedBPTTLength).tBPTTForwardLength(truncatedBPTTLength);

        MultiLayerNetwork net = new MultiLayerNetwork(builder.build());
        net.init();

        net.setListeners(new PerformanceListener(1));

        net.fit(new MyDatasetIterator());
    }
}
```

**Performance Results**
CPU

```
iteration 1; iteration time: 341 ms; samples/sec: 46.921; batches/sec: 2.933;
iteration 2; iteration time: 170 ms; samples/sec: 94.118; batches/sec: 5.882;
iteration 3; iteration time: 158 ms; samples/sec: 101.266; batches/sec: 6.329;
iteration 4; iteration time: 181 ms; samples/sec: 88.398; batches/sec: 5.525;
iteration 5; iteration time: 127 ms; samples/sec: 125.984; batches/sec: 7.874;
iteration 6; iteration time: 122 ms; samples/sec: 131.148; batches/sec: 8.197;
iteration 7; iteration time: 122 ms; samples/sec: 131.148; batches/sec: 8.197;
iteration 8; iteration time: 119 ms; samples/sec: 134.454; batches/sec: 8.403;
iteration 9; iteration time: 119 ms; samples/sec: 134.454; batches/sec: 8.403;
```

GPU

```
iteration 1; iteration time: 1667 ms; samples/sec: 9.598; batches/sec: 0.600;
iteration 2; iteration time: 1266 ms; samples/sec: 12.638; batches/sec: 0.790;
iteration 3; iteration time: 1241 ms; samples/sec: 12.893; batches/sec: 0.806;
iteration 4; iteration time: 1192 ms; samples/sec: 13.423; batches/sec: 0.839;
iteration 5; iteration time: 1204 ms; samples/sec: 13.289; batches/sec: 0.831;
iteration 6; iteration time: 1178 ms; samples/sec: 13.582; batches/sec: 0.849;
iteration 7; iteration time: 1137 ms; samples/sec: 14.072; batches/sec: 0.880;
iteration 8; iteration time: 1141 ms; samples/sec: 14.023; batches/sec: 0.876;
iteration 9; iteration time: 1183 ms; samples/sec: 13.525; batches/sec: 0.845;
```

I also tested with more data by increasing the number of time series from 16 to 16,000. With this change, the GPU does outperform the CPU (as expected); however, profiling with nsight shows only 9.1% GPU utilization (although I was only able to profile one iteration due to an issue with nsight that caused the test application to terminate early with an access violation, and the utilization percentage is probably unduly influenced by the time period before network training begins).


Changed files:
nd4j/nd4j-buffer/src/main/java/org/nd4j/linalg/api/buffer/BaseDataBuffer.java

diff --git a/nd4j/nd4j-buffer/src/main/java/org/nd4j/linalg/api/buffer/BaseDataBuffer.java b/nd4j/nd4j-buffer/src/main/java/org/nd4j/linalg/api/buffer/BaseDataBuffer.java
index e56a3abaa1..6660e2b6d2 100644
--- a/nd4j/nd4j-buffer/src/main/java/org/nd4j/linalg/api/buffer/BaseDataBuffer.java
+++ b/nd4j/nd4j-buffer/src/main/java/org/nd4j/linalg/api/buffer/BaseDataBuffer.java
@@ -822,8 +822,7 @@ public abstract class BaseDataBuffer implements DataBuffer {
 
     @Override
     public void assign(Number value) {
-        for (int i = 0; i < length(); i++)
-            assign(value, i);
+        assign(value, 0);
     }