Repository: pytorch/audio

Issue:
GPU support for Interactive ASR example
## 🐛 Bug

In the [Interactive ASR Run Section](https://github.com/pytorch/audio/tree/master/examples/interactive_asr#run) for the file and microphone examples to run, [source in utils.py](https://github.com/pytorch/audio/blob/master/examples/interactive_asr/utils.py#L107) needs to be changes to `source.cuda()` for GPU support.
However the testcase works well as it is, and gives error when `source` is changed to `source.cuda()`.

## To Reproduce

Steps to reproduce the behavior:
```
root@9797901afdf8:/workspace/audio/examples# INPUT_FILE=interactive_asr/data/sample.wav
root@9797901afdf8:/workspace/audio/examples# python -m interactive_asr.asr interactive_asr/data --input_file $INPUT_FILE --max-tokens 10000000 --nbest 1 \
>   --path interactive_asr/data/model.pt --beam 40 --task speech_recognition \
>   --user-dir interactive_asr/fairseq/examples/speech_recognition
| dictionary: 5001 types
/workspace/audio/examples/interactive_asr/utils.py:149: UserWarning: utils.load_ensemble_for_inference is deprecated. Please use checkpoint_utils.load_model_ensemble instead.
  model_arg_overrides=eval(args.model_overrides),  # noqa
READY!
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/audio/examples/interactive_asr/asr.py", line 47, in <module>
    cli_main()
  File "/workspace/audio/examples/interactive_asr/asr.py", line 43, in cli_main
    main(args)
  File "/workspace/audio/examples/interactive_asr/asr.py", line 27, in main
    transcription_time, transcription = transcribe_file(args, task, generator, models, sp, tgt_dict)
  File "/workspace/audio/examples/interactive_asr/utils.py", line 173, in transcribe_file
    waveform, args, task, generator, models, sp, tgt_dict
  File "/workspace/audio/examples/interactive_asr/utils.py", line 109, in transcribe
    hypos = task.inference_step(generator, models, sample)
  File "/workspace/audio/examples/interactive_asr/fairseq/fairseq/tasks/fairseq_task.py", line 260, in inference_step
    return generator.generate(models, sample, prefix_tokens=prefix_tokens)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/workspace/audio/examples/interactive_asr/fairseq/fairseq/sequence_generator.py", line 113, in generate
    return self._generate(model, sample, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/workspace/audio/examples/interactive_asr/fairseq/fairseq/sequence_generator.py", line 152, in _generate
    encoder_outs = model.forward_encoder(encoder_input)
  File "/opt/conda/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/workspace/audio/examples/interactive_asr/fairseq/fairseq/sequence_generator.py", line 540, in forward_encoder
    return [model.encoder(**encoder_input) for model in self.models]
  File "/workspace/audio/examples/interactive_asr/fairseq/fairseq/sequence_generator.py", line 540, in <listcomp>
    return [model.encoder(**encoder_input) for model in self.models]
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/audio/examples/interactive_asr/fairseq/examples/speech_recognition/models/vggtransformer.py", line 331, in forward
    x = self.conv_layers[layer_idx](x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/audio/examples/interactive_asr/fairseq/fairseq/modules/vggblock.py", line 115, in forward
    x = self.layers[i](x)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 343, in forward
    return self.conv2d_forward(input, self.weight)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 340, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same
```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior
```
root@9797901afdf8:/workspace/audio/examples# python -m interactive_asr.asr interactive_asr/data --input_file $INPUT_FILE --max-tokens 10000000 --nbest 1   --path interactive_asr/data/model.pt --beam 40 --task speech_recognition   --user-dir interactive_asr/fairseq/examples/speech_recognition
| dictionary: 5001 types
/workspace/audio/examples/interactive_asr/utils.py:149: UserWarning: utils.load_ensemble_for_inference is deprecated. Please use checkpoint_utils.load_model_ensemble instead.
  model_arg_overrides=eval(args.model_overrides),  # noqa
READY!
transcription: [['THE QUICK BROWN FOX JUMPS OVER THE LAZY DOG']]
transcription_time: 1.877535104751587
```

## Environment

 - I am running this example on NVIDIA NGC Pytorch container version 19.09
 - This is the I script used to setup the environment:
```
NAME=asr
PORT=8877
docker pull nvcr.io/nvidia/pytorch:19.09-py3
docker run --gpus all -d --ipc=host -it \
        -p $PORT:$PORT \
        --device /dev/snd \
        --env ALSA_CARD=Headset \
        --name $NAME nvcr.io/nvidia/pytorch:19.09-py3
docker exec $NAME /bin/sh -c 'apt-get update && apt-get -y upgrade && \
        apt-get -y install libsox-dev portaudio19-dev alsa-utils && \
docker exec -itw /workspace $NAME /bin/sh -c '\
        git clone --recursive https://github.com/pytorch/audio.git'
docker exec -w /workspace/audio/examples $NAME /bin/sh -c '\
        pip install --no-cache-dir ../ pyaudio sentencepiece && \
        git clone https://github.com/pytorch/fairseq interactive_asr/fairseq && \
        pushd interactive_asr/fairseq && \
        pip install --editable . && \
        popd && \
        echo "Downloading data..." && \
        wget -qO interactive_asr/data/dict.txt https://download.pytorch.org/models/audio/dict.txt && \
        wget -qO interactive_asr/data/spm.model https://download.pytorch.org/models/audio/spm.model && \
        wget -qO interactive_asr/data/model.pt https://download.pytorch.org/models/audio/checkpoint_avg_60_80.pt && \
        echo "Done"'
```
 - `>>> torchaudio.__version__ '0.4.0a0+ce1f8aa'`

Output from [environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py):
```Collecting environment information...
PyTorch version: 1.2.0a0+afb7a16
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.14.0

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: 
GPU 0: Tesla V100-DGXS-16GB
GPU 1: Tesla V100-DGXS-16GB
GPU 2: Tesla V100-DGXS-16GB
GPU 3: Tesla V100-DGXS-16GB

Nvidia driver version: 418.87.01
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip] msgpack-numpy==0.4.3.2
[pip] numpy==1.16.4
[pip] torch==1.2.0a0+afb7a16
[pip] torchaudio==0.4.0a0+ce1f8aa
[pip] torchtext==0.4.0
[pip] torchvision==0.3.0a0
[conda] magma-cuda100             2.5.0                         1    local
[conda] mkl                       2019.1                      144  
[conda] mkl-include               2019.1                      144  
[conda] nomkl                     3.0                           0  
[conda] torch                     1.2.0a0+afb7a16          pypi_0    pypi
[conda] torchaudio                0.4.0a0+ce1f8aa          pypi_0    pypi
[conda] torchtext                 0.4.0                    pypi_0    pypi
[conda] torchvision               0.3.0a0                  pypi_0    pypi
```


Discussion:
Thanks for reporting this! Please feel free to submit a pull request for this :)

Resolving patch:
diff --git a/examples/interactive_asr/utils.py b/examples/interactive_asr/utils.py
--- a/examples/interactive_asr/utils.py
+++ b/examples/interactive_asr/utils.py
@@ -55,8 +55,9 @@ def check_args(args):
 
 def process_predictions(args, hypos, sp, tgt_dict):
     res = []
+    device = torch.device("cuda:0" if torch.cuda.is_available() and not args.cpu else "cpu")
     for hypo in hypos[: min(len(hypos), args.nbest)]:
-        hyp_pieces = tgt_dict.string(hypo["tokens"].int().cpu())
+        hyp_pieces = tgt_dict.string(hypo["tokens"].int().to(device))
         hyp_words = sp.DecodePieces(hyp_pieces.split())
         res.append(hyp_words)
     return res
@@ -96,7 +97,8 @@ def calcMN(features):
 def transcribe(waveform, args, task, generator, models, sp, tgt_dict):
     num_features = 80
     output = torchaudio.compliance.kaldi.fbank(waveform, num_mel_bins=num_features)
-    output_cmvn = calcMN(output.cpu().detach())
+    device = torch.device("cuda:0" if torch.cuda.is_available() and not args.cpu else "cpu")
+    output_cmvn = calcMN(output.to(device).detach())
 
     # size (m, n)
     source = output_cmvn
@@ -151,7 +153,7 @@ def setup_asr(args, logger):
     optimize_models(args, use_cuda, models)
 
     # Initialize generator
-    generator = task.build_generator(args)
+    generator = task.build_generator(models, args)
 
     sp = spm.SentencePieceProcessor()
     sp.Load(os.path.join(args.data, "spm.model"))


Test patch: