Text classification unexpectedly slow

Xia_Yandi: I wrote a simple demo for short text classification but the speed is unexpectedly slow. When I tried to find out where the bottleneck is, it turns out to be intractable. At first, the bottleneck is this line: However, after commenting out the above line, it slows down at these lines in get_batch() function: Is there any problem in the code? I ran this script on GPU (Titan X) with cuda 8.0, python 2.7, ubuntu 16 and pytorch was installed by pip. The data was randomly generated. The code is attached below:

apaszke: If you want to reliably benchmark the compute time of your model do this: <SCODE>torch.cuda.synchronize()
start = # get start time
output = model(input)
torch.cuda.synchronize()
end = # get end time
<ECODE> Can you expect what “unexpectedly slow” means? What is the runtime and what did you expect? It seems that you’re using a convolutional kernel size of 3x300, and that will be incredibly costly to compute (especially with 200 output channels).

apaszke: 

Xia_Yandi: For the slow part, I time the training on one batch and print it in the code. It shows that the training speed is around 13 instances per sec. With larger graph using tensorflow, the same machine can easily achieve thousands of instances per sec. In the pytorch code above, every batch takes about 8 sec. Thanks.

apaszke: Try to use more batches, the first one will be always slower, because lot of time will be spent allocating memory that will be cached for subsequent runs. Are the batch sizes and model parameters exactly the same as in tensorflow? I can try to run your model tomorrow, but it would be helpful if you could also provide me with the code for tensorflow implementation.

Xia_Yandi: The 8 sec is not the first batch, it is on average. The batch size for tensorflow is 1024. There are four parallel convolutional layers for my tensorflow model, the kernel sizes are  1X250, 3X250, 4X250, 5X250. And then some hidden layers. That is a much larger graph than the example I showed above. Sorry, the tensorflow code has too many dependencies, I don’t think I can simply show it here. Thank you very much!

apaszke: And what is the time per batch or element that you can achieve in TensorFlow?

Xia_Yandi: here is the code: Thank you very much!

kim.seonghyeon: Seems like that there are some problems or pitfalls in Conv2d on GPU. When I used in_channels = 1, out_channels = 200, kernel height = 3, kernel width = 300 (original setting) then backward was very slow. I changed it to in channels = 300, out channels = 200, kernel height = 1, kernel width = 3 then model works well. (I got 9500/sec on GTX 1080 8) ) This seems like GPU-specific problem because no problem occurred when ran on CPU.

apaszke: 

Xia_Yandi: If it is a GPU-specific problem, then why tensorflow code works well with the same kernel size? Thanks

Xia_Yandi: So there is no way to speed it up other than reshaping the input? Thanks

apaszke: It’s not a GPU specific problem, we’re probably just using different cuDNN calls to select algorithms, and for some reason they fail to pick the fast ones for this use case. I don’t know if there’s no other way, it will take me some time until I properly benchmark it and see what’s the issue. If you want to use it now, just reshape the input.

Xia_Yandi: Thank you very much!

apaszke: With cudnn disabled the training runs at 1k samples/sec for me. With it slows down to 12.

apaszke: <SCODE>torch.backends.cudnn.benchmark = True
<ECODE> This is a flag that controls cudnn benchmark mode and should be used only if every forward pass uses the same data dimensions. After this your code runs @ 7k/s for training and 64k/s for test on my machine.

apaszke: 

Xia_Yandi: Thanks!

apaszke: 

Xia_Yandi: If I set this: What do you mean by same data dimension? Thanks.