Code change:
A common issue - for both new and more seasoned users, including myself - is how to write nice device-agnostic code. There seems to be several potential solutions (using a global CUDA flag, always calling , using etc.), used by various people in various bits of code. Ideally, we should have "best practices" documented somewhere, like ; or at least some place where we can link to.
Alternative proposal - we provide a method that deals with the most common use cases (tensors, variables and networks). We could have something similar to Chainer's , something like a , etc. We're not going to be "write once, run everywhere", but we can probably catch most use cases. Pros: It makes it much simpler for people writing CPU/1 GPU code (presumably a majority of users). Depending on the approach it'll probably be fine for multiple GPUs too. Cons: An abstraction of this form isn't foolproof, but I can't immediately think of any major issues. Setting everything to CUDA by default is probably going to break things, but a wrapper, being more manual, can be used selectively enough.
I agree we should figure out a way to make writing device-agnostic code easier. I don't think fakecuda is a good idea, it's too magical and has global side effects. Here's a list of options I see, along with some comments. In general I'm not 100% happy with any of those, but I don't have any better ideas right now. Example: I think that sooner or later we should add arg to functions anyway, so it seems like a good enough extension. I think I'm fine with this option too, but I'm afraid you would either need a single global module captured inside functions, or use it everywhere inside. In general it can be simulated using the approach using partials (of course this is an inefficient implementation, but you get the idea): This is fine, except that you have to call it a lot and it makes the code super verbose. A similar alternative that people sometimes use now is this: This is really the best option, as it will give you a tensor of exactly the same type and on exactly the same device as the source. This is usually the desired behaviour, which is why we use it all over the place in the core library. On the other hand, it feels a bit unnatural, and is annoying when you do more complicated things, because you first construct the tensor and only then fill it with the data. and helps a lot with that
Thanks for more in-depth feedback on some options. For existing practices, highlighting , and are probably the way to go. In terms of what to do when there aren't existing tensors, what I see as the main target is the . Precisely, can we do and without using the if statement? The cast/dtype solution seems good for tensors, but anything we can do for modules?
I've put together some of this and other stuff I know into "best practices" documentation in I think we still want to think about future possibilities, but there's plenty that's worth noting down now for users.
It would indeed be amazing if PyTorch could make it simpler to write device agnostic code. I really like your first suggestion. Just a quick note from my (applied) perspective: maybe it would be possible to split up the and the into two attributes? First of all, I think this would make it easier to handle cases where tensors are on different gpus and second of all, it would allow to write code like this: ((5, 5), , ) # Returns on same device as x Don't know how practical this is though.
one thing I like about suggestion is that it makes it explicit and obvious when something is happening on cuda, rather than requiring users to know that non-negative device numbers refer to cuda. Maybe this is a bad idea, but what about also taking an optional kw in every function that takes a but to have both and non- dtypes? So passing would give you a cuda float tensor on the current GPU, passing would give you a CPU float, and passing would give you a cuda tensor on GPU 5. Obviously we'd need error checking to check things like non- devices don't have non-negative s specified (which is kind of silly because if the device is specified it tells you everything), but it does seem to make the common case of not specifying a device more clear.
Speaking only as an end-use with no particular insight into how difficult it would be to code: These suggestions make me feel like end users will end up having to think more about device semantics, not less. I'd like to see the API move away from having the tensor type imply the storage location entirely. Have a context manager that sets cpu vs. cuda, default device, etc. Something like: Then once this exists, class wraps its calls to the with an appropriate context, as do Modules. I think that this results in everything Just Working(tm).
I'm not convinced that making it easy for end users to avoid thinking about device semantics is the correct goal. The different devices have different performance characteristics, amount of memory/cache, incur data transfer costs/synchronizations, etc. My worry is that by papering over these issues, we make it really easy to write non-performant code and really difficult to figure out why. This discussion is similar to a discussion in the distributed systems community: is a good discussion of why trying to unify all interfaces might not be the best idea (there are admittedly additional complexities in distributed computing like partial failures). Anyway, that's why I like making it obvious when there are cuda storage / device transfers going on, because those are obvious places to look at performance.
I'm not an expert on any of this, but about the only two things that bug me about using is when I have to cast are I don't have a prototypical tensor at hand (e.g. in a module it feels unnatural to use so it might be neat to have a "canonical tensor" in modules that reacts to .cuda() and whose is exposed as or so. I want a different type is just a bit much. Other than that, I think using it is cool, I think that there is not much advantage in over But this is only from my limited experience.
Regarding the "canonical tensor" it's hard, because the module can have a lot of different parameters that have different types and are on different devices. I agree that chains are awful. I think (and similar for all other types) would be a convenient helper.
Oh yes, something like would be cool (or ). For the "canonical tensor", I imagine that, for basic applications, having something that "does the right thing" whether called before or after would be great. For more advanced applications, the user could create their own set of prototype tensors and just do , or so. But that is just pure imagination, as I don't have multiple GPUs. :slightlysmilingface:
there are several potential goals here, and they are somewhat conflicting. One is to make CPU/GPU code seamless for end users, with just one flag somewhere. On the other hand, that's likely to results in less-performant code and harder debugging. It seems like the conversation is going towards device-aware code, but making it a little less clunky in places. Sounds like is a good step forward, and was probably in the pipeline anyway?
a latecomer here: i agree that it's not a good idea to hide things behind the curtain from users. though, in many cases, it might be easier to to have a global behavior of ignoring, e.g., gpu (i.e., make .cuda() do nothing). could we at least give some kind of global switch allowing this behiavour?
Closing as 0.4 has plenty of options for this

Related discussion:
text: Best practices -------------- <ins> Device-agnostic code ^^^^^^^^^^^^^^^^^^^^ Due to the structure of PyTorch, you may need to explicitly write device-agnostic (CPU or GPU) code; an example may be creating a new tensor as the initial hidden state of a recurrent neural network. The first step is to determine whether the GPU should be used or not. A common pattern is to use Python's `argparse` module to read in user arguments, and have a flag that can be used to disable CUDA, in combination with `torch.cuda.is_available()`. In the following, `args.cuda` results in a flag that can be used to cast tensors and modules to CUDA if desired:: import argparse import torch parser = argparse.ArgumentParser(description='PyTorch Example') parser.add_argument('--disable-cuda', action='store_true', help='Disable CUDA') args = parser.parse_args() args.cuda = not args.disable_cuda and torch.cuda.is_available() If modules or tensors need to be sent to the GPU, `args.cuda` can be used as follows:: x = torch.Tensor(8, 42) net = Network() if args.cuda: x = x.cuda() net.cuda() When creating tensors, an alternative to the if statement is to have a default datatype defined, and cast all tensors using that. An example when using a dataloader would be as follows:: dtype = torch.cuda.FloatTensor for i, x in enumerate(train_loader): x = Variable(x.type(dtype)) When working with multiple GPUs on a system, you can use the `CUDA_VISIBLE_DEVICES` environment flag to manage which GPUs are available to PyTorch. To manually control which GPU a tensor is created on, the best practice is to use the `torch.cuda.device()` context manager:: print("Outside device is 0")  # On device 0 (default in most scenarios) with torch.cuda.device(1): print("Inside device is 1")  # On device 1 print("Outside device is still 0")  # On device 0 If you have a tensor and would like to create a new tensor of the same type on the same device, then you can use the `.new()` function, which acts the same as a normal tensor constructor. Whilst the previously mentioned methods depend on the current GPU context, `new()` preserves the device of the original tensor. This is the recommended practice when creating modules in which new tensors/variables need to be created internally during the forward pass:: x_cpu = torch.FloatTensor(1) x_gpu = torch.cuda.FloatTensor(1) x_cpu_long = torch.LongTensor(1) y_cpu = x_cpu.new(8, 10, 10).fill_(0.3) y_gpu = x_gpu.new(x_gpu.size()).fill_(-5) y_cpu_long = x_cpu_long.new([[1, 2, 3]]) If you want to create a tensor of the same type and size of another tensor, and fill it with either ones or zeros, `torch.ones_like()` or `torch.zeros_like()` are provided as more convenient functions (which also preserve device):: x_cpu = torch.FloatTensor(1) x_gpu = torch.cuda.FloatTensor(1) y_cpu = torch.ones_like(x_cpu) y_gpu = torch.zeros_like(x_gpu) </ins> Use pinned memory buffers ^^^^^^^^^^^^^^^^^^^^^^^^^