Package: koila
🐨 KoilaKoila solvesCUDA error: out of memory errorpainlessly.
Fix it with just one line of code, and forget it.🚀 Features🙅 PreventsCUDA error: out of memory errorwith one single line of code.🦥 Lazily evaluates pytorch code to save computing power.✂️ Automatically splits along the batch dimension to more GPU friendly numbers (2's powers) to speed up the execution.🤏 Minimal API (wrapping all inputs will be enough).🤔 Why Koila?Ever encounteredRuntimeError: CUDA error: out of memory?
We all lovePyTorchbecause of its speed, efficiency, and transparency, but that means it doesn't do extra things. Things like preventing a very common error that has been bothering many users since2017.This library aims to prevent that by being a light-weight wrapper over nativePyTorch. When a tensor is wrapped, the libraryautomatically computes the amount of remaining GPU memory and uses the right batch size, saving everyone from having to manually fine-tune the batch size whenever a model is used.Also, the library automatically uses the right batch size to GPU. Did you know that using bigger batches doesn't always speed up processing? It's handled automatically in this library too.BecauseKoilacode isPyTorchcode, as it runsPyTorchunder the hood, you can use both together without worrying compatibility.Oh, and all that in 1 line of code! 😊⬇️ InstallationKoilais available onPyPI. To install, run the following command.pipinstallkoila🏃 Getting startedThe usage is dead simple. For example, you have the followingPyTorchcode (copied fromPyTorch'stutorial)Define the input, label, and model:# A batch of MNIST imageinput=torch.randn(8,28,28)# A batch of labelslabel=torch.randn(0,10,[8])classNeuralNetwork(Module):def__init__(self):super(NeuralNetwork,self).__init__()self.flatten=Flatten()self.linear_relu_stack=Sequential(Linear(28*28,512),ReLU(),Linear(512,512),ReLU(),Linear(512,10),)defforward(self,x):x=self.flatten(x)logits=self.linear_relu_stack(x)returnlogitsDefine the loss function, calculate output and losses.loss_fn=CrossEntropyLoss()# Calculate lossesout=nn(t)loss=loss_fn(out,label)# Backward passnn.zero_grad()loss.backward()Ok. How to adapt the code to useKoila's features?You add this line of code:# Wrap the input tensor and label tensor.# If a batch argument is provided, that dimension of the tensor would be treated as the batch.# In this case, the first dimension (dim=0) is used as batch's dimension.(input,label)=lazy(input,label,batch=0)Done. You will not run out of memory again.Seeexamples/getting-started.pyfor the full example.🏋️ How does it work under the hood?CUDA error: out of memorygenerally happens in forward pass, because temporary variables will need to be saved in memory.Koilais a thin wrapper aroundPyTorch. It is inspired by TensorFlow's static/lazy evaluation. By building the graph first, and run the model only when necessarily, the model has access to all the information necessarily to determine how much resources is really need to compute the model.In terms of memory usage, onlyshapes of temporary variables are required to calculate the memory usage of those variables used in the model. For example,+takes in two tensors with equal sizes, and outputs a tensor with a size equal to the input size, andlogtakes in one tensor, and outputs another tensor with the same shape. Broadcasting makes it a little more complicated than that, but the general ideas are the same. By tracking all these shapes, one could easily tell how much memory is used in a forward pass. And select the optimal batch size accordingly.🐌 It sounds slow. Is it?NO. Indeed, calculating shapes and computing the size and memory usage sound like a lot of work. However, keep in mind that even a gigantic model like GPT-3, which has 96 layers, has only a few hundred nodes in its computing graph. BecauseKoila's algorithms run in linear time, any modern computer will be able to handle a graph like this instantly.Most of the computing is spent on computing individual tensors, and transferring tensors across devices. And bear in mind that those checks happen in vanillaPyTorchanyways. So no, not slow at all.🔊 How to pronounce koila?This project was originally namedkoala, the laziest species in the world, and this project is about lazy evaluation of tensors. However, as that name is taken onPyPI, I had no choice but to use another name.Koilais a word made up by me, pronounced similarly tovoila(It's a French word), so sounds like koala.⭐ Give me a star!If you like what you see, please consider giving this a star (★)!🏗️ Why did I build this?Batch size search is not new. In fact, the mighty popularPyTorch Lightninghas it. So why did I go through the trouble and build this project?PyTorch Lightning's batch size search is deeply integrated in its own ecosystem. You have to use itsDataLoader, subclass from their models, and train your models accordingly. While it works well with supervised learning tasks, it's really painful to use in a reinforcement learning task, where interacting with the environment is a must.In comparison, becauseKoilais a super lightweight PyTorch wrapper, it works when PyTorch works, thus providing maximum flexibility and minimal changes to existing code.📝 Todos😌 Simplify internal workings even further. (Especially interaction betweenTensors andLazyTensors).🧩 Provide an extensible API to write custom functions for the users.🍪 Work with multiple GPUs.🚧 WarningThe code works on many cases, but it's still a work in progress. This is not (yet) a fullyPyTorchcompatible library due to limited time. Avoid using it in production environments!🥰 ContributingWe take openness and inclusiveness very seriously. We have adopted the followingCode of Conduct.