Package: ffcv
Fast Forward Computer Vision: train models at a fraction of the cost with accelerated data loading![install]
[quickstart]
[features]
[docs]
[support slack]
[homepage]Maintainers:Guillaume Leclerc,Andrew IlyasandLogan Engstromffcvis a drop-in data loading system that dramatically increases data throughput in model training:Train an ImageNet modelon one GPU in 35 minutes (98¢/model on AWS)Train a CIFAR-10 modelon one GPU in 36 seconds (2¢/model on AWS)Train a$YOUR_DATASETmodel$REALLY_FAST(for$WAY_LESS)Keep your training algorithm the same, just replace the data loader! Look at these speedups:ffcvalso comes prepacked withfast, simple codeforstandard vision benchmarks:InstallationLinuxconda create -y -n ffcv python=3.9 cupy pkg-config compilers libjpeg-turbo opencv pytorch torchvision cudatoolkit=11.3 numba -c pytorch -c conda-forge
conda activate ffcv
pip install ffcvTroubleshooting note: if the above commands result in a package conflict error, try runningconda config --env --set channel_priority flexiblein the environment and rerunning the installation command.WindowsInstallopencv4Add..../opencv/build/x64/vc15/binto PATH environment variableInstalllibjpeg-turbo, download libjpeg-turbo-x.x.x-vc64.exe, not gcc64Add..../libjpeg-turbo64/binto PATH environment variableInstallpthread, download last release.zipAfter unzip, rename Pre-build.2 folder to pthreadOpenpthread/include/pthread.h, and add the code below to the top of the file.#define HAVE_STRUCT_TIMESPECAdd..../pthread/dllto PATH environment variableInstallcupydepending on your CUDA Toolkit version.pip install ffcvCitationIf you use FFCV, please cite it as:@misc{leclerc2022ffcv,
    author = {Guillaume Leclerc and Andrew Ilyas and Logan Engstrom and Sung Min Park and Hadi Salman and Aleksander Madry},
    title = {{FFCV}: Accelerating Training by Removing Data Bottlenecks},
    year = {2022},
    howpublished = {\url{https://github.com/libffcv/ffcv/}},
    note = {commit xxxxxxx}
}(Make sure to replace xxxxxxx above with the hash of the commit used!)QuickstartAccelerateanylearning system withffcv.
First,
convert your dataset intoffcvformat (ffcvconverts both indexed PyTorch datasets andWebDatasets):fromffcv.writerimportDatasetWriterfromffcv.fieldsimportRGBImageField,IntField# Your dataset (`torch.utils.data.Dataset`) of (image, label) pairsmy_dataset=make_my_dataset()write_path='/output/path/for/converted/ds.beton'# Pass a type for each data fieldwriter=DatasetWriter(write_path,{# Tune options to optimize dataset size, throughput at train-time'image':RGBImageField(max_resolution=256,jpeg_quality=jpeg_quality),'label':IntField()})# Write datasetwriter.from_indexed_dataset(my_dataset)Then replace your old loader with theffcvloader at train time (in PyTorch,
no other changes required!):fromffcv.loaderimportLoader,OrderOptionfromffcv.transformsimportToTensor,ToDevice,ToTorchImage,Cutoutfromffcv.fields.decodersimportIntDecoder,RandomResizedCropRGBImageDecoder# Random resized cropdecoder=RandomResizedCropRGBImageDecoder((224,224))# Data decoding and augmentationimage_pipeline=[decoder,Cutout(),ToTensor(),ToTorchImage(),ToDevice(0)]label_pipeline=[IntDecoder(),ToTensor(),ToDevice(0)]# Pipeline for each data fieldpipelines={'image':image_pipeline,'label':label_pipeline}# Replaces PyTorch data loader (`torch.utils.data.Dataloader`)loader=Loader(write_path,batch_size=bs,num_workers=num_workers,order=OrderOption.RANDOM,pipelines=pipelines)# rest of training / validation proceeds identicallyforepochinrange(epochs):...See herefor a more detailed guide to deployingffcvfor your dataset.Prepackaged Computer Vision BenchmarksFrom gridding to benchmarking to fast research iteration, there are many reasons
to want faster model training. Below we present premade codebases for training
on ImageNet and CIFAR, including both (a) extensible codebases and (b)
numerous premade training configurations.ImageNetWe provide a self-contained script for training ImageNet fast.
Above we plot the training time versus
accuracy frontier, and the dataloading speeds, for 1-GPU ResNet-18 and 8-GPU
ResNet-50 alongside a few baselines.Link to Configtop_1top_5# EpochsTime (mins)ArchitectureSetupLink0.7840.9418877.2ResNet-508 x A100Link0.7800.9375649.4ResNet-508 x A100Link0.7720.9324035.6ResNet-508 x A100Link0.7660.9273228.7ResNet-508 x A100Link0.7560.9212421.7ResNet-508 x A100Link0.7380.9081614.9ResNet-508 x A100Link0.7240.90388187.3ResNet-181 x A100Link0.7130.89956119.4ResNet-181 x A100Link0.7060.8944085.5ResNet-181 x A100Link0.7000.8893268.9ResNet-181 x A100Link0.6880.8812451.6ResNet-181 x A100Link0.6690.8681635.0ResNet-181 x A100Train your own ImageNet models!You canuse our training script and premade configurationsto train any model seen on the above graphs.CIFAR-10We also include premade code for efficient training on CIFAR-10 in theexamples/directory, obtaining 93% top1 accuracy in 36 seconds on a single A100 GPU
(without optimizations such as MixUp, Ghost BatchNorm, etc. which have the
potential to raise the accuracy even further). You can find the training scripthere.FeaturesComputer vision or not, FFCV can help make training faster in a variety of
resource-constrained settings!
Ourperformance guidehas a more detailed account of the ways in which FFCV can adapt to different
performance bottlenecks.Plug-and-play with any existing training code: Rather than changing
aspects of model training itself, FFCV focuses on removingdata bottlenecks,
which turn out to be a problem everywhere from neural network training to
linear regression. This means that:FFCV can be introduced into any existing training code in just a few
lines of code (e.g., just swapping out the data loader and optionally the
augmentation pipeline);You don't have to change the model itself to make it faster (e.g., feel
free to analyze modelswithoutCutMix, Dropout, momentum scheduling, etc.);FFCV can speed up a lot more beyond just neural network training---in
fact, the more data-bottlenecked the application (e.g., linear regression,
bulk inference, etc.), the faster FFCV will make it!See ourGetting startedguide,Example walkthroughs, andCode examplesto see how easy it is to get started!Fast data processing without the pain: FFCV automatically handles data
reading, pre-fetching, caching, and transfer between devices in an extremely
efficiently way, so that users don't have to think about it.Automatically fused-and-compiled data processing: By either usingpre-writtenFFCV transformations
oreasily writing custom ones,
users can
take advantage of FFCV's compilation and pipelining abilities, which will
automatically fuse and compile simple Python augmentations to machine code
usingNumba, and schedule them asynchronously to avoid
loading delays.Load data fast from RAM, SSD, or networked disk: FFCV exposes
user-friendly options that can be adjusted based on the resources
available. For example, if a dataset fits into memory, FFCV can cache it
at the OS level and ensure that multiple concurrent processes all get fast
data access. Otherwise, FFCV can use fast process-level caching and will
optimize data loading to minimize the underlying number of disk reads. SeeThe Bottleneck Doctorguide for more information.Training multiple models per GPU: Thanks to fully asynchronous
thread-based data loading, you can now interleave training multiple models on
the same GPU efficiently, without any data-loading overhead. Seethis guidefor more info.Dedicated tools for image handling: All the features above work are
equally applicable to all sorts of machine learning models, but FFCV also
offers some vision-specific features, such as fast JPEG encoding and decoding,
storing datasets as mixtures of raw and compressed images to trade off I/O
overhead and compute overhead, etc. See theWorking with imagesguide for
more information.ContributorsGuillaume LeclercLogan EngstromAndrew IlyasSam ParkHadi Salman