Package: torchprep
TorchprepA CLI tool to prepare your Pytorch models for efficient inference. The only prerequisite is a model trained and saved withtorch.save(model_name, model_path). Seeexample.pyfor an example.Be warned:torchprepis an experimental tool so expect bugs, deprecations and limitations. That said if you like the project and would like to improve it please open up a Github issue!Install from sourceCreate a virtual environmentapt-getinstallpython3-venv
python3-mvenvvenvsourcevenv/bin/activateInstallpoetrysudopython3-mpipinstall-Upip
sudopython3-mpipinstall-Usetuptools
pipinstallpoetryInstalltorchprepcdtorchprep
poetryinstallInstall from Pypi (Coming soon)pipinstalltorchprepUsagetorchprepquantize--helpExample# Install example dependenciespipinstalltorchvisiontransformers# Download resnet examplepythonexample.py# quantize a cpu model with int8 on cpu and profile with a float tensor of shape [64,3,7,7]torchprepquantizemodels/resnet152.ptint8--input-shape64,3,7,7# profile a model for a 100 iterationstorchprepprofilemodels/resnet152.pt--iterations100--devicecpu--input-shape64,3,7,7# set omp threads to 1 to optimize cpu inferencetorchprepenv--devicecpu# Prune 30% of model weightstorchprepprunemodels/resnet152.pt--prune-amount0.3Available commandsUsage: torchprep [OPTIONS] COMMAND [ARGS]...

Options:
  --install-completion  Install completion for the current shell.
  --show-completion     Show completion for the current shell, to copy it or
                        customize the installation.
  --help                Show this message and exit.

Commands:
  distill        Create a smaller student model by setting a distillation...
  prune          Zero out small model weights using l1 norm
  env-variables  Set environment variables for optimized inference.
  fuse           Supports optimizations including conv/bn fusion, dropout...
  profile        Profile model latency 
  quantize       Quantize a saved torch model to a lower precision float...Usage instructions for a commandtorchprep <command> --helpUsage: torchprep quantize [OPTIONS] MODEL_PATH PRECISION:{int8|float16}

  Quantize a saved torch model to a lower precision float format to reduce its
  size and latency

Arguments:
  MODEL_PATH                [required]
  PRECISION:{int8|float16}  [required]

Options:
  --device [cpu|gpu]  [default: Device.cpu]
  --input-shape TEXT  Comma seperated input tensor shape
  --help              Show this message and exit.Create binariesTo create binaries and test them out locallypoetrybuild
pipinstall--user/path/to/wheelUpload to Pypipoetryconfigpypi-token.pypi<SECRET_KEY>
poetrypublish--buildRoadmapSupporting add custom model names and output pathsSupport multiple input tensors for models like BERT that expect a batch size and sequence lengthSupport multiple input tensor typesAutomatic distillation example: Reduce parameter count by 1/3torchprep distill model.pt 1/3Automated release with github actionsTensorRT, IPEX, AMP and autocast supportTraining aware optimizationsGet model input shape using fx instead of asking user for itRefactor profiling, loading and saving into seperate functionsMore environment variable setting and a way to reverse environment variables (e.g: save current ones in user file)