Package: tensorcom
TensorcomTensorcom is a way of loading training data into deep learning frameworks
quickly and portably. You can write a single data loading/augmentation
pipeline and train one or more jobs in the same or different frameworks
with it.Both Keras and PyTorch can use the PythonConnectionobject for input,
but MessagePack and ZMQ libraries exist in all major languages, making
it easy to write servers and input operators for any framework.Tensorcom replaces the use ofmultiprocessingin Python for that purpose.
Both use separate processes for loading and augmentation, but by making
the processes and communications explicit, you gain some significant advantages:the same augmentation pipeline can be used with different DL frameworksaugmentation processes can easily be run on multiple machinesoutput from a single automentation pipeline can be shared by many training jobsyou can start up and test the augmentation pipeline before you start the Dl jobsDL frameworks wanting to usetensorcomonly need a small library to handle inputUsingtensorcomfor training is very simple. First, start up a data server;
for Imagenet, there are two example jobs. Theserve-imagenet-dirprogram
illustrates how to use the standard PyTorch ImagenetDataLoaderto serve
training data:$ serve-imagenet-dir -d /data/imagenet -b 64 zpub://127.0.0.1:7880The server will give you information about the rate at which it serves image batches.
Your training loop then becomes very simple:training = tensorcom.Connection("zsub://127.0.0.1:7880", epoch=1000000)
    for xs, ys in training:
        train_batch(xs, ys)If you want multiple jobs for augmentation, just use more publishers using
Bash-style brace notation:zpub://127.0.0.1:788{0..3}andzsub://127.0.0.1:788{0..3}.Note that you can start up multiple training jobs connecting to the same server.Command Line ToolsThere are some command line programs to help with developing and debugging these
jobs:tensormon -- connect to a data server and monitor throughputtensorshow -- show images from input batchestensorstat -- compute statistics over input data samplesExamplesserve-imagenet-dir -- serve Imagenet data from a file system using PyTorchserve-imagenet-shards -- serve Imagenet from shards usingwebloaderkeras.ipynb -- simple example of using Keras with tensorcompytorch.ipynb -- simple example of using PyTorch with tensorcomZMQ URLsThere is no official standard for ZMQ URLs. This library uses the following notation:Socket types:zpush / zpull -- standard PUSH/PULL socketszrpush / zrpull -- reverse PUSH/PULL connections (PUSH socket is server / PULL socket connects)zpub / zsub -- standard PUB/SUB socketszrpub / zrsub -- reverse PUB/SUB connectionsThe pub/sub servers allow the same augmentation pipeline to be shared by multiple
learning jobs.Default transport is TCP/IP, but you can choose IPC as inzpush+ipc://mypath.Connection ObjectsThe major way of interacting with the library is through theConnectionobject.
It simply gives you an iterator over training samples.EncodingsData is encoded in a simple binary tensor format; seecodec.pyfor details.
The same format can also be used for saving and loading lists of
tensors from disk (extension:.ten).
Data is encoded on 64 byte aligned boundaries to allow easy memory
mapping and direct use by CPUs and GPUs.