Package: matorage
matorageAn efficient way to store/load and manage dataset, model and optimizer for deep learning with matorage!Matorage is tensor(multidimensional matrix) object storage manager for deep learning framework(Pytorch, Tensorflow V2, Keras).FeaturesBoilerplated data pipeline for dataset, model and optimizer.High performance on tensor storageFor researchers who need to focus on model training:Support storing data in pre-processed Tensor(multidimensional matrix), eliminate training time.Reduce storage space through multiple compression methods.Manage data and models while trainingFor AI Developer who need to focus on creating data pipeline:Concurrency data save & loadCompatible with object storage such as MinIO, S3Generate pipeline from user endpoints data.Quick Start with Pytorch ExampleFor an example of tensorflow, refer to the detailed document.
If you want to see the full code, see belowPytorch Mnist ExampleTensorflow Mnist ExampleSQuAD 1.1/2.0 ExampleContent0. Install matorage with pip1. Set up Minio Server with docker2. Save pre-processed dataset3. Load dataset from matorage4. Save & Load Model when training5. Save & Load Optimizer when trainingUnittest0. Install matorage with pip$pipinstallmatorage1. Set up Minio Server with dockerquick start with NAS(network access storage) using docker
It can be managed through the web through the addresshttp://127.0.0.1:9000/, and security is managed throughMINIO_ACCESS_KEYandMINIO_SECRET_KEY.$mkdir~/shared# create nas storage folder$dockerrun-it-p9000:9000\--restartalways-e\"MINIO_ACCESS_KEY=minio"-e\"MINIO_SECRET_KEY=miniosecretkey"\-v~/shared:/container/vol\minio/miniogatewaynas/container/vol2. Save pre-processed datasetFirst, create aDataConfigby importing matorage.
This is an example of pre-processing mnist and storing it in distributed storage.additionalis freely in the form of a dict, and records the shape and type of tensor to be stored inattributes.frommatorageimportDataConfigtraindata_config=DataConfig(endpoint='127.0.0.1:9000',access_key='minio',secret_key='miniosecretkey',dataset_name='mnist',additional={"mode":"train","framework":"pytorch",..."blah":"blah"},attributes=[('image','float32',(1,28,28)),('target','int64',(1))])Now do a simple pre-processing and save the data.frommatorageimportDataSavertraindata_saver=DataSaver(config=traindata_config)train_loader=DataLoader(dataset,batch_size=60,num_workers=8)for(image,target)intqdm(train_loader):# image shape : torch.Size([64, 1, 28, 28])# target shape : torch.Size([64])traindata_saver({'image':image,'target':target})traindata_saver.disconnect()3. Load dataset from matorageNow fetch data iteratively from storage with the same config as the saved dataset when training.frommatorage.torchimportDatasettrain_dataset=Dataset(config=traindata_config,clear=True)train_loader=DataLoader(train_dataset,batch_size=64,num_workers=8,shuffle=True)forbatch_idx,(image,target)inenumerate(tqdm(train_loader)):image,target=image.to(device),target.to(device)Only an index can be fetched through lazy load.train_dataset=Dataset(config=traindata_config,clear=True)print(train_dataset[0],len(train_dataset))4. Save & Load Model when trainingDuring training, you can save and load models of specific steps or epochs in distributed storage through inmemory.
First, make the model config the same as the dataset.frommatorageimportModelConfigfrommatorage.torchimportModelManagermodel_config=ModelConfig(endpoint='127.0.0.1:9000',access_key='minio',secret_key='miniosecretkey',model_name='mnist_simple_training',additional={"version":"1.0.1",..."blah":"blah"})model_manager=ModelManager(config=model_config)print(model_manager.get_metadata)model_manager.save(model,epoch=1)print(model_manager.get_metadata)When an empty model is loaded with specific steps or epochs, the appropriate weight is filled into the model.print(model.state_dict())model_manager.load(model,epoch=1)print(model.state_dict())# load a layer weight.print(model_manager.load('net1.0.weight',step=0))5. Save & Load Optimizer when trainingSave and load of optimizer is similar to managing model.frommatorageimportOptimizerConfigfrommatorage.torchimportOptimizerManageroptimizer_config=OptimizerConfig(endpoint='127.0.0.1:9000',access_key='minio',secret_key='miniosecretkey',optimizer_name='adam',additional={"model":"1.0.1",..."blah":"blah"})optimizer_manager=OptimizerManager(config=optimizer_config)print(optimizer_manager.get_metadata)# The optimizer contains information about the step.optimizer_manager.save(optimizer)print(optimizer_manager.get_metadata)When an empty optimizer is loaded with specific steps, the appropriate weight is filled into the optimizer.optimizer=optim.Adam(model.parameters(),lr=0.01)optimizer_manager.load(optimizer,step=938)Unittest$gitclonehttps://github.com/graykode/matorage&&cdmatorage
$python-mtests.test_suiteFramework Requirementtorch(>=1.0.0), torchvision(>=0.2.2)tensorflow(>=2.2), tensorflow_io(>=0.13)AuthorTae Hwan Jung(@graykode)We are looking for a contributor.