Package: chronica
ChronicaA data-imbalance-aware scheduler for distributed deep learningChronica is a data-imbalance-aware scheduler and associated scheduling framework for distributed deep learning.
Chronica accelerates data-parallel training with imbalanced data such as video, audio and text by reducing redundant operations and memory footprint while improving scalability and resource utilization.
The goal of Chronica is to make a fast, efficient and easy-to-use data scheduling framework, compatible with existing deep learning systems such as TensorFlow, PyTorch and JAX.Highlighted featuresData-imbalance-aware schedulingChronica provides several schedule kinds such asstaticanddynamic, each of which can be useful for workloads with different characteristics.Partitioned data setIn addition to the traditional fully sharded data sets, Chronica supports apartitioned data setwhere the data is split into multiple data partitions across nodes in the cluster.Minimal code modificationsWhile benefiting from the above features, users only need to modify a few lines in the existing code base, and the rests are transparent.Supported frameworksChronica currently supports PyTorch only.Quick startPrerequisitesYou need to have the latest version of theGocompiler installed on the master node (rank 0) and add theGOBINenvironment variable toPATH,i.e.,PATH=$GOBIN:$PATH.
Once the Go compiler is installed, you can install thechronicapip package.pipinstallchronicaHow to useTo use Chronica, make the following modifications to your program:Make your data set inherit from Chronica's data set class instead of existing data set class:e.g., for PyTorch, usechronica.torch.utils.data.Datasetinstead oftorch.utils.data.Dataset.Overwrite__sizeof__in your data set, which represents the relative size of each data sample:e.g., for video data sets, the relative size of each data sample is determined by the number of frames.
Thus you can overwrite__sizeof__using OpenCV as follows.def__sizeof__(self,index:int)->int:returnint(cv2.VideoCapture(self.videos[index]).get(cv2.CAP_PROP_FRAME_COUNT))Use Chronica's data sampler instead of existing data sampler:e.g., for PyTorch, usechronica.torch.utils.data.DistributedSamplerinstead oftorch.utils.data.DistributedSampler.Pass additional parameters to the data sampler:e.g., setkind="dynamic"if you need dynamic scheduling.[!TIP]
In most cases, the above modifications can be done by addingchronica.to import statements and overwriting__sizeof__, as shown below:-from torch.utils.data import Dataset+from chronica.torch.utils.data import Dataset+    def __sizeof__(self, index: int) -> int:+        return int(cv2.VideoCapture(self.videos[index]).get(cv2.CAP_PROP_FRAME_COUNT))-from torch.utils.data import DistributedSampler+from chronica.torch.utils.data import DistributedSamplerPublicationsSanha Maeng, Gordon Euhyun Moon and Sungyong Park,Chronica: A Data-Imbalance-Aware Scheduler for Distributed Deep Learning, in Proceedings of the2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)CitationIf you use Chronica in your work, we would appreciate citations using the following metadata:@inproceedings{maeng2023chronica,author={Sanha Maeng, Gordon Euhyun Moon and Sungyong Park},booktitle={2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)},title={Chronica: A Data-Imbalance-Aware Scheduler for Distributed Deep Learning},year={2023},pages={262--272},doi={10.1109/CCGrid57682.2023.00033}}