Package: torch-exid
Extended Iterable Dataset for PyTorchAn extension of PyTorchIterableDataset, this package introduces functionalities for shuffling, limiting, and offsetting data.InstallationDirectly fromPyPI:pipinstalltorch-exidOr usingPoetry:poetryaddtorch-exidUsageBegin by subclassingExtendedIterableDatasetand implement thegeneratormethod to yield items.Here's a simple example using anIntegersDataset:fromtorch_exidimportExtendedIterableDatasetclassIntegersDataset(ExtendedIterableDataset):defgenerator(self)->Iterator[int]:n=0whileTrue:yieldnn+=1# Will print out integers 0, 1, ..., 9:forninIntegersDataset(limit=10):print(n)Constructor ParametersExtendedIterableDatasetintroduces several parameters to provide additional control:limit: intSets the maximum number of data points to return. If negative, all data points are returned. Default is-1(return all data).# Will print out "0, 1, 2"forninIntegersDataset(limit=3)print(n)offset: intDetermines the number of initial data points to skip. Default is0.# Will print out "2, 3, 4"forninIntegersDataset(limit=3,offset=2)print(n)shuffle_buffer: intThis specifies the buffer size for shuffling. If greater than1, data is buffered and shuffled prior to being returned. If set to1(default), no shuffling occurs.# Will print out "0, 1, 3, 2" for the first time...forninIntegersDataset(limit=4,shuffle_buffer=2)print(n)# ...and 1, 0, 2, 3 second timeforninIntegersDataset(limit=4,shuffle_buffer=2)print(n)shuffle_seed: intDefines the seed for the random number generator used in shuffling. If not provided, a random seed is used:# Will print out "1, 0, 3, 2" both times:forninIntegersDataset(limit=4,shuffle_buffer=2,shuffle_seed=42)print(n)forninIntegersDataset(limit=4,shuffle_buffer=2,shuffle_seed=42)print(n)transforms: List[Callable[[Any], Any]]A list of transformations to apply to the data. Default is an empty list.ds=IntegersDataset(limit=3,transforms=[lambdan:n+1,lambdan:n**2,],)# Will print out "1, 4, 9"forninds:print(n)In addition to the above, any arguments or keyword arguments for theIterableDatasetsuperclass can also be passed.Methodsskip_next: Callable[[None], None]This method allows the skipping of the next item that would be yielded by thegenerator. Usingskip_nextwill not affect thelimitoroffset.classEvensDataset(ExtendedIterableDataset):defgenerator(self)->Iterator[int]:n=0whileTrue:ifn%2!=0:self.skip_next()yieldnn+=1ds=EvensDataset(limit=5)# Will print out "0, 2, 4, 6, 8"forninds:print(n)In other words, it allows you to bypass the next item without modifying the overall iteration parameters.ContributingContributions are greatly appreciated! Improvement can be made by submitting issues, proposing new features, or submitting pull requests with bug fixes or new functionalities.Getting started with contributingHere are the steps to get started with development:# Clone the repository:gitclonehttps://github.com/arlegotin/torch_exid.gitcdtorch_exid# Install the project and its dependencies using Poetry:poetryinstall# Spawn a shell within the virtual environment:poetryshell# Run tests to ensure everything is working correctly:pytesttests/Please ensure all changes are accompanied by relevant unit tests, and that all tests pass before submitting a pull request. This helps maintain the quality and reliability of the project.