Package: tsconcat
tsconcatThis is a neat little tool that supports the concatenation oftensorstoredatasets using then5/zarrdriver along a given axis to obtain concatenated datasets, without having to write a new dataset which would imply doubling the storage used on disk or using a pythonfor-loop to iterate over all datasets.NOTE: This tool works under the assumption yourindividual datasets are homogeneous, i.e. they all use the same block structure, datatype, and compression scheme.InstallationYou can install the package from PyPIpip install tsconcator directly from GitHub by runningpip install git+https://github.com/luisherrmann/tsconcatHow it worksThe tool exploits the file hierarchy ofn5/zarrdriver datasets to create concatenated datasets. Suppose you have two tensorstore datasetsds1.ds2with shape[2, 2, 2]and[2, 3, 2], respectively. Furthermore, let's say the datasets use a block size of[1, 2, 1]. Then, the respective nested file hierarchies would be:ds1.
├──metadata
├──0│└──0│├──0│└──1└──1└──0├──0└──1andds2.
├──metadata
├──0│├──0││├──0││└──1│└──1│├──0│└──1└──1├──0│├──0│└──1└──1├──0└──1, where directory nesting level 0 corresponds to dimension 0, nesting level 1 corresponds to dimension 1 and so on.Let's say we want toconcatenate along dimension 1to obtain a datasetds12. By assumption of homogeneity (same block size), the file hierarchy for a concatenated dataset must look exactlythe same on all nesting levels above the concatenation level. So, all we have to do is link all directories from nesting level 1 into common directories. For the above example:ds1/0/0 -> ds12/0/0ds2/0/0 -> ds12/0/1ds2/0/1 -> ds12/0/2ds1/1/0 -> ds12/1/0ds2/1/0 -> ds12/1/1ds2/1/1 -> ds12/1/1This leading to the following file hierarchy in the concatenated dataset:ds12.
├──metadata
├──0│├──0<----'ds1/0/0'││├──0││└──1│├──1<----'ds2/0/0'││├──0││└──1│└──2<----'ds2/0/1'│├──0│└──1└──1├──0<----'ds1/1/0'│├──0│└──1├──1<----'ds2/1/0'│├──0│└──1└──2<----'ds2/1/1'├──0└──1The metadata object of*ds12*is written to match the concatenated dataset.NOTE: Then5andzarrdriver specifications require all blocks to be the same size (except for termination blocks). The specification is respected in the above example where all blocks have size[1, 2, 1], except for termination blocks0/2/1and1/2/1at a block size of[1, 1, 1]. But what if we change the concatenation order and build a datasetds21from concatenatingds2withds1?In that case, all blocks would have size[1, 2, 1]except for termination blocks0/1/1and1/1/1. But these are NOT termination blocks! However, we can pretend they are blocks of the correct size[1, 2, 1]and mask out the excess.Thezarrdriver specification additionally allows for flat directory structures using.as a dimension separator. E.g.ds1in the above example would look like this:ds1...metadata0.0.00.0.11.0.01.0.1Since the number of blocks and thus of files to be linked does not change, scenarios with flat or nested directory structures are equivalent with the exception of the dimension separator.UsageYou can perform the concatenation by runningtsconcat<CONCAT_PATH><TS_PATH1><TS_PATH2>...<CAT_DIM>[-d<DRIVER>][-s<DIMSEP>][-p], whereCONCAT_PATHis the path of the target directory where to concatenate the datasets, andTS_PATH1, ... are the paths to the tensorstore datasets to be concatenated.CAT_DIMis the dimension along which to concatenate. As optional arguments, you can provide the<DRIVER>used in your tensorstores ('n5'or'zarr') and the dimension separator<DIMSEP>to use (where'.'is only supported for the zarr driver). Using the-pflag enables a progress bar.You can read from the datasets in python as follows:fromtsconcatimportConcatDatasetds=ConcatDataset(concat_path)data=ds[:].read().result()