Package: csv-dataset
csv-datasetCsvDatasethelps to read a csv file and create descriptive and efficient input pipelines for deep learning.CsvDatasetiterates the records of the csv file in a streaming fashion, so the full dataset does not need to fit into memory.Install$pipinstallcsv-datasetUsageSuppose we have a csv file whose absolute path isfilepath:open_time,open,high,low,close,volume
1576771200000,7145.99,7150.0,7141.01,7142.33,21.094283
1576771260000,7142.89,7142.99,7120.7,7125.73,118.279931
1576771320000,7125.76,7134.46,7123.12,7123.12,41.03628
1576771380000,7123.74,7128.06,7117.12,7126.57,39.885367
1576771440000,7127.34,7137.84,7126.71,7134.99,25.138154
1576771500000,7134.99,7144.13,7132.84,7141.64,26.467308
...fromcsv_datasetimport(Dataset,CsvReader)dataset=CsvDataset(CsvReader(filepath,float,# Abandon the first column and only pick the followingindexes=[1,2,3,4,5],header=True)).window(3,1).batch(2)forelementindataset:print(element)The following output shows one print.[[[7145.99,7150.0,7141.01,7142.33,21.094283][7142.89,7142.99,7120.7,7125.73,118.279931][7125.76,7134.46,7123.12,7123.12,41.03628]][[7142.89,7142.99,7120.7,7125.73,118.279931][7125.76,7134.46,7123.12,7123.12,41.03628][7123.74,7128.06,7117.12,7126.57,39.885367]]]...Dataset(reader: AbstractReader)dataset.window(size: int, shift: int = None, stride: int = 1) -> selfDefines the window size, shift and stride.The default window size is1which means the dataset has no window.Parameter explanationSuppose we have a raw data set[ 1  2  3  4  5  6  7  8  9 ... ]And the following is a window of(size=4, shift=3, stride=2)|-------------- size:4 --------------|
          |- stride:2 -|                       |
          |            |                       |
win 0:  [ 1            3           5           7  ] --------|-----
                                                       shift:3
win 1:  [ 4            6           8           10 ] --------|-----

win 2:  [ 7            9           11          13 ]

...dataset.batch(batch: int) -> selfDefines batch size.The default batch size of the dataset is1which means it is single-batchIf batch is2batch 0:  [[ 1            3           5           7  ]
           [ 4            6           8           10 ]]

batch 1:  [[ 7            9           11          13 ]
           [ 10           12          14          16 ]]

...dataset.get() -> Optional[np.ndarray]Gets the data of the next batchdataset.reset() -> selfResets datasetdataset.read(amount: int, reset_buffer: bool = False)amountthe maximum length of data the dataset will readreset_bufferifTrue, the dataset will reset the data of the previous window in the bufferReads multiple batches at a timeIf wereset_buffer, then the next read will not use existing data in the buffer, and the result will have no overlap with the last read.dataset.reset_buffer() -> NoneReset buffer, so that the next read will have no overlap with the last onedataset.lines_need(reads: int) -> intCalculates and returns how many lines of the underlying datum are needed for readingreadstimesdataset.max_reads(max_lines: int) -> int | NoneCalculatesmax_lineslines could afford how many readsdataset.max_reads() -> int | NoneCalculates the current reader could afford how many reads.Ifmax_linesof current reader is unset, then it returnsNoneCsvReader(filepath, dtype, indexes, **kwargs)filepathstrabsolute path of the csv filedtypeCallabledata type. We should only usefloatorintfor this argument.indexesList[int]column indexes to pick from the lines of the csv filekwargsheaderbool = Falsewhether we should skip reading the header line.splitterstr = ','the column splitter of the csv filenormalizerList[NormalizerProtocol]list of normalizer to normalize each column of data. ANormalizerProtocolshould contains two methods,normalize(float) -> floatto normalize the given datum andrestore(float) -> floatto restore the normalized datum.max_linesint = -1max lines of the csv file to be read. Defaults to-1which means no limit.reader.reset()Resets reader posproperty reader.max_linesGetsmax_linessetter reader.max_lines = linesChangesmax_linesreader.readline() -> listReturns the converted value of the next linereader csvReader.linesReturns number of lines has been readLicenseMIT