Using iterable datasets

pzg: <p>Hello,</p>
<p>Im trying to set up a processes to deal with streaming sensor data. I understand this is the exact usecase for iterable datasets. Unfortunately it seems to be a relatively new feature and tutorials on how to make and use iterable datasets appear to be rare.</p>
<p>To start I am mocking by making my <strong>iter</strong> function grab from a CSV.  I am not sure exactly what my <strong>iter</strong> function should be returning.  Currently its returning a ‘TensorDataset’ because that’s what I was able to find examples of how to consume in training/validation but it complains about that TypeError: iter() returned non-iterator of type ‘TensorDataset’.</p>
<p>How do I put together the information of the data and its labels for a DataLoader to use so I can train my model?</p>
<p>Thank you</p>

JamesTrick: <p>Hi <a class="mention" href="/u/pzg">@pzg</a> Welcome to the forums!</p>
<p>Have you taken a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">this tutorial on dataloading</a>?</p>
<p>For specific advice, it’d be easier if you can post an example of your code so far.</p>

pzg: <p>Hello <a class="mention" href="/u/jamestrick">@JamesTrick</a><br>
Thank you for the advice. I had seen that tutorial but thought it didnt apply because it was about mapped datasets which I understand doesnt translate well to streaming data. I tried setting it back to a plain Dataset but unfortunately I would have no idea how to set the length in the case when I am getting real live sensor data in real time which was what I was hoping Iterable Datasets would help solve.</p>
<p>The real goal is to set up a training flow to accept real time streaming live data with no discernible end beforehand.</p>
<p>I have taken the advice to return a dict as stated in the tutorial but it complains that its not getting an iterable</p>
<p>I am calling it from the following snippet (between the ~~~~~~~~~) in the training function and it returns the following error message (never prints)</p>
<p>TypeError: iter() returned non-iterator of type ‘dict’</p>
<pre><code class="lang-auto">trainer = datasets.MyLSLIterableDataset(
                     path_list=path_list_train)        
for i, (features, labels) in enumerate(trainer):
            print('entered first enumeration')
</code></pre>
<p>My dataset implementation is :</p>
<pre><code class="lang-auto">class MyLSLIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, num_columns, chunk_size=512, path_list=None): #, start, end):
        super(torch.utils.data.dataset.IterableDataset).__init__()
        self.resolver = ContinuousResolver()
        self.inlets = []
        self.path_list = path_list
        self.already_given = 0
        self.path_no = 0
        self.chunk_size = chunk_size

    def __iter__(self):
        #take data from buffer, lets say a chunk is 1000 buffered samples and clear it?
        result = None
        while(result is None):
            #try:
                result = get_data(path_list = self.path_list, path_no = self.path_no, start_from = self.already_given, num_to_read = self.chunk_size, skip_columns = ['experiment'])
                self.already_given += self.chunk_size
                print(result)
                self.data = result
                #raise Exception('Im sure this is the wrong way to debug')
                if result is None:
                    self.path_no += 1
                    self.already_given = 0
                    if self.path_no &gt; len(self.path_list):
                        raise StopIteration
        return result


def preprocess_data(dataframe, label):
    scaler = StandardScaler()
    X_train = dataframe.drop(columns=[label], axis = 1)
    y_train = dataframe[label]
    #X_train = scaler.fit_transform(X_train) -&gt; this would make it noniterable, not sure why
    X_train = X_train.to_numpy()
    y_train = pd.get_dummies(y_train)
    features_train = torch.from_numpy(X_train).float()
    labels_train = torch.from_numpy(y_train.to_numpy())#.type(torch.LongTensor)
    return {'features' : features_train, 'labels' :labels_train} 


def get_data(path_list, path_no, start_from, num_to_read, skip_columns = ['plzskip']):
    if path_list is None:
        raise Exception('Path is None but Live Mode not yet implemented!')
    else: #please presepperate validation and test data somehow not sure how to do it here
        try:
            cols = list(pd.read_csv(path_list[path_no], nrows =1))
            return_data_as_df =  pd.read_csv(path_list[path_no], skiprows = start_from, nrows = num_to_read, usecols =[i for i in cols if i not in skip_columns] )

            return_data_as_df = preprocess_data(
                                return_data_as_df, 'event')
            #print('getting first data')
            return return_data_as_df
        except Exception as e:
            print(e)
            return None #probably ran out of stuff on that path
</code></pre>
<p>Ive tried several things including having it return itself an iterable database and it still gives me that same type error:</p>
<p>TypeError: iter() returned non-iterator of type ‘MyLSLIterableDataset’</p>
<p>I would really love your advice on how to handle either this error or a better strategy for dealing with streaming data</p>

pzg: <p>In case anyone else runs into this. What it turns out it was is that to implement an iterable of any kind <strong>iter</strong> needs to return self and you also need to implement a <strong>next</strong> function that returns the actual data.</p>
<p>This may be clear to anyone familiar with iterators but wasn’t immediately obvious to me just from the iterable dataset documentation which only spoke of overwriting <strong>iter</strong></p>

Aris_L: <p>weird why documentation didn’t mention <strong>next</strong> at all, consider it plays a important role of actually fetching data.</p>