Generating new data while training.
Hello!
I'm working on an inference engine where I don't have any data before I start training. The idea is that at each epoch, a new data point is generated by a simulator and that data is fed to the NN only at that epoch.
I'm wondering if there is a way to do this with pytorch-lightning. I looked at the IterableDataset class but the only thing I came up with is to generate the data beforehand and then iterate over it.
Thanks!
hi @pjovanovski1 in this case you could follow a similar strategy proposed in this example of lightning-bolts for RL
https://github.com/PyTorchLightning/lightning-bolts/blob/master/pl_bolts/models/rl/reinforce_model.py#L254