LSPI modified a little. It seems like no one uses LSPI in an online way like how I'm using it here, which could explain the results I'm getting as well as suggesting that others have gotten the same. It seems to work fine if you collect lots and lots of samples, and then update once. But doing this alternating data collection and policy improvement doesn't seem to be working very well. I'm leaving it for now, but will be on the lookout for solutions.