Somewhat unrelated, but it would be good, in the beginning of fit, to check that self.algorithm is in the accepted values, and raise an error if it isn't.

The reason it is important is that if someone wants to input 'l-bfgs' and makes a typo, for instance 'lbgfs', it could run in mini-batch mode without the user realizing.