GSoC 2014: Standard Extreme Learning Machines

Finished implementing the standard extreme learning machines (ELMs). I am getting the following results with 550 hidden neurons against the digits datasets,

Training accuracy using the logistic activation function: 0.999444
Training accuracy using the tanh activation function: 1.000000

Fortunately, this algorithm is much easier to implement and debug than multi-layer perceptron :).
I will push a test file soon.

@ogrisel , @larsmans
Agreed - computing for `self._H_sub` is unnecessarily costly for the `standard` algorithm, but I could address this with a conditional, `if self.algorithm == 'recursion_lsqr' `
The problem I am facing from extracting the first batch outside the for loop is the following,
The first batch is treated differently from the second batch on wards.
So the only way I got this working is through this code,
```python
        # Run the least-square algorithm on the whole dataset
        first_call = False
        if self.algorithm == 'standard':
            batch_size = n_samples
            first_call = True

        # Run the recursive least-square algorithm on mini-batches
        elif self.algorithm == 'recursion_lsqr':
            batch_size = self.batch_size

            if not warm_start or self._H_sub is None:
                first_call = True

        batches = list(gen_batches(n_samples, batch_size))

        if first_call:
            H_batch = self._compute_hidden_activations(X[batches[0]])
            # beta_{0} = inv(H_{0}^T H_{0} + (1./C)*Id) * H_{0}.T y_{0}
            self.coef_output_ = ridge_regression(H_batch, y[batches[0]],
                                                 1. / self.C,
                                                 sample_weight=sample_weight).T
            if self.algorithm == 'recursion_lsqr':
                # K_{0} = H_{0}^T H_{0}
                self._H_sub = safe_sparse_dot(H_batch.T, H_batch)

            batches = batches[1:]

        for batch, batch_slice in enumerate(batches):
            # Compute hidden activations H_{i} for batch i
            H_batch = self._compute_hidden_activations(X[batch_slice])

            # Recursive step for batch 1, 2, ..., n
            # Update K_{i+1} by H_{i}^T * H_{i}
            self._H_sub += safe_sparse_dot(H_batch.T, H_batch)

            # Update beta_{i+1} by
            # K_{i+1}^{-1}H{i+1}^T(y{i+1} - H_{i+1} * beta_{i})
            y_sub = y[batch_slice] - safe_sparse_dot(H_batch,
                                                     self.coef_output_)
            Hy_sub = safe_sparse_dot(H_batch.T, y_sub)

            # Update hidden-to-output coefficients
            self.coef_output_ += np.linalg.solve(self._H_sub +
                                                 identity(self.n_hidden) *
                                                (1. / self.C),
                                                 Hy_sub)

            if self.verbose:
                scores = self._decision_scores(X[batch_slice])
                print("Batch %d, Training mean square error = %f" %
                     (batch, mean_squared_error(y[batch_slice], scores)))
```
The `first_call` variable is necessary to pass the following test,
```python
        elm_with_fit = ELMRegressor(algorithm='recursion_lsqr',
                           random_state=random_state,
                           activation=activation, batch_size=batch_size)
        elm_with_fit.fit(X, y)
        elm_with_partial_fit = ELMRegressor(algorithm='recursion_lsqr', activation=activation,
                           random_state=random_state, batch_size=batch_size)
        for batch_slice in gen_batches(n_samples, batch_size):
            elm_with_partial_fit.partial_fit(X[batch_slice], y[batch_slice])
```
`elm_with_partial_fit` and `elm_with_fit` should have the same `self.coef_output_`.
Also, `verbose` might need to be repeated, do you think extracting the first batch outside is worth it?

Thanks.