Issue 1:
Description

I've been unable to perform a grid-search on a pipeline that changes sample numbers. The pipeline itself works fine, since I use a FunctionTransformer() that feeds into the classifier with the fewer number of samples.

The idea here is that I need to train a topic model embedding of a large corpus, and since this is unsupervised, I use the entire corpus I have to learn these topics (or alternatively use w2v). Then, only a subset of these are labeled, so I extract the labeled docs and train a LinearSVC to predict some label based on the topic vectors. I cobbled this pipe together using Textacy (based on SpaCy) and SKLearn, but it does work, taking a corpus object as input and getting a classification as output.

I would like to grid search the topic number using the cross-validation score of the classifier. This seems perfectly reasonable to me, since the parameters I'm picking are in the large-sample space and the CV score I get is in the small-sample space, but neither affect the ordering of the other, in principal.

Steps/Code to Reproduce

Here's a notebook gist, but note that there are a few extra classes I've defined externally to process the text corpus and return topic vectors. Still, the pipe works just not in the grid-search.

Expected Results

CV score of a classifier fed topic vectors (trained from more samples than the classifier has labels for) for varying numbers of topics/model types.

Actual Results

See gist.

Versions

Windows-7-6.1.7601-SP1

('Python', '2.7.13 |Anaconda custom (64-bit)| (default, Dec 19 2016, 13:29:36) [MSC v.1500 64 bit (AMD64)]')

('NumPy', '1.11.3')

('SciPy', '0.18.1')

('Scikit-Learn', '0.18.1')

Issue 2:
Some data transformations -- including over/under-sampling (#1454), outlier removal, instance reduction, and other forms of dataset compression, like that used in BIRCH (#3802) -- entail altering a dataset at training time, but leaving it unaltered at prediction time. (In some cases, such as outlier removal, it makes sense to reapply a fitted model to new data, while in others model reuse after fitting seems less applicable. )

As noted elsewhere, transformers that change the number of samples are not currently supported, certainly in the context of Pipelines where a transformation is applied both at fit and predict time (although a hack might abuse fit_transform to make this not so). Pipelines of Transformers also would not cope with changes in the sample size at fit time for supervised problems because Transformers do not return a modified y, only X.

To handle this class of problems, I propose introducing a new category of estimator, called a Resampler. It must define at least a fit_resample method, which Pipeline will call at fit time, passing the data unchanged at other times. (For this reason, a Resampler cannot also be a Transformer, or else we need to define their precedence.)

For many models, fit_resample needs only return sample_weight. For sample compression approaches (e.g. that in BIRCH), this is not sufficient as the representative centroids are modified from the input samples. Hence I think fit_resample should return altered data directly, in the form of a dict with keys X, y, sample_weight as required. (It still might be appropriate for many Resamplers to only modify sample_weight; if necessary, another Resampler can be chained that realises the weights as replicated or deleted entries in X and y.)