Issue 1:
Description

Example: Machine Learning practioners want to try every single classification algorithm for a dataset, how about having a modularized module which is an additional feature of GridSearchCV that now takes MULTIPLE classification algorithms with multiple parameters for each classification algorithm and shoots out the model that best classifies the data based on a chosen scorer

Steps/Code to Reproduce

Example:

hyperparametertuning.py

(C) 2017 by Abhishek Babuji <abhishekb2209@gmail.com>

Contains methods to return a pipeline object and a dictionary containing
classifier parameters
"""

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline
from sklearn.svm import SVC

class HyperParameterTuning:
"""
Contains methods to return a pipeline object and a dictionary containing
classifier parameters
"""

def __init__(self, classifier, vectorizer):
"""
Args:
classifier (One of 6 sklearn classifier objects): 'logreg', 'svm', 'nb',
'knn', 'xgboost', 'randomforests'
vectorizer (CountVectorizer or TfidfVectorizer): Type of vector space model

Returns:
pipeline (sklearn pipeline object): Returns a pipeline object which is used
by GridSearchCV
model_params[self.classifier] (dict): Returns a dictionary of parameters
for the specified type of classifier

"""

self.classifier = classifier
self.vectorizer = vectorizer

def get_pipeline(self):
"""
Args:

classifier (One of 6 sklearn classifier objects): 'logreg', 'svm', 'nb',
'knn', 'xgboost', 'randomforests'
vectorizer (CountVectorizer or TfidfVectorizer): Type of vector space model

Returns:
pipeline (sklearn pipeline object): Returns a pipeline object which is
used by GridSearchCV
model_params[self.classifier] (dict): Returns a dictionary of parameters
for the specified type of classifier
"""

classifier_objects = {'logreg': LogisticRegression(),
'svm': SVC(),
'knn': KNeighborsClassifier(),
'xgboost': GradientBoostingClassifier(),
'randomforests': RandomForestClassifier(),
'nb': MultinomialNB()}
pipeline = Pipeline([('vect', self.vectorizer),
('clf', classifier_objects[self.classifier])])

return pipeline

def get_params(self):
"""
Args:
self

Returns:
model_params[self.classifier] (dict): Returns a dictionary of parameters for the
specified type of classifier

"""
model_params = {'logreg': {'clf__C': (1, 10, 100), 'clf__penalty': ('l1', 'l2')},
'svm': {'clf__C': (1, 10, 100),
'clf__kernel': ('linear', 'poly', 'rbf', 'sigmoid')},
'knn': {'clf__n_neighbors': (5, 10, 50, 100)},
'xgboost': {'clf__n_estimators': (100, 500, 1000)},
'randomforests': {'clf__n_estimators': (100, 500, 1000)},
'nb': {'clf__alpha': (0, 1), 'clf__fit_prior': (True, False)}}
return model_params[self.classifier]

An example like this, that tries EVERY classifier. Of course this is very computationally and time expensive.. But if you're trying to fit it for small datasets where Machine Learning is applicable, we could brainstorm further to maybe understand which default HyperParameters to tune for each algorithm using a scorer of your choice

Issue 2:
I came across a situation where I needed to split my data into more than 2 groups, and realised that this can't be done that simply with sklearn's splitter classes as it stands. I note that stack overflow has several similar questions from other users.

One can repeatedly apply splits, but this seems quite clunky given that the code seems to be readily extensible to handle n_groups > 2. Was the choice to limit to a train/test split consciously made for any specific reasons?