Optimize inference time for sklearn models in real time for single example

### Describe the bug

It is a multi-class classification model with sklearn. 

I am using `OneVsOneClassifier` model to train and predict `150 intents`. Its a multi-class classification problem.

**Data:**

    text          intents
    
    text1         int1
    text2         int2

I convert these intents in labels using:

    le = LabelEncoder()
    y_train = le.fit_transform(y_train)
    y_test = le.fit_transform(y_test)

 **Expectation:**

Without changing the training pipeline or parameters, note the inference time. Currently, it's slow, ~1second for 1 inference. 
`To Check if` : this is expected; there is a problem in setting (e.g., the kernels, the libarries); there is a bug in inference code

**Code:**

```python
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.compose import ColumnTransformer
    from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
    from sklearn.pipeline import Pipeline
    from sklearn.svm import SVC,LinearSVC
    
    def create_pipe(clf):
        
        # Each pipeline uses the same column transformer.  
        column_trans = ColumnTransformer(
                [('Text', TfidfVectorizer(), 'text')
                 ],
                remainder='drop') 
        
        pipeline = Pipeline([('prep',column_trans),                     
                             ('clf', clf)])
         
        return pipeline
    
    def fit_and_print(pipeline):
        
        pipeline.fit(X_train, y_train)
        y_pred = pipeline.predict(X_test)
    
        print(metrics.classification_report(y_test, y_pred, 
                                            target_names=le.classes_, 
                                            digits=3))
    clf = OneVsOneClassifier(LinearSVC(random_state=42, class_weight='balanced'))
    pipeline = create_pipe(clf)
    %time fit_and_print(pipeline)
    
    # convert input to df
    
    def create_test_data(x):
        d = {'text' : x}
        df = pd.DataFrame(d, index=[0])
        return df
    
    revs=[]
    for idx in [948, 5717, 458]:
         cur = test.loc[idx, 'text']
         revs.append(cur)
    print(revs) 
    
    revs=sam['text'].values
    
    %%time
    for rev in revs:
        c_res = pipeline.predict(create_test_data(rev))
        print(rev, '=', labels[c_res[0]])
```

For inferencing on `one data at at time` its taking `~1 sec.` As in real, live data one data point will be given to inferencing model.

But when I do `batch inferencing` it is optimised, as for `60 samples` it just takes `~1 second.`

**Is there a way to improve inference time on 1 sample? Or is this the expected behaviour?**


### Steps/Code to Reproduce

     from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.compose import ColumnTransformer
    from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
    from sklearn.pipeline import Pipeline
    from sklearn.svm import SVC,LinearSVC
    
    def create_pipe(clf):
        
        # Each pipeline uses the same column transformer.  
        column_trans = ColumnTransformer(
                [('Text', TfidfVectorizer(), 'text')
                 ],
                remainder='drop') 
        
        pipeline = Pipeline([('prep',column_trans),                     
                             ('clf', clf)])
         
        return pipeline
    
    def fit_and_print(pipeline):
        
        pipeline.fit(X_train, y_train)
        y_pred = pipeline.predict(X_test)
    
        print(metrics.classification_report(y_test, y_pred, 
                                            target_names=le.classes_, 
                                            digits=3))
    clf = OneVsOneClassifier(LinearSVC(random_state=42, class_weight='balanced'))
    pipeline = create_pipe(clf)
    %time fit_and_print(pipeline)
    
    # convert input to df
    
    def create_test_data(x):
        d = {'text' : x}
        df = pd.DataFrame(d, index=[0])
        return df
    
    revs=[]
    for idx in [948, 5717, 458]:
         cur = test.loc[idx, 'text']
         revs.append(cur)
    print(revs) 
    
    revs=sam['text'].values
    
    %%time
    for rev in revs:
        c_res = pipeline.predict(create_test_data(rev))
        print(rev, '=', labels[c_res[0]])

### Expected Results

Time for single example should be in milliseconds.

### Actual Results

Time for single example is 1.33 seconds

### Versions

```shell
System:
    python: 3.6.13 |Anaconda, Inc.| (default, Jun  4 2021, 14:25:59)  [GCC 7.5.0]
executable: /opt/conda/bin/python
   machine: Linux-3.10.0-1160.25.1.el7.x86_64-x86_64-with-debian-stretch-sid

Python dependencies:
          pip: 21.2.3
   setuptools: 57.4.0
      sklearn: 0.24.2
        numpy: 1.19.5
        scipy: 1.5.4
       Cython: 0.29.24
       pandas: 1.1.5
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 3.0.0

Built with OpenMP: True
```

COMMENT:
Scikit-learn is currently not optimum for data stream prediction.

Many checks are carried on that become too expensive compared with the cost of making the prediction.

To check that this is the case here, one should profile the call to `predict`.

One workaround here is to use `sklearn-onnx` (https://onnx.ai/sklearn-onnx/) for the inference that is optimized for such a use case.

Labels: Performance