neulab/docprompting-conala huggingface dataset:
sklearn.metrics.top_k_accuracy_score(y_true, y_score, *, k=2, normalize=True, sample_weight=None, labels=None) [source]
 
Top-k Accuracy classification score. This metric computes the number of times where the correct label is among the top k labels predicted (ranked by predicted scores). Note that the multilabel case isn’t covered here. Read more in the User Guide  Parameters 
 
y_truearray-like of shape (n_samples,) 

True labels.  
y_scorearray-like of shape (n_samples,) or (n_samples, n_classes) 

Target scores. These can be either probability estimates or non-thresholded decision values (as returned by decision_function on some classifiers). The binary case expects scores with shape (n_samples,) while the multiclass case expects scores with shape (n_samples, n_classes). In the nulticlass case, the order of the class scores must correspond to the order of labels, if provided, or else to the numerical or lexicographical order of the labels in y_true.  
kint, default=2 

Number of most likely outcomes considered to find the correct label.  
normalizebool, default=True 

If True, return the fraction of correctly classified samples. Otherwise, return the number of correctly classified samples.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights. If None, all samples are given the same weight.  
labelsarray-like of shape (n_classes,), default=None 

Multiclass only. List of labels that index the classes in y_score. If None, the numerical or lexicographical order of the labels in y_true is used.    Returns 
 
scorefloat 

The top-k accuracy score. The best performance is 1 with normalize == True and the number of samples with normalize == False.      See also  
accuracy_score

  Notes In cases where two or more labels are assigned equal predicted scores, the labels with the highest indices will be chosen first. This might impact the result if the correct label falls after the threshold because of that. Examples >>> import numpy as np
>>> from sklearn.metrics import top_k_accuracy_score
>>> y_true = np.array([0, 1, 2, 2])
>>> y_score = np.array([[0.5, 0.2, 0.2],  # 0 is in top 2
...                     [0.3, 0.4, 0.2],  # 1 is in top 2
...                     [0.2, 0.4, 0.3],  # 2 is in top 2
...                     [0.7, 0.2, 0.1]]) # 2 isn't in top 2
>>> top_k_accuracy_score(y_true, y_score, k=2)
0.75
>>> # Not normalizing gives the number of "correctly" classified samples
>>> top_k_accuracy_score(y_true, y_score, k=2, normalize=False)
3
sklearn.metrics.v_measure_score(labels_true, labels_pred, *, beta=1.0) [source]
 
V-measure cluster labeling given a ground truth. This score is identical to normalized_mutual_info_score with the 'arithmetic' option for averaging. The V-measure is the harmonic mean between homogeneity and completeness: v = (1 + beta) * homogeneity * completeness
     / (beta * homogeneity + completeness)
 This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way. This metric is furthermore symmetric: switching label_true with label_pred will return the same score value. This can be useful to measure the agreement of two independent label assignments strategies on the same dataset when the real ground truth is not known. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

cluster labels to evaluate  
betafloat, default=1.0 

Ratio of weight attributed to homogeneity vs completeness. If beta is greater than 1, completeness is weighted more strongly in the calculation. If beta is less than 1, homogeneity is weighted more strongly.    Returns 
 
v_measurefloat 

score between 0.0 and 1.0. 1.0 stands for perfectly complete labeling      See also  
homogeneity_score


completeness_score


normalized_mutual_info_score

  References  
1  
Andrew Rosenberg and Julia Hirschberg, 2007. V-Measure: A conditional entropy-based external cluster evaluation measure   Examples Perfect labelings are both homogeneous and complete, hence have score 1.0: >>> from sklearn.metrics.cluster import v_measure_score
>>> v_measure_score([0, 0, 1, 1], [0, 0, 1, 1])
1.0
>>> v_measure_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 Labelings that assign all classes members to the same clusters are complete be not homogeneous, hence penalized: >>> print("%.6f" % v_measure_score([0, 0, 1, 2], [0, 0, 1, 1]))
0.8...
>>> print("%.6f" % v_measure_score([0, 1, 2, 3], [0, 0, 1, 1]))
0.66...
 Labelings that have pure clusters with members coming from the same classes are homogeneous but un-necessary splits harms completeness and thus penalize V-measure as well: >>> print("%.6f" % v_measure_score([0, 0, 1, 1], [0, 0, 1, 2]))
0.8...
>>> print("%.6f" % v_measure_score([0, 0, 1, 1], [0, 1, 2, 3]))
0.66...
 If classes members are completely split across different clusters, the assignment is totally incomplete, hence the V-Measure is null: >>> print("%.6f" % v_measure_score([0, 0, 0, 0], [0, 1, 2, 3]))
0.0...
 Clusters that include samples from totally different classes totally destroy the homogeneity of the labeling, hence: >>> print("%.6f" % v_measure_score([0, 0, 1, 1], [0, 0, 0, 0]))
0.0...
sklearn.metrics.zero_one_loss(y_true, y_pred, *, normalize=True, sample_weight=None) [source]
 
Zero-one classification loss. If normalize is True, return the fraction of misclassifications (float), else it returns the number of misclassifications (int). The best performance is 0. Read more in the User Guide.  Parameters 
 
y_true1d array-like, or label indicator array / sparse matrix 

Ground truth (correct) labels.  
y_pred1d array-like, or label indicator array / sparse matrix 

Predicted labels, as returned by a classifier.  
normalizebool, default=True 

If False, return the number of misclassifications. Otherwise, return the fraction of misclassifications.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
lossfloat or int, 

If normalize == True, return the fraction of misclassifications (float), else it returns the number of misclassifications (int).      See also  
accuracy_score, hamming_loss, jaccard_score

  Notes In multilabel classification, the zero_one_loss function corresponds to the subset zero-one loss: for each sample, the entire set of labels must be correctly predicted, otherwise the loss for that sample is equal to one. Examples >>> from sklearn.metrics import zero_one_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> zero_one_loss(y_true, y_pred)
0.25
>>> zero_one_loss(y_true, y_pred, normalize=False)
1
 In the multilabel case with binary label indicators: >>> import numpy as np
>>> zero_one_loss(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5
fit(X, y=None) [source]
 
Estimate model parameters with the EM algorithm. The method fits the model n_init times and sets the parameters with which the model has the largest likelihood or lower bound. Within each trial, the method iterates between E-step and M-step for max_iter times until the change of likelihood or lower bound is less than tol, otherwise, a ConvergenceWarning is raised. If warm_start is True, then n_init is ignored and a single initialization is performed upon the first call. Upon consecutive calls, training starts where it left off.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

List of n_features-dimensional data points. Each row corresponds to a single data point.    Returns 
 self
fit_predict(X, y=None) [source]
 
Estimate model parameters using X and predict the labels for X. The method fits the model n_init times and sets the parameters with which the model has the largest likelihood or lower bound. Within each trial, the method iterates between E-step and M-step for max_iter times until the change of likelihood or lower bound is less than tol, otherwise, a ConvergenceWarning is raised. After fitting, it predicts the most probable label for the input data points.  New in version 0.20.   Parameters 
 
Xarray-like of shape (n_samples, n_features) 

List of n_features-dimensional data points. Each row corresponds to a single data point.    Returns 
 
labelsarray, shape (n_samples,) 

Component labels.
predict(X) [source]
 
Predict the labels for the data samples in X using trained model.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

List of n_features-dimensional data points. Each row corresponds to a single data point.    Returns 
 
labelsarray, shape (n_samples,) 

Component labels.
predict_proba(X) [source]
 
Predict posterior probability of each component given the data.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

List of n_features-dimensional data points. Each row corresponds to a single data point.    Returns 
 
resparray, shape (n_samples, n_components) 

Returns the probability each Gaussian (state) in the model given each sample.
sample(n_samples=1) [source]
 
Generate random samples from the fitted Gaussian distribution.  Parameters 
 
n_samplesint, default=1 

Number of samples to generate.    Returns 
 
Xarray, shape (n_samples, n_features) 

Randomly generated sample  
yarray, shape (nsamples,) 

Component labels
score(X, y=None) [source]
 
Compute the per-sample average log-likelihood of the given data X.  Parameters 
 
Xarray-like of shape (n_samples, n_dimensions) 

List of n_features-dimensional data points. Each row corresponds to a single data point.    Returns 
 
log_likelihoodfloat 

Log likelihood of the Gaussian mixture given X.
score_samples(X) [source]
 
Compute the weighted log probabilities for each sample.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

List of n_features-dimensional data points. Each row corresponds to a single data point.    Returns 
 
log_probarray, shape (n_samples,) 

Log probabilities of each data point in X.
aic(X) [source]
 
Akaike information criterion for the current model on the input X.  Parameters 
 
Xarray of shape (n_samples, n_dimensions) 
  Returns 
 
aicfloat 

The lower the better.
bic(X) [source]
 
Bayesian information criterion for the current model on the input X.  Parameters 
 
Xarray of shape (n_samples, n_dimensions) 
  Returns 
 
bicfloat 

The lower the better.
sklearn.model_selection.check_cv(cv=5, y=None, *, classifier=False) [source]
 
Input checker utility for building a cross-validator  Parameters 
 
cvint, cross-validation generator or an iterable, default=None 

Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 5-fold cross validation, - integer, to specify the number of folds. - CV splitter, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if classifier is True and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used. Refer User Guide for the various cross-validation strategies that can be used here.  Changed in version 0.22: cv default value changed from 3-fold to 5-fold.   
yarray-like, default=None 

The target variable for supervised learning problems.  
classifierbool, default=False 

Whether the task is a classification task, in which case stratified KFold will be used.    Returns 
 
checked_cva cross-validator instance. 

The return value is a cross-validator which generates the train/test splits via the split method.
decision_function(X) [source]
 
Call decision_function on the estimator with the best found parameters. Only available if refit=True and the underlying estimator supports decision_function.  Parameters 
 
Xindexable, length n_samples 

Must fulfill the input assumptions of the underlying estimator.
fit(X, y=None, *, groups=None, **fit_params) [source]
 
Run fit with all sets of parameters.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

Training vector, where n_samples is the number of samples and n_features is the number of features.  
yarray-like of shape (n_samples, n_output) or (n_samples,), default=None 

Target relative to X for classification or regression; None for unsupervised learning.  
groupsarray-like of shape (n_samples,), default=None 

Group labels for the samples used while splitting the dataset into train/test set. Only used in conjunction with a “Group” cv instance (e.g., GroupKFold).  
**fit_paramsdict of str -> object 

Parameters passed to the fit method of the estimator
inverse_transform(Xt) [source]
 
Call inverse_transform on the estimator with the best found params. Only available if the underlying estimator implements inverse_transform and refit=True.  Parameters 
 
Xtindexable, length n_samples 

Must fulfill the input assumptions of the underlying estimator.
predict(X) [source]
 
Call predict on the estimator with the best found parameters. Only available if refit=True and the underlying estimator supports predict.  Parameters 
 
Xindexable, length n_samples 

Must fulfill the input assumptions of the underlying estimator.
predict_log_proba(X) [source]
 
Call predict_log_proba on the estimator with the best found parameters. Only available if refit=True and the underlying estimator supports predict_log_proba.  Parameters 
 
Xindexable, length n_samples 

Must fulfill the input assumptions of the underlying estimator.
predict_proba(X) [source]
 
Call predict_proba on the estimator with the best found parameters. Only available if refit=True and the underlying estimator supports predict_proba.  Parameters 
 
Xindexable, length n_samples 

Must fulfill the input assumptions of the underlying estimator.
score(X, y=None) [source]
 
Returns the score on the given data, if the estimator has been refit. This uses the score defined by scoring where provided, and the best_estimator_.score method otherwise.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

Input data, where n_samples is the number of samples and n_features is the number of features.  
yarray-like of shape (n_samples, n_output) or (n_samples,), default=None 

Target relative to X for classification or regression; None for unsupervised learning.    Returns 
 
scorefloat
score_samples(X) [source]
 
Call score_samples on the estimator with the best found parameters. Only available if refit=True and the underlying estimator supports score_samples.  New in version 0.24.   Parameters 
 
Xiterable 

Data to predict on. Must fulfill input requirements of the underlying estimator.    Returns 
 
y_scorendarray of shape (n_samples,)
transform(X) [source]
 
Call transform on the estimator with the best found parameters. Only available if the underlying estimator supports transform and refit=True.  Parameters 
 
Xindexable, length n_samples 

Must fulfill the input assumptions of the underlying estimator.
class sklearn.model_selection.GroupKFold(n_splits=5) [source]
 
K-fold iterator variant with non-overlapping groups. The same group will not appear in two different folds (the number of distinct groups has to be at least equal to the number of folds). The folds are approximately balanced in the sense that the number of distinct groups is approximately the same in each fold. Read more in the User Guide.  Parameters 
 
n_splitsint, default=5 

Number of folds. Must be at least 2.  Changed in version 0.22: n_splits default value changed from 3 to 5.       See also  
LeaveOneGroupOut


For splitting the data according to explicit domain-specific stratification of the dataset.    Examples >>> import numpy as np
>>> from sklearn.model_selection import GroupKFold
>>> X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
>>> y = np.array([1, 2, 3, 4])
>>> groups = np.array([0, 0, 2, 2])
>>> group_kfold = GroupKFold(n_splits=2)
>>> group_kfold.get_n_splits(X, y, groups)
2
>>> print(group_kfold)
GroupKFold(n_splits=2)
>>> for train_index, test_index in group_kfold.split(X, y, groups):
...     print("TRAIN:", train_index, "TEST:", test_index)
...     X_train, X_test = X[train_index], X[test_index]
...     y_train, y_test = y[train_index], y[test_index]
...     print(X_train, X_test, y_train, y_test)
...
TRAIN: [0 1] TEST: [2 3]
[[1 2]
 [3 4]] [[5 6]
 [7 8]] [1 2] [3 4]
TRAIN: [2 3] TEST: [0 1]
[[5 6]
 [7 8]] [[1 2]
 [3 4]] [3 4] [1 2]
 Methods  
get_n_splits([X, y, groups]) Returns the number of splitting iterations in the cross-validator  
split(X[, y, groups]) Generate indices to split data into training and test set.    
get_n_splits(X=None, y=None, groups=None) [source]
 
Returns the number of splitting iterations in the cross-validator  Parameters 
 
Xobject 

Always ignored, exists for compatibility.  
yobject 

Always ignored, exists for compatibility.  
groupsobject 

Always ignored, exists for compatibility.    Returns 
 
n_splitsint 

Returns the number of splitting iterations in the cross-validator.     
  
split(X, y=None, groups=None) [source]
 
Generate indices to split data into training and test set.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

Training data, where n_samples is the number of samples and n_features is the number of features.  
yarray-like of shape (n_samples,), default=None 

The target variable for supervised learning problems.  
groupsarray-like of shape (n_samples,) 

Group labels for the samples used while splitting the dataset into train/test set.    Yields 
 
trainndarray 

The training set indices for that split.  
testndarray 

The testing set indices for that split.
sklearn.model_selection.GroupKFold  
class sklearn.model_selection.GroupKFold(n_splits=5) [source]
 
K-fold iterator variant with non-overlapping groups. The same group will not appear in two different folds (the number of distinct groups has to be at least equal to the number of folds). The folds are approximately balanced in the sense that the number of distinct groups is approximately the same in each fold. Read more in the User Guide.  Parameters 
 
n_splitsint, default=5 

Number of folds. Must be at least 2.  Changed in version 0.22: n_splits default value changed from 3 to 5.       See also  
LeaveOneGroupOut


For splitting the data according to explicit domain-specific stratification of the dataset.    Examples >>> import numpy as np
>>> from sklearn.model_selection import GroupKFold
>>> X = np.array([[1, 2], [3, 4], [5, 6], [7, 8]])
>>> y = np.array([1, 2, 3, 4])
>>> groups = np.array([0, 0, 2, 2])
>>> group_kfold = GroupKFold(n_splits=2)
>>> group_kfold.get_n_splits(X, y, groups)
2
>>> print(group_kfold)
GroupKFold(n_splits=2)
>>> for train_index, test_index in group_kfold.split(X, y, groups):
...     print("TRAIN:", train_index, "TEST:", test_index)
...     X_train, X_test = X[train_index], X[test_index]
...     y_train, y_test = y[train_index], y[test_index]
...     print(X_train, X_test, y_train, y_test)
...
TRAIN: [0 1] TEST: [2 3]
[[1 2]
 [3 4]] [[5 6]
 [7 8]] [1 2] [3 4]
TRAIN: [2 3] TEST: [0 1]
[[5 6]
 [7 8]] [[1 2]
 [3 4]] [3 4] [1 2]
 Methods  
get_n_splits([X, y, groups]) Returns the number of splitting iterations in the cross-validator  
split(X[, y, groups]) Generate indices to split data into training and test set.    
get_n_splits(X=None, y=None, groups=None) [source]
 
Returns the number of splitting iterations in the cross-validator  Parameters 
 
Xobject 

Always ignored, exists for compatibility.  
yobject 

Always ignored, exists for compatibility.  
groupsobject 

Always ignored, exists for compatibility.    Returns 
 
n_splitsint 

Returns the number of splitting iterations in the cross-validator.     
  
split(X, y=None, groups=None) [source]
 
Generate indices to split data into training and test set.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

Training data, where n_samples is the number of samples and n_features is the number of features.  
yarray-like of shape (n_samples,), default=None 

The target variable for supervised learning problems.  
groupsarray-like of shape (n_samples,) 

Group labels for the samples used while splitting the dataset into train/test set.    Yields 
 
trainndarray 

The training set indices for that split.  
testndarray 

The testing set indices for that split.     
 
 Examples using sklearn.model_selection.GroupKFold
 
  Visualizing cross-validation behavior in scikit-learn
get_n_splits(X=None, y=None, groups=None) [source]
 
Returns the number of splitting iterations in the cross-validator  Parameters 
 
Xobject 

Always ignored, exists for compatibility.  
yobject 

Always ignored, exists for compatibility.  
groupsobject 

Always ignored, exists for compatibility.    Returns 
 
n_splitsint 

Returns the number of splitting iterations in the cross-validator.
split(X, y=None, groups=None) [source]
 
Generate indices to split data into training and test set.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

Training data, where n_samples is the number of samples and n_features is the number of features.  
yarray-like of shape (n_samples,), default=None 

The target variable for supervised learning problems.  
groupsarray-like of shape (n_samples,) 

Group labels for the samples used while splitting the dataset into train/test set.    Yields 
 
trainndarray 

The training set indices for that split.  
testndarray 

The testing set indices for that split.
split(X, y=None, groups=None) [source]
 
Generate indices to split data into training and test set.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

Training data, where n_samples is the number of samples and n_features is the number of features.  
yarray-like of shape (n_samples,), default=None 

The target variable for supervised learning problems.  
groupsarray-like of shape (n_samples,) 

Group labels for the samples used while splitting the dataset into train/test set.    Yields 
 
trainndarray 

The training set indices for that split.  
testndarray 

The testing set indices for that split.     Notes Randomized CV splitters may return different results for each call of split. You can make the results identical by setting random_state to an integer.
fit(X, y=None, groups=None, **fit_params) [source]
 
Run fit with all sets of parameters.  Parameters 
 
Xarray-like, shape (n_samples, n_features) 

Training vector, where n_samples is the number of samples and n_features is the number of features.  
yarray-like, shape (n_samples,) or (n_samples, n_output), optional 

Target relative to X for classification or regression; None for unsupervised learning.  
groupsarray-like of shape (n_samples,), default=None 

Group labels for the samples used while splitting the dataset into train/test set. Only used in conjunction with a “Group” cv instance (e.g., GroupKFold).  
**fit_paramsdict of string -> object 

Parameters passed to the fit method of the estimator
split(X, y=None, groups=None) [source]
 
Generate indices to split data into training and test set.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

Training data, where n_samples is the number of samples and n_features is the number of features.  
yarray-like of shape (n_samples,), default=None 

The target variable for supervised learning problems.  
groupsarray-like of shape (n_samples,), default=None 

Group labels for the samples used while splitting the dataset into train/test set.    Yields 
 
trainndarray 

The training set indices for that split.  
testndarray 

The testing set indices for that split.