neulab/docprompting-conala huggingface dataset:
sklearn.metrics.explained_variance_score  
sklearn.metrics.explained_variance_score(y_true, y_pred, *, sample_weight=None, multioutput='uniform_average') [source]
 
Explained variance regression score function. Best possible score is 1.0, lower values are worse. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) or (n_samples, n_outputs) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) or (n_samples, n_outputs) 

Estimated target values.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
multioutput{‘raw_values’, ‘uniform_average’, ‘variance_weighted’} or array-like of shape (n_outputs,), default=’uniform_average’ 

Defines aggregating of multiple output scores. Array-like value defines weights used to average scores.  ‘raw_values’ :

Returns a full set of scores in case of multioutput input.  ‘uniform_average’ :

Scores of all outputs are averaged with uniform weight.  ‘variance_weighted’ :

Scores of all outputs are averaged, weighted by the variances of each individual output.      Returns 
 
scorefloat or ndarray of floats 

The explained variance or ndarray if ‘multioutput’ is ‘raw_values’.     Notes This is not a symmetric function. Examples >>> from sklearn.metrics import explained_variance_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> explained_variance_score(y_true, y_pred)
0.957...
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> explained_variance_score(y_true, y_pred, multioutput='uniform_average')
0.983...
sklearn.metrics.fowlkes_mallows_score  
sklearn.metrics.fowlkes_mallows_score(labels_true, labels_pred, *, sparse=False) [source]
 
Measure the similarity of two clusterings of a set of points.  New in version 0.18.  The Fowlkes-Mallows index (FMI) is defined as the geometric mean between of the precision and recall: FMI = TP / sqrt((TP + FP) * (TP + FN))
 Where TP is the number of True Positive (i.e. the number of pair of points that belongs in the same clusters in both labels_true and labels_pred), FP is the number of False Positive (i.e. the number of pair of points that belongs in the same clusters in labels_true and not in labels_pred) and FN is the number of False Negative (i.e the number of pair of points that belongs in the same clusters in labels_pred and not in labels_True). The score ranges from 0 to 1. A high value indicates a good similarity between two clusters. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = (n_samples,) 

A clustering of the data into disjoint subsets.  
labels_predarray, shape = (n_samples, ) 

A clustering of the data into disjoint subsets.  
sparsebool, default=False 

Compute contingency matrix internally with sparse matrix.    Returns 
 
scorefloat 

The resulting Fowlkes-Mallows score.     References  
1  
E. B. Fowkles and C. L. Mallows, 1983. “A method for comparing two hierarchical clusterings”. Journal of the American Statistical Association  
2  
Wikipedia entry for the Fowlkes-Mallows Index   Examples Perfect labelings are both homogeneous and complete, hence have score 1.0: >>> from sklearn.metrics.cluster import fowlkes_mallows_score
>>> fowlkes_mallows_score([0, 0, 1, 1], [0, 0, 1, 1])
1.0
>>> fowlkes_mallows_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 If classes members are completely split across different clusters, the assignment is totally random, hence the FMI is null: >>> fowlkes_mallows_score([0, 0, 0, 0], [0, 1, 2, 3])
0.0
sklearn.metrics.get_scorer  
sklearn.metrics.get_scorer(scoring) [source]
 
Get a scorer from string. Read more in the User Guide.  Parameters 
 
scoringstr or callable 

Scoring method as string. If callable it is returned as is.    Returns 
 
scorercallable 

The scorer.
sklearn.metrics.hamming_loss  
sklearn.metrics.hamming_loss(y_true, y_pred, *, sample_weight=None) [source]
 
Compute the average Hamming loss. The Hamming loss is the fraction of labels that are incorrectly predicted. Read more in the User Guide.  Parameters 
 
y_true1d array-like, or label indicator array / sparse matrix 

Ground truth (correct) labels.  
y_pred1d array-like, or label indicator array / sparse matrix 

Predicted labels, as returned by a classifier.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.18.     Returns 
 
lossfloat or int 

Return the average Hamming loss between element of y_true and y_pred.      See also  
accuracy_score, jaccard_score, zero_one_loss

  Notes In multiclass classification, the Hamming loss corresponds to the Hamming distance between y_true and y_pred which is equivalent to the subset zero_one_loss function, when normalize parameter is set to True. In multilabel classification, the Hamming loss is different from the subset zero-one loss. The zero-one loss considers the entire set of labels for a given sample incorrect if it does not entirely match the true set of labels. Hamming loss is more forgiving in that it penalizes only the individual labels. The Hamming loss is upperbounded by the subset zero-one loss, when normalize parameter is set to True. It is always between 0 and 1, lower being better. References  
1  
Grigorios Tsoumakas, Ioannis Katakis. Multi-Label Classification: An Overview. International Journal of Data Warehousing & Mining, 3(3), 1-13, July-September 2007.  
2  
Wikipedia entry on the Hamming distance.   Examples >>> from sklearn.metrics import hamming_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> hamming_loss(y_true, y_pred)
0.25
 In the multilabel case with binary label indicators: >>> import numpy as np
>>> hamming_loss(np.array([[0, 1], [1, 1]]), np.zeros((2, 2)))
0.75
 
 Examples using sklearn.metrics.hamming_loss
 
  Model Complexity Influence
sklearn.metrics.hinge_loss  
sklearn.metrics.hinge_loss(y_true, pred_decision, *, labels=None, sample_weight=None) [source]
 
Average hinge loss (non-regularized). In binary class case, assuming labels in y_true are encoded with +1 and -1, when a prediction mistake is made, margin = y_true * pred_decision is always negative (since the signs disagree), implying 1 - margin is always greater than 1. The cumulated hinge loss is therefore an upper bound of the number of mistakes made by the classifier. In multiclass case, the function expects that either all the labels are included in y_true or an optional labels argument is provided which contains all the labels. The multilabel margin is calculated according to Crammer-Singer’s method. As in the binary case, the cumulated hinge loss is an upper bound of the number of mistakes made by the classifier. Read more in the User Guide.  Parameters 
 
y_truearray of shape (n_samples,) 

True target, consisting of integers of two values. The positive label must be greater than the negative label.  
pred_decisionarray of shape (n_samples,) or (n_samples, n_classes) 

Predicted decisions, as output by decision_function (floats).  
labelsarray-like, default=None 

Contains all the labels for the problem. Used in multiclass hinge loss.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
lossfloat 
   References  
1  
Wikipedia entry on the Hinge loss.  
2  
Koby Crammer, Yoram Singer. On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. Journal of Machine Learning Research 2, (2001), 265-292.  
3  
L1 AND L2 Regularization for Multiclass Hinge Loss Models by Robert C. Moore, John DeNero.   Examples >>> from sklearn import svm
>>> from sklearn.metrics import hinge_loss
>>> X = [[0], [1]]
>>> y = [-1, 1]
>>> est = svm.LinearSVC(random_state=0)
>>> est.fit(X, y)
LinearSVC(random_state=0)
>>> pred_decision = est.decision_function([[-2], [3], [0.5]])
>>> pred_decision
array([-2.18...,  2.36...,  0.09...])
>>> hinge_loss([-1, 1, 1], pred_decision)
0.30...
 In the multiclass case: >>> import numpy as np
>>> X = np.array([[0], [1], [2], [3]])
>>> Y = np.array([0, 1, 2, 3])
>>> labels = np.array([0, 1, 2, 3])
>>> est = svm.LinearSVC()
>>> est.fit(X, Y)
LinearSVC()
>>> pred_decision = est.decision_function([[-1], [2], [3]])
>>> y_true = [0, 2, 3]
>>> hinge_loss(y_true, pred_decision, labels=labels)
0.56...
sklearn.metrics.homogeneity_completeness_v_measure  
sklearn.metrics.homogeneity_completeness_v_measure(labels_true, labels_pred, *, beta=1.0) [source]
 
Compute the homogeneity and completeness and V-Measure scores at once. Those metrics are based on normalized conditional entropy measures of the clustering labeling to evaluate given the knowledge of a Ground Truth class labels of the same samples. A clustering result satisfies homogeneity if all of its clusters contain only data points which are members of a single class. A clustering result satisfies completeness if all the data points that are members of a given class are elements of the same cluster. Both scores have positive values between 0.0 and 1.0, larger values being desirable. Those 3 metrics are independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score values in any way. V-Measure is furthermore symmetric: swapping labels_true and label_pred will give the same score. This does not hold for homogeneity and completeness. V-Measure is identical to normalized_mutual_info_score with the arithmetic averaging method. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

cluster labels to evaluate  
betafloat, default=1.0 

Ratio of weight attributed to homogeneity vs completeness. If beta is greater than 1, completeness is weighted more strongly in the calculation. If beta is less than 1, homogeneity is weighted more strongly.    Returns 
 
homogeneityfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly homogeneous labeling  
completenessfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly complete labeling  
v_measurefloat 

harmonic mean of the first two      See also  
homogeneity_score


completeness_score


v_measure_score
sklearn.metrics.homogeneity_score  
sklearn.metrics.homogeneity_score(labels_true, labels_pred) [source]
 
Homogeneity metric of a cluster labeling given a ground truth. A clustering result satisfies homogeneity if all of its clusters contain only data points which are members of a single class. This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way. This metric is not symmetric: switching label_true with label_pred will return the completeness_score which will be different in general. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

cluster labels to evaluate    Returns 
 
homogeneityfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly homogeneous labeling      See also  
completeness_score


v_measure_score

  References  
1  
Andrew Rosenberg and Julia Hirschberg, 2007. V-Measure: A conditional entropy-based external cluster evaluation measure   Examples Perfect labelings are homogeneous: >>> from sklearn.metrics.cluster import homogeneity_score
>>> homogeneity_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 Non-perfect labelings that further split classes into more clusters can be perfectly homogeneous: >>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 0, 1, 2]))
1.000000
>>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 1, 2, 3]))
1.000000
 Clusters that include samples from different classes do not make for an homogeneous labeling: >>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 1, 0, 1]))
0.0...
>>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 0, 0, 0]))
0.0...
 
 Examples using sklearn.metrics.homogeneity_score
 
  Demo of affinity propagation clustering algorithm  

  Demo of DBSCAN clustering algorithm  

  A demo of K-Means clustering on the handwritten digits data  

  Clustering text documents using k-means
sklearn.metrics.label_ranking_average_precision_score  
sklearn.metrics.label_ranking_average_precision_score(y_true, y_score, *, sample_weight=None) [source]
 
Compute ranking-based average precision. Label ranking average precision (LRAP) is the average over each ground truth label assigned to each sample, of the ratio of true vs. total labels with lower score. This metric is used in multilabel ranking problem, where the goal is to give better rank to the labels associated to each sample. The obtained score is always strictly greater than 0 and the best value is 1. Read more in the User Guide.  Parameters 
 
y_true{ndarray, sparse matrix} of shape (n_samples, n_labels) 

True binary labels in binary indicator format.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.20.     Returns 
 
scorefloat 
   Examples >>> import numpy as np
>>> from sklearn.metrics import label_ranking_average_precision_score
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_average_precision_score(y_true, y_score)
0.416...
sklearn.metrics.label_ranking_loss  
sklearn.metrics.label_ranking_loss(y_true, y_score, *, sample_weight=None) [source]
 
Compute Ranking loss measure. Compute the average number of label pairs that are incorrectly ordered given y_score weighted by the size of the label set and the number of labels not in the label set. This is similar to the error set size, but weighted by the number of relevant and irrelevant labels. The best performance is achieved with a ranking loss of zero. Read more in the User Guide.  New in version 0.17: A function label_ranking_loss   Parameters 
 
y_true{ndarray, sparse matrix} of shape (n_samples, n_labels) 

True binary labels in binary indicator format.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
lossfloat 
   References  
1  
Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US.
sklearn.metrics.log_loss  
sklearn.metrics.log_loss(y_true, y_pred, *, eps=1e-15, normalize=True, sample_weight=None, labels=None) [source]
 
Log loss, aka logistic loss or cross-entropy loss. This is the loss function used in (multinomial) logistic regression and extensions of it such as neural networks, defined as the negative log-likelihood of a logistic model that returns y_pred probabilities for its training data y_true. The log loss is only defined for two or more labels. For a single sample with true label \(y \in \{0,1\}\) and and a probability estimate \(p = \operatorname{Pr}(y = 1)\), the log loss is:  \[L_{\log}(y, p) = -(y \log (p) + (1 - y) \log (1 - p))\] Read more in the User Guide.  Parameters 
 
y_truearray-like or label indicator matrix 

Ground truth (correct) labels for n_samples samples.  
y_predarray-like of float, shape = (n_samples, n_classes) or (n_samples,) 

Predicted probabilities, as returned by a classifier’s predict_proba method. If y_pred.shape = (n_samples,) the probabilities provided are assumed to be that of the positive class. The labels in y_pred are assumed to be ordered alphabetically, as done by preprocessing.LabelBinarizer.  
epsfloat, default=1e-15 

Log loss is undefined for p=0 or p=1, so probabilities are clipped to max(eps, min(1 - eps, p)).  
normalizebool, default=True 

If true, return the mean loss per sample. Otherwise, return the sum of the per-sample losses.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
labelsarray-like, default=None 

If not provided, labels will be inferred from y_true. If labels is None and y_pred has shape (n_samples,) the labels are assumed to be binary and are inferred from y_true.  New in version 0.18.     Returns 
 
lossfloat 
   Notes The logarithm used is the natural logarithm (base-e). References C.M. Bishop (2006). Pattern Recognition and Machine Learning. Springer, p. 209. Examples >>> from sklearn.metrics import log_loss
>>> log_loss(["spam", "ham", "ham", "spam"],
...          [[.1, .9], [.9, .1], [.8, .2], [.35, .65]])
0.21616...
 
 Examples using sklearn.metrics.log_loss
 
  Probability Calibration for 3-class classification  

  Probabilistic predictions with Gaussian process classification (GPC)
sklearn.metrics.make_scorer  
sklearn.metrics.make_scorer(score_func, *, greater_is_better=True, needs_proba=False, needs_threshold=False, **kwargs) [source]
 
Make a scorer from a performance metric or loss function. This factory function wraps scoring functions for use in GridSearchCV and cross_val_score. It takes a score function, such as accuracy_score, mean_squared_error, adjusted_rand_index or average_precision and returns a callable that scores an estimator’s output. The signature of the call is (estimator, X, y) where estimator is the model to be evaluated, X is the data and y is the ground truth labeling (or None in the case of unsupervised models). Read more in the User Guide.  Parameters 
 
score_funccallable 

Score function (or loss function) with signature score_func(y, y_pred, **kwargs).  
greater_is_betterbool, default=True 

Whether score_func is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object will sign-flip the outcome of the score_func.  
needs_probabool, default=False 

Whether score_func requires predict_proba to get probability estimates out of a classifier. If True, for binary y_true, the score function is supposed to accept a 1D y_pred (i.e., probability of the positive class, shape (n_samples,)).  
needs_thresholdbool, default=False 

Whether score_func takes a continuous decision certainty. This only works for binary classification using estimators that have either a decision_function or predict_proba method. If True, for binary y_true, the score function is supposed to accept a 1D y_pred (i.e., probability of the positive class or the decision function, shape (n_samples,)). For example average_precision or the area under the roc curve can not be computed using discrete predictions alone.  
**kwargsadditional arguments 

Additional parameters to be passed to score_func.    Returns 
 
scorercallable 

Callable object that returns a scalar score; greater is better.     Notes If needs_proba=False and needs_threshold=False, the score function is supposed to accept the output of predict. If needs_proba=True, the score function is supposed to accept the output of predict_proba (For binary y_true, the score function is supposed to accept probability of the positive class). If needs_threshold=True, the score function is supposed to accept the output of decision_function. Examples >>> from sklearn.metrics import fbeta_score, make_scorer
>>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
>>> ftwo_scorer
make_scorer(fbeta_score, beta=2)
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.svm import LinearSVC
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]},
...                     scoring=ftwo_scorer)
 
 Examples using sklearn.metrics.make_scorer
 
  Demonstration of multi-metric evaluation on cross_val_score and GridSearchCV
sklearn.metrics.matthews_corrcoef  
sklearn.metrics.matthews_corrcoef(y_true, y_pred, *, sample_weight=None) [source]
 
Compute the Matthews correlation coefficient (MCC). The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary and multiclass classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient. [source: Wikipedia] Binary and multiclass labels are supported. Only in the binary case does this relate to information about true and false positives and negatives. See references below. Read more in the User Guide.  Parameters 
 
y_truearray, shape = [n_samples] 

Ground truth (correct) target values.  
y_predarray, shape = [n_samples] 

Estimated targets as returned by a classifier.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.18.     Returns 
 
mccfloat 

The Matthews correlation coefficient (+1 represents a perfect prediction, 0 an average random prediction and -1 and inverse prediction).     References  
1  
Baldi, Brunak, Chauvin, Andersen and Nielsen, (2000). Assessing the accuracy of prediction algorithms for classification: an overview.  
2  
Wikipedia entry for the Matthews Correlation Coefficient.  
3  
Gorodkin, (2004). Comparing two K-category assignments by a K-category correlation coefficient.  
4  
Jurman, Riccadonna, Furlanello, (2012). A Comparison of MCC and CEN Error Measures in MultiClass Prediction.   Examples >>> from sklearn.metrics import matthews_corrcoef
>>> y_true = [+1, +1, +1, -1]
>>> y_pred = [+1, -1, +1, +1]
>>> matthews_corrcoef(y_true, y_pred)
-0.33...
sklearn.metrics.max_error  
sklearn.metrics.max_error(y_true, y_pred) [source]
 
max_error metric calculates the maximum residual error. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) 

Estimated target values.    Returns 
 
max_errorfloat 

A positive floating point value (the best value is 0.0).     Examples >>> from sklearn.metrics import max_error
>>> y_true = [3, 2, 7, 1]
>>> y_pred = [4, 2, 7, 1]
>>> max_error(y_true, y_pred)
1
sklearn.metrics.mean_absolute_error  
sklearn.metrics.mean_absolute_error(y_true, y_pred, *, sample_weight=None, multioutput='uniform_average') [source]
 
Mean absolute error regression loss. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) or (n_samples, n_outputs) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) or (n_samples, n_outputs) 

Estimated target values.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
multioutput{‘raw_values’, ‘uniform_average’} or array-like of shape (n_outputs,), default=’uniform_average’ 

Defines aggregating of multiple output values. Array-like value defines weights used to average errors.  ‘raw_values’ :

Returns a full set of errors in case of multioutput input.  ‘uniform_average’ :

Errors of all outputs are averaged with uniform weight.      Returns 
 
lossfloat or ndarray of floats 

If multioutput is ‘raw_values’, then mean absolute error is returned for each output separately. If multioutput is ‘uniform_average’ or an ndarray of weights, then the weighted average of all output errors is returned. MAE output is non-negative floating point. The best value is 0.0.     Examples >>> from sklearn.metrics import mean_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_absolute_error(y_true, y_pred)
0.5
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_absolute_error(y_true, y_pred)
0.75
>>> mean_absolute_error(y_true, y_pred, multioutput='raw_values')
array([0.5, 1. ])
>>> mean_absolute_error(y_true, y_pred, multioutput=[0.3, 0.7])
0.85...
 
 Examples using sklearn.metrics.mean_absolute_error
 
  Poisson regression and non-normal loss  

  Tweedie regression on insurance claims