neulab/docprompting-conala huggingface dataset:
sklearn.metrics.hamming_loss(y_true, y_pred, *, sample_weight=None) [source]
 
Compute the average Hamming loss. The Hamming loss is the fraction of labels that are incorrectly predicted. Read more in the User Guide.  Parameters 
 
y_true1d array-like, or label indicator array / sparse matrix 

Ground truth (correct) labels.  
y_pred1d array-like, or label indicator array / sparse matrix 

Predicted labels, as returned by a classifier.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.18.     Returns 
 
lossfloat or int 

Return the average Hamming loss between element of y_true and y_pred.      See also  
accuracy_score, jaccard_score, zero_one_loss

  Notes In multiclass classification, the Hamming loss corresponds to the Hamming distance between y_true and y_pred which is equivalent to the subset zero_one_loss function, when normalize parameter is set to True. In multilabel classification, the Hamming loss is different from the subset zero-one loss. The zero-one loss considers the entire set of labels for a given sample incorrect if it does not entirely match the true set of labels. Hamming loss is more forgiving in that it penalizes only the individual labels. The Hamming loss is upperbounded by the subset zero-one loss, when normalize parameter is set to True. It is always between 0 and 1, lower being better. References  
1  
Grigorios Tsoumakas, Ioannis Katakis. Multi-Label Classification: An Overview. International Journal of Data Warehousing & Mining, 3(3), 1-13, July-September 2007.  
2  
Wikipedia entry on the Hamming distance.   Examples >>> from sklearn.metrics import hamming_loss
>>> y_pred = [1, 2, 3, 4]
>>> y_true = [2, 2, 3, 4]
>>> hamming_loss(y_true, y_pred)
0.25
 In the multilabel case with binary label indicators: >>> import numpy as np
>>> hamming_loss(np.array([[0, 1], [1, 1]]), np.zeros((2, 2)))
0.75
sklearn.metrics.hinge_loss(y_true, pred_decision, *, labels=None, sample_weight=None) [source]
 
Average hinge loss (non-regularized). In binary class case, assuming labels in y_true are encoded with +1 and -1, when a prediction mistake is made, margin = y_true * pred_decision is always negative (since the signs disagree), implying 1 - margin is always greater than 1. The cumulated hinge loss is therefore an upper bound of the number of mistakes made by the classifier. In multiclass case, the function expects that either all the labels are included in y_true or an optional labels argument is provided which contains all the labels. The multilabel margin is calculated according to Crammer-Singer’s method. As in the binary case, the cumulated hinge loss is an upper bound of the number of mistakes made by the classifier. Read more in the User Guide.  Parameters 
 
y_truearray of shape (n_samples,) 

True target, consisting of integers of two values. The positive label must be greater than the negative label.  
pred_decisionarray of shape (n_samples,) or (n_samples, n_classes) 

Predicted decisions, as output by decision_function (floats).  
labelsarray-like, default=None 

Contains all the labels for the problem. Used in multiclass hinge loss.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
lossfloat 
   References  
1  
Wikipedia entry on the Hinge loss.  
2  
Koby Crammer, Yoram Singer. On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines. Journal of Machine Learning Research 2, (2001), 265-292.  
3  
L1 AND L2 Regularization for Multiclass Hinge Loss Models by Robert C. Moore, John DeNero.   Examples >>> from sklearn import svm
>>> from sklearn.metrics import hinge_loss
>>> X = [[0], [1]]
>>> y = [-1, 1]
>>> est = svm.LinearSVC(random_state=0)
>>> est.fit(X, y)
LinearSVC(random_state=0)
>>> pred_decision = est.decision_function([[-2], [3], [0.5]])
>>> pred_decision
array([-2.18...,  2.36...,  0.09...])
>>> hinge_loss([-1, 1, 1], pred_decision)
0.30...
 In the multiclass case: >>> import numpy as np
>>> X = np.array([[0], [1], [2], [3]])
>>> Y = np.array([0, 1, 2, 3])
>>> labels = np.array([0, 1, 2, 3])
>>> est = svm.LinearSVC()
>>> est.fit(X, Y)
LinearSVC()
>>> pred_decision = est.decision_function([[-1], [2], [3]])
>>> y_true = [0, 2, 3]
>>> hinge_loss(y_true, pred_decision, labels=labels)
0.56...
sklearn.metrics.homogeneity_completeness_v_measure(labels_true, labels_pred, *, beta=1.0) [source]
 
Compute the homogeneity and completeness and V-Measure scores at once. Those metrics are based on normalized conditional entropy measures of the clustering labeling to evaluate given the knowledge of a Ground Truth class labels of the same samples. A clustering result satisfies homogeneity if all of its clusters contain only data points which are members of a single class. A clustering result satisfies completeness if all the data points that are members of a given class are elements of the same cluster. Both scores have positive values between 0.0 and 1.0, larger values being desirable. Those 3 metrics are independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score values in any way. V-Measure is furthermore symmetric: swapping labels_true and label_pred will give the same score. This does not hold for homogeneity and completeness. V-Measure is identical to normalized_mutual_info_score with the arithmetic averaging method. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

cluster labels to evaluate  
betafloat, default=1.0 

Ratio of weight attributed to homogeneity vs completeness. If beta is greater than 1, completeness is weighted more strongly in the calculation. If beta is less than 1, homogeneity is weighted more strongly.    Returns 
 
homogeneityfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly homogeneous labeling  
completenessfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly complete labeling  
v_measurefloat 

harmonic mean of the first two      See also  
homogeneity_score


completeness_score


v_measure_score
sklearn.metrics.homogeneity_score(labels_true, labels_pred) [source]
 
Homogeneity metric of a cluster labeling given a ground truth. A clustering result satisfies homogeneity if all of its clusters contain only data points which are members of a single class. This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way. This metric is not symmetric: switching label_true with label_pred will return the completeness_score which will be different in general. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

cluster labels to evaluate    Returns 
 
homogeneityfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly homogeneous labeling      See also  
completeness_score


v_measure_score

  References  
1  
Andrew Rosenberg and Julia Hirschberg, 2007. V-Measure: A conditional entropy-based external cluster evaluation measure   Examples Perfect labelings are homogeneous: >>> from sklearn.metrics.cluster import homogeneity_score
>>> homogeneity_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 Non-perfect labelings that further split classes into more clusters can be perfectly homogeneous: >>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 0, 1, 2]))
1.000000
>>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 1, 2, 3]))
1.000000
 Clusters that include samples from different classes do not make for an homogeneous labeling: >>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 1, 0, 1]))
0.0...
>>> print("%.6f" % homogeneity_score([0, 0, 1, 1], [0, 0, 0, 0]))
0.0...
sklearn.metrics.label_ranking_average_precision_score(y_true, y_score, *, sample_weight=None) [source]
 
Compute ranking-based average precision. Label ranking average precision (LRAP) is the average over each ground truth label assigned to each sample, of the ratio of true vs. total labels with lower score. This metric is used in multilabel ranking problem, where the goal is to give better rank to the labels associated to each sample. The obtained score is always strictly greater than 0 and the best value is 1. Read more in the User Guide.  Parameters 
 
y_true{ndarray, sparse matrix} of shape (n_samples, n_labels) 

True binary labels in binary indicator format.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.20.     Returns 
 
scorefloat 
   Examples >>> import numpy as np
>>> from sklearn.metrics import label_ranking_average_precision_score
>>> y_true = np.array([[1, 0, 0], [0, 0, 1]])
>>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]])
>>> label_ranking_average_precision_score(y_true, y_score)
0.416...
sklearn.metrics.label_ranking_loss(y_true, y_score, *, sample_weight=None) [source]
 
Compute Ranking loss measure. Compute the average number of label pairs that are incorrectly ordered given y_score weighted by the size of the label set and the number of labels not in the label set. This is similar to the error set size, but weighted by the number of relevant and irrelevant labels. The best performance is achieved with a ranking loss of zero. Read more in the User Guide.  New in version 0.17: A function label_ranking_loss   Parameters 
 
y_true{ndarray, sparse matrix} of shape (n_samples, n_labels) 

True binary labels in binary indicator format.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
lossfloat 
   References  
1  
Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US.
sklearn.metrics.log_loss(y_true, y_pred, *, eps=1e-15, normalize=True, sample_weight=None, labels=None) [source]
 
Log loss, aka logistic loss or cross-entropy loss. This is the loss function used in (multinomial) logistic regression and extensions of it such as neural networks, defined as the negative log-likelihood of a logistic model that returns y_pred probabilities for its training data y_true. The log loss is only defined for two or more labels. For a single sample with true label \(y \in \{0,1\}\) and and a probability estimate \(p = \operatorname{Pr}(y = 1)\), the log loss is:  \[L_{\log}(y, p) = -(y \log (p) + (1 - y) \log (1 - p))\] Read more in the User Guide.  Parameters 
 
y_truearray-like or label indicator matrix 

Ground truth (correct) labels for n_samples samples.  
y_predarray-like of float, shape = (n_samples, n_classes) or (n_samples,) 

Predicted probabilities, as returned by a classifier’s predict_proba method. If y_pred.shape = (n_samples,) the probabilities provided are assumed to be that of the positive class. The labels in y_pred are assumed to be ordered alphabetically, as done by preprocessing.LabelBinarizer.  
epsfloat, default=1e-15 

Log loss is undefined for p=0 or p=1, so probabilities are clipped to max(eps, min(1 - eps, p)).  
normalizebool, default=True 

If true, return the mean loss per sample. Otherwise, return the sum of the per-sample losses.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
labelsarray-like, default=None 

If not provided, labels will be inferred from y_true. If labels is None and y_pred has shape (n_samples,) the labels are assumed to be binary and are inferred from y_true.  New in version 0.18.     Returns 
 
lossfloat 
   Notes The logarithm used is the natural logarithm (base-e). References C.M. Bishop (2006). Pattern Recognition and Machine Learning. Springer, p. 209. Examples >>> from sklearn.metrics import log_loss
>>> log_loss(["spam", "ham", "ham", "spam"],
...          [[.1, .9], [.9, .1], [.8, .2], [.35, .65]])
0.21616...
sklearn.metrics.make_scorer(score_func, *, greater_is_better=True, needs_proba=False, needs_threshold=False, **kwargs) [source]
 
Make a scorer from a performance metric or loss function. This factory function wraps scoring functions for use in GridSearchCV and cross_val_score. It takes a score function, such as accuracy_score, mean_squared_error, adjusted_rand_index or average_precision and returns a callable that scores an estimator’s output. The signature of the call is (estimator, X, y) where estimator is the model to be evaluated, X is the data and y is the ground truth labeling (or None in the case of unsupervised models). Read more in the User Guide.  Parameters 
 
score_funccallable 

Score function (or loss function) with signature score_func(y, y_pred, **kwargs).  
greater_is_betterbool, default=True 

Whether score_func is a score function (default), meaning high is good, or a loss function, meaning low is good. In the latter case, the scorer object will sign-flip the outcome of the score_func.  
needs_probabool, default=False 

Whether score_func requires predict_proba to get probability estimates out of a classifier. If True, for binary y_true, the score function is supposed to accept a 1D y_pred (i.e., probability of the positive class, shape (n_samples,)).  
needs_thresholdbool, default=False 

Whether score_func takes a continuous decision certainty. This only works for binary classification using estimators that have either a decision_function or predict_proba method. If True, for binary y_true, the score function is supposed to accept a 1D y_pred (i.e., probability of the positive class or the decision function, shape (n_samples,)). For example average_precision or the area under the roc curve can not be computed using discrete predictions alone.  
**kwargsadditional arguments 

Additional parameters to be passed to score_func.    Returns 
 
scorercallable 

Callable object that returns a scalar score; greater is better.     Notes If needs_proba=False and needs_threshold=False, the score function is supposed to accept the output of predict. If needs_proba=True, the score function is supposed to accept the output of predict_proba (For binary y_true, the score function is supposed to accept probability of the positive class). If needs_threshold=True, the score function is supposed to accept the output of decision_function. Examples >>> from sklearn.metrics import fbeta_score, make_scorer
>>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
>>> ftwo_scorer
make_scorer(fbeta_score, beta=2)
>>> from sklearn.model_selection import GridSearchCV
>>> from sklearn.svm import LinearSVC
>>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]},
...                     scoring=ftwo_scorer)
sklearn.metrics.matthews_corrcoef(y_true, y_pred, *, sample_weight=None) [source]
 
Compute the Matthews correlation coefficient (MCC). The Matthews correlation coefficient is used in machine learning as a measure of the quality of binary and multiclass classifications. It takes into account true and false positives and negatives and is generally regarded as a balanced measure which can be used even if the classes are of very different sizes. The MCC is in essence a correlation coefficient value between -1 and +1. A coefficient of +1 represents a perfect prediction, 0 an average random prediction and -1 an inverse prediction. The statistic is also known as the phi coefficient. [source: Wikipedia] Binary and multiclass labels are supported. Only in the binary case does this relate to information about true and false positives and negatives. See references below. Read more in the User Guide.  Parameters 
 
y_truearray, shape = [n_samples] 

Ground truth (correct) target values.  
y_predarray, shape = [n_samples] 

Estimated targets as returned by a classifier.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.18.     Returns 
 
mccfloat 

The Matthews correlation coefficient (+1 represents a perfect prediction, 0 an average random prediction and -1 and inverse prediction).     References  
1  
Baldi, Brunak, Chauvin, Andersen and Nielsen, (2000). Assessing the accuracy of prediction algorithms for classification: an overview.  
2  
Wikipedia entry for the Matthews Correlation Coefficient.  
3  
Gorodkin, (2004). Comparing two K-category assignments by a K-category correlation coefficient.  
4  
Jurman, Riccadonna, Furlanello, (2012). A Comparison of MCC and CEN Error Measures in MultiClass Prediction.   Examples >>> from sklearn.metrics import matthews_corrcoef
>>> y_true = [+1, +1, +1, -1]
>>> y_pred = [+1, -1, +1, +1]
>>> matthews_corrcoef(y_true, y_pred)
-0.33...
sklearn.metrics.max_error(y_true, y_pred) [source]
 
max_error metric calculates the maximum residual error. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) 

Estimated target values.    Returns 
 
max_errorfloat 

A positive floating point value (the best value is 0.0).     Examples >>> from sklearn.metrics import max_error
>>> y_true = [3, 2, 7, 1]
>>> y_pred = [4, 2, 7, 1]
>>> max_error(y_true, y_pred)
1
sklearn.metrics.mean_absolute_error(y_true, y_pred, *, sample_weight=None, multioutput='uniform_average') [source]
 
Mean absolute error regression loss. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) or (n_samples, n_outputs) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) or (n_samples, n_outputs) 

Estimated target values.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
multioutput{‘raw_values’, ‘uniform_average’} or array-like of shape (n_outputs,), default=’uniform_average’ 

Defines aggregating of multiple output values. Array-like value defines weights used to average errors.  ‘raw_values’ :

Returns a full set of errors in case of multioutput input.  ‘uniform_average’ :

Errors of all outputs are averaged with uniform weight.      Returns 
 
lossfloat or ndarray of floats 

If multioutput is ‘raw_values’, then mean absolute error is returned for each output separately. If multioutput is ‘uniform_average’ or an ndarray of weights, then the weighted average of all output errors is returned. MAE output is non-negative floating point. The best value is 0.0.     Examples >>> from sklearn.metrics import mean_absolute_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_absolute_error(y_true, y_pred)
0.5
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_absolute_error(y_true, y_pred)
0.75
>>> mean_absolute_error(y_true, y_pred, multioutput='raw_values')
array([0.5, 1. ])
>>> mean_absolute_error(y_true, y_pred, multioutput=[0.3, 0.7])
0.85...
sklearn.metrics.mean_absolute_percentage_error(y_true, y_pred, sample_weight=None, multioutput='uniform_average') [source]
 
Mean absolute percentage error regression loss. Note here that we do not represent the output as a percentage in range [0, 100]. Instead, we represent it in range [0, 1/eps]. Read more in the User Guide.  New in version 0.24.   Parameters 
 
y_truearray-like of shape (n_samples,) or (n_samples, n_outputs) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) or (n_samples, n_outputs) 

Estimated target values.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
multioutput{‘raw_values’, ‘uniform_average’} or array-like 

Defines aggregating of multiple output values. Array-like value defines weights used to average errors. If input is list then the shape must be (n_outputs,).  ‘raw_values’ :

Returns a full set of errors in case of multioutput input.  ‘uniform_average’ :

Errors of all outputs are averaged with uniform weight.      Returns 
 
lossfloat or ndarray of floats in the range [0, 1/eps] 

If multioutput is ‘raw_values’, then mean absolute percentage error is returned for each output separately. If multioutput is ‘uniform_average’ or an ndarray of weights, then the weighted average of all output errors is returned. MAPE output is non-negative floating point. The best value is 0.0. But note the fact that bad predictions can lead to arbitarily large MAPE values, especially if some y_true values are very close to zero. Note that we return a large value instead of inf when y_true is zero.     Examples >>> from sklearn.metrics import mean_absolute_percentage_error
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> mean_absolute_percentage_error(y_true, y_pred)
0.3273...
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> mean_absolute_percentage_error(y_true, y_pred)
0.5515...
>>> mean_absolute_percentage_error(y_true, y_pred, multioutput=[0.3, 0.7])
0.6198...
sklearn.metrics.mean_gamma_deviance(y_true, y_pred, *, sample_weight=None) [source]
 
Mean Gamma deviance regression loss. Gamma deviance is equivalent to the Tweedie deviance with the power parameter power=2. It is invariant to scaling of the target variable, and measures relative errors. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) 

Ground truth (correct) target values. Requires y_true > 0.  
y_predarray-like of shape (n_samples,) 

Estimated target values. Requires y_pred > 0.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
lossfloat 

A non-negative floating point value (the best value is 0.0).     Examples >>> from sklearn.metrics import mean_gamma_deviance
>>> y_true = [2, 0.5, 1, 4]
>>> y_pred = [0.5, 0.5, 2., 2.]
>>> mean_gamma_deviance(y_true, y_pred)
1.0568...
sklearn.metrics.mean_poisson_deviance(y_true, y_pred, *, sample_weight=None) [source]
 
Mean Poisson deviance regression loss. Poisson deviance is equivalent to the Tweedie deviance with the power parameter power=1. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) 

Ground truth (correct) target values. Requires y_true >= 0.  
y_predarray-like of shape (n_samples,) 

Estimated target values. Requires y_pred > 0.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
lossfloat 

A non-negative floating point value (the best value is 0.0).     Examples >>> from sklearn.metrics import mean_poisson_deviance
>>> y_true = [2, 0, 1, 4]
>>> y_pred = [0.5, 0.5, 2., 2.]
>>> mean_poisson_deviance(y_true, y_pred)
1.4260...