neulab/docprompting-conala huggingface dataset:
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None) [source]
 
Accuracy classification score. In multilabel classification, this function computes subset accuracy: the set of labels predicted for a sample must exactly match the corresponding set of labels in y_true. Read more in the User Guide.  Parameters 
 
y_true1d array-like, or label indicator array / sparse matrix 

Ground truth (correct) labels.  
y_pred1d array-like, or label indicator array / sparse matrix 

Predicted labels, as returned by a classifier.  
normalizebool, default=True 

If False, return the number of correctly classified samples. Otherwise, return the fraction of correctly classified samples.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
scorefloat 

If normalize == True, return the fraction of correctly classified samples (float), else returns the number of correctly classified samples (int). The best performance is 1 with normalize == True and the number of samples with normalize == False.      See also  
jaccard_score, hamming_loss, zero_one_loss

  Notes In binary and multiclass classification, this function is equal to the jaccard_score function. Examples >>> from sklearn.metrics import accuracy_score
>>> y_pred = [0, 2, 1, 3]
>>> y_true = [0, 1, 2, 3]
>>> accuracy_score(y_true, y_pred)
0.5
>>> accuracy_score(y_true, y_pred, normalize=False)
2
 In the multilabel case with binary label indicators: >>> import numpy as np
>>> accuracy_score(np.array([[0, 1], [1, 1]]), np.ones((2, 2)))
0.5
sklearn.metrics.adjusted_mutual_info_score(labels_true, labels_pred, *, average_method='arithmetic') [source]
 
Adjusted Mutual Information between two clusterings. Adjusted Mutual Information (AMI) is an adjustment of the Mutual Information (MI) score to account for chance. It accounts for the fact that the MI is generally higher for two clusterings with a larger number of clusters, regardless of whether there is actually more information shared. For two clusterings \(U\) and \(V\), the AMI is given as: AMI(U, V) = [MI(U, V) - E(MI(U, V))] / [avg(H(U), H(V)) - E(MI(U, V))]
 This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way. This metric is furthermore symmetric: switching label_true with label_pred will return the same score value. This can be useful to measure the agreement of two independent label assignments strategies on the same dataset when the real ground truth is not known. Be mindful that this function is an order of magnitude slower than other metrics, such as the Adjusted Rand Index. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

A clustering of the data into disjoint subsets.  
labels_predint array-like of shape (n_samples,) 

A clustering of the data into disjoint subsets.  
average_methodstr, default=’arithmetic’ 

How to compute the normalizer in the denominator. Possible options are ‘min’, ‘geometric’, ‘arithmetic’, and ‘max’.  New in version 0.20.   Changed in version 0.22: The default value of average_method changed from ‘max’ to ‘arithmetic’.     Returns 
 ami: float (upperlimited by 1.0)

The AMI returns a value of 1 when the two partitions are identical (ie perfectly matched). Random partitions (independent labellings) have an expected AMI around 0 on average hence can be negative.      See also  
adjusted_rand_score


Adjusted Rand Index.  
mutual_info_score


Mutual Information (not adjusted for chance).    References  
1  
Vinh, Epps, and Bailey, (2010). Information Theoretic Measures for Clusterings Comparison: Variants, Properties, Normalization and Correction for Chance, JMLR  
2  
Wikipedia entry for the Adjusted Mutual Information   Examples Perfect labelings are both homogeneous and complete, hence have score 1.0: >>> from sklearn.metrics.cluster import adjusted_mutual_info_score
>>> adjusted_mutual_info_score([0, 0, 1, 1], [0, 0, 1, 1])
... 
1.0
>>> adjusted_mutual_info_score([0, 0, 1, 1], [1, 1, 0, 0])
... 
1.0
 If classes members are completely split across different clusters, the assignment is totally in-complete, hence the AMI is null: >>> adjusted_mutual_info_score([0, 0, 0, 0], [0, 1, 2, 3])
... 
0.0
sklearn.metrics.adjusted_rand_score(labels_true, labels_pred) [source]
 
Rand index adjusted for chance. The Rand Index computes a similarity measure between two clusterings by considering all pairs of samples and counting pairs that are assigned in the same or different clusters in the predicted and true clusterings. The raw RI score is then “adjusted for chance” into the ARI score using the following scheme: ARI = (RI - Expected_RI) / (max(RI) - Expected_RI)
 The adjusted Rand index is thus ensured to have a value close to 0.0 for random labeling independently of the number of clusters and samples and exactly 1.0 when the clusterings are identical (up to a permutation). ARI is a symmetric measure: adjusted_rand_score(a, b) == adjusted_rand_score(b, a)
 Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

Ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

Cluster labels to evaluate    Returns 
 
ARIfloat 

Similarity score between -1.0 and 1.0. Random labelings have an ARI close to 0.0. 1.0 stands for perfect match.      See also  
adjusted_mutual_info_score


Adjusted Mutual Information.    References  
Hubert1985  
L. Hubert and P. Arabie, Comparing Partitions, Journal of Classification 1985 https://link.springer.com/article/10.1007%2FBF01908075  
Steinley2004  
D. Steinley, Properties of the Hubert-Arabie adjusted Rand index, Psychological Methods 2004  
wk  
https://en.wikipedia.org/wiki/Rand_index#Adjusted_Rand_index   Examples Perfectly matching labelings have a score of 1 even >>> from sklearn.metrics.cluster import adjusted_rand_score
>>> adjusted_rand_score([0, 0, 1, 1], [0, 0, 1, 1])
1.0
>>> adjusted_rand_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 Labelings that assign all classes members to the same clusters are complete but may not always be pure, hence penalized: >>> adjusted_rand_score([0, 0, 1, 2], [0, 0, 1, 1])
0.57...
 ARI is symmetric, so labelings that have pure clusters with members coming from the same classes but unnecessary splits are penalized: >>> adjusted_rand_score([0, 0, 1, 1], [0, 0, 1, 2])
0.57...
 If classes members are completely split across different clusters, the assignment is totally incomplete, hence the ARI is very low: >>> adjusted_rand_score([0, 0, 0, 0], [0, 1, 2, 3])
0.0
sklearn.metrics.auc(x, y) [source]
 
Compute Area Under the Curve (AUC) using the trapezoidal rule. This is a general function, given points on a curve. For computing the area under the ROC-curve, see roc_auc_score. For an alternative way to summarize a precision-recall curve, see average_precision_score.  Parameters 
 
xndarray of shape (n,) 

x coordinates. These must be either monotonic increasing or monotonic decreasing.  
yndarray of shape, (n,) 

y coordinates.    Returns 
 
aucfloat 
    See also  
roc_auc_score


Compute the area under the ROC curve.  
average_precision_score


Compute average precision from prediction scores.  
precision_recall_curve


Compute precision-recall pairs for different probability thresholds.    Examples >>> import numpy as np
>>> from sklearn import metrics
>>> y = np.array([1, 1, 2, 2])
>>> pred = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = metrics.roc_curve(y, pred, pos_label=2)
>>> metrics.auc(fpr, tpr)
0.75
sklearn.metrics.average_precision_score(y_true, y_score, *, average='macro', pos_label=1, sample_weight=None) [source]
 
Compute average precision (AP) from prediction scores. AP summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight:  \[\text{AP} = \sum_n (R_n - R_{n-1}) P_n\] where \(P_n\) and \(R_n\) are the precision and recall at the nth threshold [1]. This implementation is not interpolated and is different from computing the area under the precision-recall curve with the trapezoidal rule, which uses linear interpolation and can be too optimistic. Note: this implementation is restricted to the binary classification task or multilabel classification task. Read more in the User Guide.  Parameters 
 
y_truendarray of shape (n_samples,) or (n_samples, n_classes) 

True binary labels or binary label indicators.  
y_scorendarray of shape (n_samples,) or (n_samples, n_classes) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by decision_function on some classifiers).  
average{‘micro’, ‘samples’, ‘weighted’, ‘macro’} or None, default=’macro’ 

If None, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data:  
'micro': 

Calculate metrics globally by considering each element of the label indicator matrix as a label.  
'macro': 

Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.  
'weighted': 

Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label).  
'samples': 

Calculate metrics for each instance, and find their average.   Will be ignored when y_true is binary.  
pos_labelint or str, default=1 

The label of the positive class. Only applied to binary y_true. For multilabel-indicator y_true, pos_label is fixed to 1.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
average_precisionfloat 
    See also  
roc_auc_score


Compute the area under the ROC curve.  
precision_recall_curve


Compute precision-recall pairs for different probability thresholds.    Notes  Changed in version 0.19: Instead of linearly interpolating between operating points, precisions are weighted by the change in recall since the last operating point.  References  
1  
Wikipedia entry for the Average precision   Examples >>> import numpy as np
>>> from sklearn.metrics import average_precision_score
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> average_precision_score(y_true, y_scores)
0.83...
sklearn.metrics.balanced_accuracy_score(y_true, y_pred, *, sample_weight=None, adjusted=False) [source]
 
Compute the balanced accuracy. The balanced accuracy in binary and multiclass classification problems to deal with imbalanced datasets. It is defined as the average of recall obtained on each class. The best value is 1 and the worst value is 0 when adjusted=False. Read more in the User Guide.  New in version 0.20.   Parameters 
 
y_true1d array-like 

Ground truth (correct) target values.  
y_pred1d array-like 

Estimated targets as returned by a classifier.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
adjustedbool, default=False 

When true, the result is adjusted for chance, so that random performance would score 0, and perfect performance scores 1.    Returns 
 
balanced_accuracyfloat 
    See also  
recall_score, roc_auc_score

  Notes Some literature promotes alternative definitions of balanced accuracy. Our definition is equivalent to accuracy_score with class-balanced sample weights, and shares desirable properties with the binary case. See the User Guide. References  
1  
Brodersen, K.H.; Ong, C.S.; Stephan, K.E.; Buhmann, J.M. (2010). The balanced accuracy and its posterior distribution. Proceedings of the 20th International Conference on Pattern Recognition, 3121-24.  
2  
John. D. Kelleher, Brian Mac Namee, Aoife D’Arcy, (2015). Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies.   Examples >>> from sklearn.metrics import balanced_accuracy_score
>>> y_true = [0, 1, 0, 0, 1, 0]
>>> y_pred = [0, 1, 0, 0, 0, 1]
>>> balanced_accuracy_score(y_true, y_pred)
0.625
sklearn.metrics.brier_score_loss(y_true, y_prob, *, sample_weight=None, pos_label=None) [source]
 
Compute the Brier score loss. The smaller the Brier score loss, the better, hence the naming with “loss”. The Brier score measures the mean squared difference between the predicted probability and the actual outcome. The Brier score always takes on a value between zero and one, since this is the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 and 1). It can be decomposed is the sum of refinement loss and calibration loss. The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but is inappropriate for ordinal variables which can take on three or more values (this is because the Brier score assumes that all possible outcomes are equivalently “distant” from one another). Which label is considered to be the positive label is controlled via the parameter pos_label, which defaults to the greater label unless y_true is all 0 or all -1, in which case pos_label defaults to 1. Read more in the User Guide.  Parameters 
 
y_truearray of shape (n_samples,) 

True targets.  
y_probarray of shape (n_samples,) 

Probabilities of the positive class.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
pos_labelint or str, default=None 

Label of the positive class. pos_label will be infered in the following manner:  if y_true in {-1, 1} or {0, 1}, pos_label defaults to 1; else if y_true contains string, an error will be raised and pos_label should be explicitely specified; otherwise, pos_label defaults to the greater label, i.e. np.unique(y_true)[-1].     Returns 
 
scorefloat 

Brier score loss.     References  
1  
Wikipedia entry for the Brier score.   Examples >>> import numpy as np
>>> from sklearn.metrics import brier_score_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
>>> y_prob = np.array([0.1, 0.9, 0.8, 0.3])
>>> brier_score_loss(y_true, y_prob)
0.037...
>>> brier_score_loss(y_true, 1-y_prob, pos_label=0)
0.037...
>>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
0.037...
>>> brier_score_loss(y_true, np.array(y_prob) > 0.5)
0.0
sklearn.metrics.calinski_harabasz_score(X, labels) [source]
 
Compute the Calinski and Harabasz score. It is also known as the Variance Ratio Criterion. The score is defined as ratio between the within-cluster dispersion and the between-cluster dispersion. Read more in the User Guide.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

A list of n_features-dimensional data points. Each row corresponds to a single data point.  
labelsarray-like of shape (n_samples,) 

Predicted labels for each sample.    Returns 
 
scorefloat 

The resulting Calinski-Harabasz score.     References  
1  
T. Calinski and J. Harabasz, 1974. “A dendrite method for cluster analysis”. Communications in Statistics
sklearn.metrics.check_scoring(estimator, scoring=None, *, allow_none=False) [source]
 
Determine scorer from user options. A TypeError will be thrown if the estimator cannot be scored.  Parameters 
 
estimatorestimator object implementing ‘fit’ 

The object to use to fit the data.  
scoringstr or callable, default=None 

A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y).  
allow_nonebool, default=False 

If no scoring is specified and the estimator has no score function, we can either return None or raise an exception.    Returns 
 
scoringcallable 

A scorer callable object / function with signature scorer(estimator, X, y).
sklearn.metrics.cluster.contingency_matrix(labels_true, labels_pred, *, eps=None, sparse=False, dtype=<class 'numpy.int64'>) [source]
 
Build a contingency matrix describing the relationship between labels.  Parameters 
 
labels_trueint array, shape = [n_samples] 

Ground truth class labels to be used as a reference.  
labels_predarray-like of shape (n_samples,) 

Cluster labels to evaluate.  
epsfloat, default=None 

If a float, that value is added to all values in the contingency matrix. This helps to stop NaN propagation. If None, nothing is adjusted.  
sparsebool, default=False 

If True, return a sparse CSR continency matrix. If eps is not None and sparse is True will raise ValueError.  New in version 0.18.   
dtypenumeric type, default=np.int64 

Output dtype. Ignored if eps is not None.  New in version 0.24.     Returns 
 
contingency{array-like, sparse}, shape=[n_classes_true, n_classes_pred] 

Matrix \(C\) such that \(C_{i, j}\) is the number of samples in true class \(i\) and in predicted class \(j\). If eps is None, the dtype of this array will be integer unless set otherwise with the dtype argument. If eps is given, the dtype will be float. Will be a sklearn.sparse.csr_matrix if sparse=True.
sklearn.metrics.cluster.pair_confusion_matrix(labels_true, labels_pred) [source]
 
Pair confusion matrix arising from two clusterings. The pair confusion matrix \(C\) computes a 2 by 2 similarity matrix between two clusterings by considering all pairs of samples and counting pairs that are assigned into the same or into different clusters under the true and predicted clusterings. Considering a pair of samples that is clustered together a positive pair, then as in binary classification the count of true negatives is \(C_{00}\), false negatives is \(C_{10}\), true positives is \(C_{11}\) and false positives is \(C_{01}\). Read more in the User Guide.  Parameters 
 
labels_truearray-like of shape (n_samples,), dtype=integral 

Ground truth class labels to be used as a reference.  
labels_predarray-like of shape (n_samples,), dtype=integral 

Cluster labels to evaluate.    Returns 
 
Cndarray of shape (2, 2), dtype=np.int64 

The contingency matrix.      See also  
rand_score 

Rand Score  
adjusted_rand_score 

Adjusted Rand Score  
adjusted_mutual_info_score 

Adjusted Mutual Information    References Examples Perfectly matching labelings have all non-zero entries on the diagonal regardless of actual label values: >>> from sklearn.metrics.cluster import pair_confusion_matrix
>>> pair_confusion_matrix([0, 0, 1, 1], [1, 1, 0, 0])
array([[8, 0],
       [0, 4]]...
 Labelings that assign all classes members to the same clusters are complete but may be not always pure, hence penalized, and have some off-diagonal non-zero entries: >>> pair_confusion_matrix([0, 0, 1, 2], [0, 0, 1, 1])
array([[8, 2],
       [0, 2]]...
 Note that the matrix is not symmetric.
sklearn.metrics.cohen_kappa_score(y1, y2, *, labels=None, weights=None, sample_weight=None) [source]
 
Cohen’s kappa: a statistic that measures inter-annotator agreement. This function computes Cohen’s kappa [1], a score that expresses the level of agreement between two annotators on a classification problem. It is defined as  \[\kappa = (p_o - p_e) / (1 - p_e)\] where \(p_o\) is the empirical probability of agreement on the label assigned to any sample (the observed agreement ratio), and \(p_e\) is the expected agreement when both annotators assign labels randomly. \(p_e\) is estimated using a per-annotator empirical prior over the class labels [2]. Read more in the User Guide.  Parameters 
 
y1array of shape (n_samples,) 

Labels assigned by the first annotator.  
y2array of shape (n_samples,) 

Labels assigned by the second annotator. The kappa statistic is symmetric, so swapping y1 and y2 doesn’t change the value.  
labelsarray-like of shape (n_classes,), default=None 

List of labels to index the matrix. This may be used to select a subset of labels. If None, all labels that appear at least once in y1 or y2 are used.  
weights{‘linear’, ‘quadratic’}, default=None 

Weighting type to calculate the score. None means no weighted; “linear” means linear weighted; “quadratic” means quadratic weighted.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
kappafloat 

The kappa statistic, which is a number between -1 and 1. The maximum value means complete agreement; zero or lower means chance agreement.     References  
1  
J. Cohen (1960). “A coefficient of agreement for nominal scales”. Educational and Psychological Measurement 20(1):37-46. doi:10.1177/001316446002000104.  
2  
R. Artstein and M. Poesio (2008). “Inter-coder agreement for computational linguistics”. Computational Linguistics 34(4):555-596.  
3  
Wikipedia entry for the Cohen’s kappa.
sklearn.metrics.completeness_score(labels_true, labels_pred) [source]
 
Completeness metric of a cluster labeling given a ground truth. A clustering result satisfies completeness if all the data points that are members of a given class are elements of the same cluster. This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way. This metric is not symmetric: switching label_true with label_pred will return the homogeneity_score which will be different in general. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

cluster labels to evaluate    Returns 
 
completenessfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly complete labeling      See also  
homogeneity_score


v_measure_score

  References  
1  
Andrew Rosenberg and Julia Hirschberg, 2007. V-Measure: A conditional entropy-based external cluster evaluation measure   Examples Perfect labelings are complete: >>> from sklearn.metrics.cluster import completeness_score
>>> completeness_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 Non-perfect labelings that assign all classes members to the same clusters are still complete: >>> print(completeness_score([0, 0, 1, 1], [0, 0, 0, 0]))
1.0
>>> print(completeness_score([0, 1, 2, 3], [0, 0, 1, 1]))
0.999...
 If classes members are split across different clusters, the assignment cannot be complete: >>> print(completeness_score([0, 0, 1, 1], [0, 1, 0, 1]))
0.0
>>> print(completeness_score([0, 0, 0, 0], [0, 1, 2, 3]))
0.0