neulab/docprompting-conala huggingface dataset:
sklearn.metrics.balanced_accuracy_score  
sklearn.metrics.balanced_accuracy_score(y_true, y_pred, *, sample_weight=None, adjusted=False) [source]
 
Compute the balanced accuracy. The balanced accuracy in binary and multiclass classification problems to deal with imbalanced datasets. It is defined as the average of recall obtained on each class. The best value is 1 and the worst value is 0 when adjusted=False. Read more in the User Guide.  New in version 0.20.   Parameters 
 
y_true1d array-like 

Ground truth (correct) target values.  
y_pred1d array-like 

Estimated targets as returned by a classifier.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
adjustedbool, default=False 

When true, the result is adjusted for chance, so that random performance would score 0, and perfect performance scores 1.    Returns 
 
balanced_accuracyfloat 
    See also  
recall_score, roc_auc_score

  Notes Some literature promotes alternative definitions of balanced accuracy. Our definition is equivalent to accuracy_score with class-balanced sample weights, and shares desirable properties with the binary case. See the User Guide. References  
1  
Brodersen, K.H.; Ong, C.S.; Stephan, K.E.; Buhmann, J.M. (2010). The balanced accuracy and its posterior distribution. Proceedings of the 20th International Conference on Pattern Recognition, 3121-24.  
2  
John. D. Kelleher, Brian Mac Namee, Aoife D’Arcy, (2015). Fundamentals of Machine Learning for Predictive Data Analytics: Algorithms, Worked Examples, and Case Studies.   Examples >>> from sklearn.metrics import balanced_accuracy_score
>>> y_true = [0, 1, 0, 0, 1, 0]
>>> y_pred = [0, 1, 0, 0, 0, 1]
>>> balanced_accuracy_score(y_true, y_pred)
0.625
sklearn.metrics.brier_score_loss  
sklearn.metrics.brier_score_loss(y_true, y_prob, *, sample_weight=None, pos_label=None) [source]
 
Compute the Brier score loss. The smaller the Brier score loss, the better, hence the naming with “loss”. The Brier score measures the mean squared difference between the predicted probability and the actual outcome. The Brier score always takes on a value between zero and one, since this is the largest possible difference between a predicted probability (which must be between zero and one) and the actual outcome (which can take on values of only 0 and 1). It can be decomposed is the sum of refinement loss and calibration loss. The Brier score is appropriate for binary and categorical outcomes that can be structured as true or false, but is inappropriate for ordinal variables which can take on three or more values (this is because the Brier score assumes that all possible outcomes are equivalently “distant” from one another). Which label is considered to be the positive label is controlled via the parameter pos_label, which defaults to the greater label unless y_true is all 0 or all -1, in which case pos_label defaults to 1. Read more in the User Guide.  Parameters 
 
y_truearray of shape (n_samples,) 

True targets.  
y_probarray of shape (n_samples,) 

Probabilities of the positive class.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
pos_labelint or str, default=None 

Label of the positive class. pos_label will be infered in the following manner:  if y_true in {-1, 1} or {0, 1}, pos_label defaults to 1; else if y_true contains string, an error will be raised and pos_label should be explicitely specified; otherwise, pos_label defaults to the greater label, i.e. np.unique(y_true)[-1].     Returns 
 
scorefloat 

Brier score loss.     References  
1  
Wikipedia entry for the Brier score.   Examples >>> import numpy as np
>>> from sklearn.metrics import brier_score_loss
>>> y_true = np.array([0, 1, 1, 0])
>>> y_true_categorical = np.array(["spam", "ham", "ham", "spam"])
>>> y_prob = np.array([0.1, 0.9, 0.8, 0.3])
>>> brier_score_loss(y_true, y_prob)
0.037...
>>> brier_score_loss(y_true, 1-y_prob, pos_label=0)
0.037...
>>> brier_score_loss(y_true_categorical, y_prob, pos_label="ham")
0.037...
>>> brier_score_loss(y_true, np.array(y_prob) > 0.5)
0.0
 
 Examples using sklearn.metrics.brier_score_loss
 
  Probability Calibration curves  

  Probability calibration of classifiers
sklearn.metrics.calinski_harabasz_score  
sklearn.metrics.calinski_harabasz_score(X, labels) [source]
 
Compute the Calinski and Harabasz score. It is also known as the Variance Ratio Criterion. The score is defined as ratio between the within-cluster dispersion and the between-cluster dispersion. Read more in the User Guide.  Parameters 
 
Xarray-like of shape (n_samples, n_features) 

A list of n_features-dimensional data points. Each row corresponds to a single data point.  
labelsarray-like of shape (n_samples,) 

Predicted labels for each sample.    Returns 
 
scorefloat 

The resulting Calinski-Harabasz score.     References  
1  
T. Calinski and J. Harabasz, 1974. “A dendrite method for cluster analysis”. Communications in Statistics
sklearn.metrics.check_scoring  
sklearn.metrics.check_scoring(estimator, scoring=None, *, allow_none=False) [source]
 
Determine scorer from user options. A TypeError will be thrown if the estimator cannot be scored.  Parameters 
 
estimatorestimator object implementing ‘fit’ 

The object to use to fit the data.  
scoringstr or callable, default=None 

A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y).  
allow_nonebool, default=False 

If no scoring is specified and the estimator has no score function, we can either return None or raise an exception.    Returns 
 
scoringcallable 

A scorer callable object / function with signature scorer(estimator, X, y).
sklearn.metrics.cluster.contingency_matrix  
sklearn.metrics.cluster.contingency_matrix(labels_true, labels_pred, *, eps=None, sparse=False, dtype=<class 'numpy.int64'>) [source]
 
Build a contingency matrix describing the relationship between labels.  Parameters 
 
labels_trueint array, shape = [n_samples] 

Ground truth class labels to be used as a reference.  
labels_predarray-like of shape (n_samples,) 

Cluster labels to evaluate.  
epsfloat, default=None 

If a float, that value is added to all values in the contingency matrix. This helps to stop NaN propagation. If None, nothing is adjusted.  
sparsebool, default=False 

If True, return a sparse CSR continency matrix. If eps is not None and sparse is True will raise ValueError.  New in version 0.18.   
dtypenumeric type, default=np.int64 

Output dtype. Ignored if eps is not None.  New in version 0.24.     Returns 
 
contingency{array-like, sparse}, shape=[n_classes_true, n_classes_pred] 

Matrix \(C\) such that \(C_{i, j}\) is the number of samples in true class \(i\) and in predicted class \(j\). If eps is None, the dtype of this array will be integer unless set otherwise with the dtype argument. If eps is given, the dtype will be float. Will be a sklearn.sparse.csr_matrix if sparse=True.
sklearn.metrics.cluster.pair_confusion_matrix  
sklearn.metrics.cluster.pair_confusion_matrix(labels_true, labels_pred) [source]
 
Pair confusion matrix arising from two clusterings. The pair confusion matrix \(C\) computes a 2 by 2 similarity matrix between two clusterings by considering all pairs of samples and counting pairs that are assigned into the same or into different clusters under the true and predicted clusterings. Considering a pair of samples that is clustered together a positive pair, then as in binary classification the count of true negatives is \(C_{00}\), false negatives is \(C_{10}\), true positives is \(C_{11}\) and false positives is \(C_{01}\). Read more in the User Guide.  Parameters 
 
labels_truearray-like of shape (n_samples,), dtype=integral 

Ground truth class labels to be used as a reference.  
labels_predarray-like of shape (n_samples,), dtype=integral 

Cluster labels to evaluate.    Returns 
 
Cndarray of shape (2, 2), dtype=np.int64 

The contingency matrix.      See also  
rand_score 

Rand Score  
adjusted_rand_score 

Adjusted Rand Score  
adjusted_mutual_info_score 

Adjusted Mutual Information    References Examples Perfectly matching labelings have all non-zero entries on the diagonal regardless of actual label values: >>> from sklearn.metrics.cluster import pair_confusion_matrix
>>> pair_confusion_matrix([0, 0, 1, 1], [1, 1, 0, 0])
array([[8, 0],
       [0, 4]]...
 Labelings that assign all classes members to the same clusters are complete but may be not always pure, hence penalized, and have some off-diagonal non-zero entries: >>> pair_confusion_matrix([0, 0, 1, 2], [0, 0, 1, 1])
array([[8, 2],
       [0, 2]]...
 Note that the matrix is not symmetric.
sklearn.metrics.cohen_kappa_score  
sklearn.metrics.cohen_kappa_score(y1, y2, *, labels=None, weights=None, sample_weight=None) [source]
 
Cohen’s kappa: a statistic that measures inter-annotator agreement. This function computes Cohen’s kappa [1], a score that expresses the level of agreement between two annotators on a classification problem. It is defined as  \[\kappa = (p_o - p_e) / (1 - p_e)\] where \(p_o\) is the empirical probability of agreement on the label assigned to any sample (the observed agreement ratio), and \(p_e\) is the expected agreement when both annotators assign labels randomly. \(p_e\) is estimated using a per-annotator empirical prior over the class labels [2]. Read more in the User Guide.  Parameters 
 
y1array of shape (n_samples,) 

Labels assigned by the first annotator.  
y2array of shape (n_samples,) 

Labels assigned by the second annotator. The kappa statistic is symmetric, so swapping y1 and y2 doesn’t change the value.  
labelsarray-like of shape (n_classes,), default=None 

List of labels to index the matrix. This may be used to select a subset of labels. If None, all labels that appear at least once in y1 or y2 are used.  
weights{‘linear’, ‘quadratic’}, default=None 

Weighting type to calculate the score. None means no weighted; “linear” means linear weighted; “quadratic” means quadratic weighted.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
kappafloat 

The kappa statistic, which is a number between -1 and 1. The maximum value means complete agreement; zero or lower means chance agreement.     References  
1  
J. Cohen (1960). “A coefficient of agreement for nominal scales”. Educational and Psychological Measurement 20(1):37-46. doi:10.1177/001316446002000104.  
2  
R. Artstein and M. Poesio (2008). “Inter-coder agreement for computational linguistics”. Computational Linguistics 34(4):555-596.  
3  
Wikipedia entry for the Cohen’s kappa.
sklearn.metrics.completeness_score  
sklearn.metrics.completeness_score(labels_true, labels_pred) [source]
 
Completeness metric of a cluster labeling given a ground truth. A clustering result satisfies completeness if all the data points that are members of a given class are elements of the same cluster. This metric is independent of the absolute values of the labels: a permutation of the class or cluster label values won’t change the score value in any way. This metric is not symmetric: switching label_true with label_pred will return the homogeneity_score which will be different in general. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = [n_samples] 

ground truth class labels to be used as a reference  
labels_predarray-like of shape (n_samples,) 

cluster labels to evaluate    Returns 
 
completenessfloat 

score between 0.0 and 1.0. 1.0 stands for perfectly complete labeling      See also  
homogeneity_score


v_measure_score

  References  
1  
Andrew Rosenberg and Julia Hirschberg, 2007. V-Measure: A conditional entropy-based external cluster evaluation measure   Examples Perfect labelings are complete: >>> from sklearn.metrics.cluster import completeness_score
>>> completeness_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 Non-perfect labelings that assign all classes members to the same clusters are still complete: >>> print(completeness_score([0, 0, 1, 1], [0, 0, 0, 0]))
1.0
>>> print(completeness_score([0, 1, 2, 3], [0, 0, 1, 1]))
0.999...
 If classes members are split across different clusters, the assignment cannot be complete: >>> print(completeness_score([0, 0, 1, 1], [0, 1, 0, 1]))
0.0
>>> print(completeness_score([0, 0, 0, 0], [0, 1, 2, 3]))
0.0
 
 Examples using sklearn.metrics.completeness_score
 
  Release Highlights for scikit-learn 0.23  

  Demo of affinity propagation clustering algorithm  

  Demo of DBSCAN clustering algorithm  

  A demo of K-Means clustering on the handwritten digits data  

  Clustering text documents using k-means
sklearn.metrics.confusion_matrix  
sklearn.metrics.confusion_matrix(y_true, y_pred, *, labels=None, sample_weight=None, normalize=None) [source]
 
Compute confusion matrix to evaluate the accuracy of a classification. By definition a confusion matrix \(C\) is such that \(C_{i, j}\) is equal to the number of observations known to be in group \(i\) and predicted to be in group \(j\). Thus in binary classification, the count of true negatives is \(C_{0,0}\), false negatives is \(C_{1,0}\), true positives is \(C_{1,1}\) and false positives is \(C_{0,1}\). Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) 

Estimated targets as returned by a classifier.  
labelsarray-like of shape (n_classes), default=None 

List of labels to index the matrix. This may be used to reorder or select a subset of labels. If None is given, those that appear at least once in y_true or y_pred are used in sorted order.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.18.   
normalize{‘true’, ‘pred’, ‘all’}, default=None 

Normalizes confusion matrix over the true (rows), predicted (columns) conditions or all the population. If None, confusion matrix will not be normalized.    Returns 
 
Cndarray of shape (n_classes, n_classes) 

Confusion matrix whose i-th row and j-th column entry indicates the number of samples with true label being i-th class and predicted label being j-th class.      See also  
plot_confusion_matrix


Plot Confusion Matrix.  
ConfusionMatrixDisplay


Confusion Matrix visualization.    References  
1  
Wikipedia entry for the Confusion matrix (Wikipedia and other references may use a different convention for axes).   Examples >>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])
 >>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
>>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
>>> confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])
 In the binary case, we can extract true positives, etc as follows: >>> tn, fp, fn, tp = confusion_matrix([0, 1, 0, 1], [1, 1, 1, 0]).ravel()
>>> (tn, fp, fn, tp)
(0, 2, 1, 1)
 
 Examples using sklearn.metrics.confusion_matrix
 
  Faces recognition example using eigenfaces and SVMs  

  Visualizations with Display Objects  

  Label Propagation digits: Demonstrating performance  

  Label Propagation digits active learning  

  Classification of text documents using sparse features
sklearn.metrics.consensus_score  
sklearn.metrics.consensus_score(a, b, *, similarity='jaccard') [source]
 
The similarity of two sets of biclusters. Similarity between individual biclusters is computed. Then the best matching between sets is found using the Hungarian algorithm. The final score is the sum of similarities divided by the size of the larger set. Read more in the User Guide.  Parameters 
 
a(rows, columns) 

Tuple of row and column indicators for a set of biclusters.  
b(rows, columns) 

Another set of biclusters like a.  
similarity‘jaccard’ or callable, default=’jaccard’ 

May be the string “jaccard” to use the Jaccard coefficient, or any function that takes four arguments, each of which is a 1d indicator vector: (a_rows, a_columns, b_rows, b_columns).     References  Hochreiter, Bodenhofer, et. al., 2010. FABIA: factor analysis for bicluster acquisition.  
 Examples using sklearn.metrics.consensus_score
 
  A demo of the Spectral Co-Clustering algorithm  

  A demo of the Spectral Biclustering algorithm
sklearn.metrics.coverage_error  
sklearn.metrics.coverage_error(y_true, y_score, *, sample_weight=None) [source]
 
Coverage error measure. Compute how far we need to go through the ranked scores to cover all true labels. The best value is equal to the average number of labels in y_true per sample. Ties in y_scores are broken by giving maximal rank that would have been assigned to all tied values. Note: Our implementation’s score is 1 greater than the one given in Tsoumakas et al., 2010. This extends it to handle the degenerate case in which an instance has 0 true labels. Read more in the User Guide.  Parameters 
 
y_truendarray of shape (n_samples, n_labels) 

True binary labels in binary indicator format.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
coverage_errorfloat 
   References  
1  
Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US.
sklearn.metrics.davies_bouldin_score  
sklearn.metrics.davies_bouldin_score(X, labels) [source]
 
Computes the Davies-Bouldin score. The score is defined as the average similarity measure of each cluster with its most similar cluster, where similarity is the ratio of within-cluster distances to between-cluster distances. Thus, clusters which are farther apart and less dispersed will result in a better score. The minimum score is zero, with lower values indicating better clustering. Read more in the User Guide.  New in version 0.20.   Parameters 
 
Xarray-like of shape (n_samples, n_features) 

A list of n_features-dimensional data points. Each row corresponds to a single data point.  
labelsarray-like of shape (n_samples,) 

Predicted labels for each sample.    Returns 
 score: float

The resulting Davies-Bouldin score.     References  
1  
Davies, David L.; Bouldin, Donald W. (1979). “A Cluster Separation Measure”. IEEE Transactions on Pattern Analysis and Machine Intelligence. PAMI-1 (2): 224-227
sklearn.metrics.dcg_score  
sklearn.metrics.dcg_score(y_true, y_score, *, k=None, log_base=2, sample_weight=None, ignore_ties=False) [source]
 
Compute Discounted Cumulative Gain. Sum the true scores ranked in the order induced by the predicted scores, after applying a logarithmic discount. This ranking metric yields a high value if true labels are ranked high by y_score. Usually the Normalized Discounted Cumulative Gain (NDCG, computed by ndcg_score) is preferred.  Parameters 
 
y_truendarray of shape (n_samples, n_labels) 

True targets of multilabel classification, or true scores of entities to be ranked.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
kint, default=None 

Only consider the highest k scores in the ranking. If None, use all outputs.  
log_basefloat, default=2 

Base of the logarithm used for the discount. A low value means a sharper discount (top results are more important).  
sample_weightndarray of shape (n_samples,), default=None 

Sample weights. If None, all samples are given the same weight.  
ignore_tiesbool, default=False 

Assume that there are no ties in y_score (which is likely to be the case if y_score is continuous) for efficiency gains.    Returns 
 
discounted_cumulative_gainfloat 

The averaged sample DCG scores.      See also  
ndcg_score


The Discounted Cumulative Gain divided by the Ideal Discounted Cumulative Gain (the DCG obtained for a perfect ranking), in order to have a score between 0 and 1.    References Wikipedia entry for Discounted Cumulative Gain. Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422-446. Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, May). A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th Annual Conference on Learning Theory (COLT 2013). McSherry, F., & Najork, M. (2008, March). Computing information retrieval performance measures efficiently in the presence of tied scores. In European conference on information retrieval (pp. 414-421). Springer, Berlin, Heidelberg. Examples >>> from sklearn.metrics import dcg_score
>>> # we have groud-truth relevance of some answers to a query:
>>> true_relevance = np.asarray([[10, 0, 0, 1, 5]])
>>> # we predict scores for the answers
>>> scores = np.asarray([[.1, .2, .3, 4, 70]])
>>> dcg_score(true_relevance, scores)
9.49...
>>> # we can set k to truncate the sum; only top k answers contribute
>>> dcg_score(true_relevance, scores, k=2)
5.63...
>>> # now we have some ties in our prediction
>>> scores = np.asarray([[1, 0, 0, 0, 1]])
>>> # by default ties are averaged, so here we get the average true
>>> # relevance of our top predictions: (10 + 5) / 2 = 7.5
>>> dcg_score(true_relevance, scores, k=1)
7.5
>>> # we can choose to ignore ties for faster results, but only
>>> # if we know there aren't ties in our scores, otherwise we get
>>> # wrong results:
>>> dcg_score(true_relevance,
...           scores, k=1, ignore_ties=True)
5.0
sklearn.metrics.det_curve  
sklearn.metrics.det_curve(y_true, y_score, pos_label=None, sample_weight=None) [source]
 
Compute error rates for different probability thresholds.  Note This metric is used for evaluation of ranking and error tradeoffs of a binary classification task.  Read more in the User Guide.  New in version 0.24.   Parameters 
 
y_truendarray of shape (n_samples,) 

True binary labels. If labels are not either {-1, 1} or {0, 1}, then pos_label should be explicitly given.  
y_scorendarray of shape of (n_samples,) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
pos_labelint or str, default=None 

The label of the positive class. When pos_label=None, if y_true is in {-1, 1} or {0, 1}, pos_label is set to 1, otherwise an error will be raised.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
fprndarray of shape (n_thresholds,) 

False positive rate (FPR) such that element i is the false positive rate of predictions with score >= thresholds[i]. This is occasionally referred to as false acceptance propability or fall-out.  
fnrndarray of shape (n_thresholds,) 

False negative rate (FNR) such that element i is the false negative rate of predictions with score >= thresholds[i]. This is occasionally referred to as false rejection or miss rate.  
thresholdsndarray of shape (n_thresholds,) 

Decreasing score values.      See also  
plot_det_curve


Plot detection error tradeoff (DET) curve.  
DetCurveDisplay


DET curve visualization.  
roc_curve


Compute Receiver operating characteristic (ROC) curve.  
precision_recall_curve


Compute precision-recall curve.    Examples >>> import numpy as np
>>> from sklearn.metrics import det_curve
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, fnr, thresholds = det_curve(y_true, y_scores)
>>> fpr
array([0.5, 0.5, 0. ])
>>> fnr
array([0. , 0.5, 0.5])
>>> thresholds
array([0.35, 0.4 , 0.8 ])
 
 Examples using sklearn.metrics.det_curve
 
  Detection error tradeoff (DET) curve