neulab/docprompting-conala huggingface dataset:
class sklearn.metrics.ConfusionMatrixDisplay(confusion_matrix, *, display_labels=None) [source]
 
Confusion Matrix visualization. It is recommend to use plot_confusion_matrix to create a ConfusionMatrixDisplay. All parameters are stored as attributes. Read more in the User Guide.  Parameters 
 
confusion_matrixndarray of shape (n_classes, n_classes) 

Confusion matrix.  
display_labelsndarray of shape (n_classes,), default=None 

Display labels for plot. If None, display labels are set from 0 to n_classes - 1.    Attributes 
 
im_matplotlib AxesImage 

Image representing the confusion matrix.  
text_ndarray of shape (n_classes, n_classes), dtype=matplotlib Text, or None 

Array of matplotlib axes. None if include_values is false.  
ax_matplotlib Axes 

Axes with confusion matrix.  
figure_matplotlib Figure 

Figure containing the confusion matrix.      See also  
confusion_matrix


Compute Confusion Matrix to evaluate the accuracy of a classification.  
plot_confusion_matrix


Plot Confusion Matrix.    Examples >>> from sklearn.datasets import make_classification
>>> from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.svm import SVC
>>> X, y = make_classification(random_state=0)
>>> X_train, X_test, y_train, y_test = train_test_split(X, y,
...                                                     random_state=0)
>>> clf = SVC(random_state=0)
>>> clf.fit(X_train, y_train)
SVC(random_state=0)
>>> predictions = clf.predict(X_test)
>>> cm = confusion_matrix(y_test, predictions, labels=clf.classes_)
>>> disp = ConfusionMatrixDisplay(confusion_matrix=cm,
...                               display_labels=clf.classes_)
>>> disp.plot() 
 Methods  
plot(*[, include_values, cmap, …]) Plot visualization.    
plot(*, include_values=True, cmap='viridis', xticks_rotation='horizontal', values_format=None, ax=None, colorbar=True) [source]
 
Plot visualization.  Parameters 
 
include_valuesbool, default=True 

Includes values in confusion matrix.  
cmapstr or matplotlib Colormap, default=’viridis’ 

Colormap recognized by matplotlib.  
xticks_rotation{‘vertical’, ‘horizontal’} or float, default=’horizontal’ 

Rotation of xtick labels.  
values_formatstr, default=None 

Format specification for values in confusion matrix. If None, the format specification is ‘d’ or ‘.2g’ whichever is shorter.  
axmatplotlib axes, default=None 

Axes object to plot on. If None, a new figure and axes is created.  
colorbarbool, default=True 

Whether or not to add a colorbar to the plot.    Returns 
 
displayConfusionMatrixDisplay
sklearn.metrics.ConfusionMatrixDisplay  
class sklearn.metrics.ConfusionMatrixDisplay(confusion_matrix, *, display_labels=None) [source]
 
Confusion Matrix visualization. It is recommend to use plot_confusion_matrix to create a ConfusionMatrixDisplay. All parameters are stored as attributes. Read more in the User Guide.  Parameters 
 
confusion_matrixndarray of shape (n_classes, n_classes) 

Confusion matrix.  
display_labelsndarray of shape (n_classes,), default=None 

Display labels for plot. If None, display labels are set from 0 to n_classes - 1.    Attributes 
 
im_matplotlib AxesImage 

Image representing the confusion matrix.  
text_ndarray of shape (n_classes, n_classes), dtype=matplotlib Text, or None 

Array of matplotlib axes. None if include_values is false.  
ax_matplotlib Axes 

Axes with confusion matrix.  
figure_matplotlib Figure 

Figure containing the confusion matrix.      See also  
confusion_matrix


Compute Confusion Matrix to evaluate the accuracy of a classification.  
plot_confusion_matrix


Plot Confusion Matrix.    Examples >>> from sklearn.datasets import make_classification
>>> from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.svm import SVC
>>> X, y = make_classification(random_state=0)
>>> X_train, X_test, y_train, y_test = train_test_split(X, y,
...                                                     random_state=0)
>>> clf = SVC(random_state=0)
>>> clf.fit(X_train, y_train)
SVC(random_state=0)
>>> predictions = clf.predict(X_test)
>>> cm = confusion_matrix(y_test, predictions, labels=clf.classes_)
>>> disp = ConfusionMatrixDisplay(confusion_matrix=cm,
...                               display_labels=clf.classes_)
>>> disp.plot() 
 Methods  
plot(*[, include_values, cmap, …]) Plot visualization.    
plot(*, include_values=True, cmap='viridis', xticks_rotation='horizontal', values_format=None, ax=None, colorbar=True) [source]
 
Plot visualization.  Parameters 
 
include_valuesbool, default=True 

Includes values in confusion matrix.  
cmapstr or matplotlib Colormap, default=’viridis’ 

Colormap recognized by matplotlib.  
xticks_rotation{‘vertical’, ‘horizontal’} or float, default=’horizontal’ 

Rotation of xtick labels.  
values_formatstr, default=None 

Format specification for values in confusion matrix. If None, the format specification is ‘d’ or ‘.2g’ whichever is shorter.  
axmatplotlib axes, default=None 

Axes object to plot on. If None, a new figure and axes is created.  
colorbarbool, default=True 

Whether or not to add a colorbar to the plot.    Returns 
 
displayConfusionMatrixDisplay 
   
 
 Examples using sklearn.metrics.ConfusionMatrixDisplay
 
  Visualizations with Display Objects
plot(*, include_values=True, cmap='viridis', xticks_rotation='horizontal', values_format=None, ax=None, colorbar=True) [source]
 
Plot visualization.  Parameters 
 
include_valuesbool, default=True 

Includes values in confusion matrix.  
cmapstr or matplotlib Colormap, default=’viridis’ 

Colormap recognized by matplotlib.  
xticks_rotation{‘vertical’, ‘horizontal’} or float, default=’horizontal’ 

Rotation of xtick labels.  
values_formatstr, default=None 

Format specification for values in confusion matrix. If None, the format specification is ‘d’ or ‘.2g’ whichever is shorter.  
axmatplotlib axes, default=None 

Axes object to plot on. If None, a new figure and axes is created.  
colorbarbool, default=True 

Whether or not to add a colorbar to the plot.    Returns 
 
displayConfusionMatrixDisplay
sklearn.metrics.confusion_matrix(y_true, y_pred, *, labels=None, sample_weight=None, normalize=None) [source]
 
Compute confusion matrix to evaluate the accuracy of a classification. By definition a confusion matrix \(C\) is such that \(C_{i, j}\) is equal to the number of observations known to be in group \(i\) and predicted to be in group \(j\). Thus in binary classification, the count of true negatives is \(C_{0,0}\), false negatives is \(C_{1,0}\), true positives is \(C_{1,1}\) and false positives is \(C_{0,1}\). Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) 

Estimated targets as returned by a classifier.  
labelsarray-like of shape (n_classes), default=None 

List of labels to index the matrix. This may be used to reorder or select a subset of labels. If None is given, those that appear at least once in y_true or y_pred are used in sorted order.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  New in version 0.18.   
normalize{‘true’, ‘pred’, ‘all’}, default=None 

Normalizes confusion matrix over the true (rows), predicted (columns) conditions or all the population. If None, confusion matrix will not be normalized.    Returns 
 
Cndarray of shape (n_classes, n_classes) 

Confusion matrix whose i-th row and j-th column entry indicates the number of samples with true label being i-th class and predicted label being j-th class.      See also  
plot_confusion_matrix


Plot Confusion Matrix.  
ConfusionMatrixDisplay


Confusion Matrix visualization.    References  
1  
Wikipedia entry for the Confusion matrix (Wikipedia and other references may use a different convention for axes).   Examples >>> from sklearn.metrics import confusion_matrix
>>> y_true = [2, 0, 2, 2, 0, 1]
>>> y_pred = [0, 0, 2, 2, 0, 2]
>>> confusion_matrix(y_true, y_pred)
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])
 >>> y_true = ["cat", "ant", "cat", "cat", "ant", "bird"]
>>> y_pred = ["ant", "ant", "cat", "cat", "ant", "cat"]
>>> confusion_matrix(y_true, y_pred, labels=["ant", "bird", "cat"])
array([[2, 0, 0],
       [0, 0, 1],
       [1, 0, 2]])
 In the binary case, we can extract true positives, etc as follows: >>> tn, fp, fn, tp = confusion_matrix([0, 1, 0, 1], [1, 1, 1, 0]).ravel()
>>> (tn, fp, fn, tp)
(0, 2, 1, 1)
sklearn.metrics.consensus_score(a, b, *, similarity='jaccard') [source]
 
The similarity of two sets of biclusters. Similarity between individual biclusters is computed. Then the best matching between sets is found using the Hungarian algorithm. The final score is the sum of similarities divided by the size of the larger set. Read more in the User Guide.  Parameters 
 
a(rows, columns) 

Tuple of row and column indicators for a set of biclusters.  
b(rows, columns) 

Another set of biclusters like a.  
similarity‘jaccard’ or callable, default=’jaccard’ 

May be the string “jaccard” to use the Jaccard coefficient, or any function that takes four arguments, each of which is a 1d indicator vector: (a_rows, a_columns, b_rows, b_columns).     References  Hochreiter, Bodenhofer, et. al., 2010. FABIA: factor analysis for bicluster acquisition.
sklearn.metrics.coverage_error(y_true, y_score, *, sample_weight=None) [source]
 
Coverage error measure. Compute how far we need to go through the ranked scores to cover all true labels. The best value is equal to the average number of labels in y_true per sample. Ties in y_scores are broken by giving maximal rank that would have been assigned to all tied values. Note: Our implementation’s score is 1 greater than the one given in Tsoumakas et al., 2010. This extends it to handle the degenerate case in which an instance has 0 true labels. Read more in the User Guide.  Parameters 
 
y_truendarray of shape (n_samples, n_labels) 

True binary labels in binary indicator format.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
coverage_errorfloat 
   References  
1  
Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US.
sklearn.metrics.davies_bouldin_score(X, labels) [source]
 
Computes the Davies-Bouldin score. The score is defined as the average similarity measure of each cluster with its most similar cluster, where similarity is the ratio of within-cluster distances to between-cluster distances. Thus, clusters which are farther apart and less dispersed will result in a better score. The minimum score is zero, with lower values indicating better clustering. Read more in the User Guide.  New in version 0.20.   Parameters 
 
Xarray-like of shape (n_samples, n_features) 

A list of n_features-dimensional data points. Each row corresponds to a single data point.  
labelsarray-like of shape (n_samples,) 

Predicted labels for each sample.    Returns 
 score: float

The resulting Davies-Bouldin score.     References  
1  
Davies, David L.; Bouldin, Donald W. (1979). “A Cluster Separation Measure”. IEEE Transactions on Pattern Analysis and Machine Intelligence. PAMI-1 (2): 224-227
sklearn.metrics.dcg_score(y_true, y_score, *, k=None, log_base=2, sample_weight=None, ignore_ties=False) [source]
 
Compute Discounted Cumulative Gain. Sum the true scores ranked in the order induced by the predicted scores, after applying a logarithmic discount. This ranking metric yields a high value if true labels are ranked high by y_score. Usually the Normalized Discounted Cumulative Gain (NDCG, computed by ndcg_score) is preferred.  Parameters 
 
y_truendarray of shape (n_samples, n_labels) 

True targets of multilabel classification, or true scores of entities to be ranked.  
y_scorendarray of shape (n_samples, n_labels) 

Target scores, can either be probability estimates, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
kint, default=None 

Only consider the highest k scores in the ranking. If None, use all outputs.  
log_basefloat, default=2 

Base of the logarithm used for the discount. A low value means a sharper discount (top results are more important).  
sample_weightndarray of shape (n_samples,), default=None 

Sample weights. If None, all samples are given the same weight.  
ignore_tiesbool, default=False 

Assume that there are no ties in y_score (which is likely to be the case if y_score is continuous) for efficiency gains.    Returns 
 
discounted_cumulative_gainfloat 

The averaged sample DCG scores.      See also  
ndcg_score


The Discounted Cumulative Gain divided by the Ideal Discounted Cumulative Gain (the DCG obtained for a perfect ranking), in order to have a score between 0 and 1.    References Wikipedia entry for Discounted Cumulative Gain. Jarvelin, K., & Kekalainen, J. (2002). Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4), 422-446. Wang, Y., Wang, L., Li, Y., He, D., Chen, W., & Liu, T. Y. (2013, May). A theoretical analysis of NDCG ranking measures. In Proceedings of the 26th Annual Conference on Learning Theory (COLT 2013). McSherry, F., & Najork, M. (2008, March). Computing information retrieval performance measures efficiently in the presence of tied scores. In European conference on information retrieval (pp. 414-421). Springer, Berlin, Heidelberg. Examples >>> from sklearn.metrics import dcg_score
>>> # we have groud-truth relevance of some answers to a query:
>>> true_relevance = np.asarray([[10, 0, 0, 1, 5]])
>>> # we predict scores for the answers
>>> scores = np.asarray([[.1, .2, .3, 4, 70]])
>>> dcg_score(true_relevance, scores)
9.49...
>>> # we can set k to truncate the sum; only top k answers contribute
>>> dcg_score(true_relevance, scores, k=2)
5.63...
>>> # now we have some ties in our prediction
>>> scores = np.asarray([[1, 0, 0, 0, 1]])
>>> # by default ties are averaged, so here we get the average true
>>> # relevance of our top predictions: (10 + 5) / 2 = 7.5
>>> dcg_score(true_relevance, scores, k=1)
7.5
>>> # we can choose to ignore ties for faster results, but only
>>> # if we know there aren't ties in our scores, otherwise we get
>>> # wrong results:
>>> dcg_score(true_relevance,
...           scores, k=1, ignore_ties=True)
5.0
class sklearn.metrics.DetCurveDisplay(*, fpr, fnr, estimator_name=None, pos_label=None) [source]
 
DET curve visualization. It is recommend to use plot_det_curve to create a visualizer. All parameters are stored as attributes. Read more in the User Guide.  New in version 0.24.   Parameters 
 
fprndarray 

False positive rate.  
tprndarray 

True positive rate.  
estimator_namestr, default=None 

Name of estimator. If None, the estimator name is not shown.  
pos_labelstr or int, default=None 

The label of the positive class.    Attributes 
 
line_matplotlib Artist 

DET Curve.  
ax_matplotlib Axes 

Axes with DET Curve.  
figure_matplotlib Figure 

Figure containing the curve.      See also  
det_curve


Compute error rates for different probability thresholds.  
plot_det_curve


Plot detection error tradeoff (DET) curve.    Examples >>> import matplotlib.pyplot as plt  
>>> import numpy as np
>>> from sklearn import metrics
>>> y = np.array([0, 0, 1, 1])
>>> pred = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, fnr, thresholds = metrics.det_curve(y, pred)
>>> display = metrics.DetCurveDisplay(
...     fpr=fpr, fnr=fnr, estimator_name='example estimator'
... )
>>> display.plot()  
>>> plt.show()      
 Methods  
plot([ax, name]) Plot visualization.    
plot(ax=None, *, name=None, **kwargs) [source]
 
Plot visualization.  Parameters 
 
axmatplotlib axes, default=None 

Axes object to plot on. If None, a new figure and axes is created.  
namestr, default=None 

Name of DET curve for labeling. If None, use the name of the estimator.    Returns 
 
displayDetCurveDisplay 

Object that stores computed values.
sklearn.metrics.DetCurveDisplay  
class sklearn.metrics.DetCurveDisplay(*, fpr, fnr, estimator_name=None, pos_label=None) [source]
 
DET curve visualization. It is recommend to use plot_det_curve to create a visualizer. All parameters are stored as attributes. Read more in the User Guide.  New in version 0.24.   Parameters 
 
fprndarray 

False positive rate.  
tprndarray 

True positive rate.  
estimator_namestr, default=None 

Name of estimator. If None, the estimator name is not shown.  
pos_labelstr or int, default=None 

The label of the positive class.    Attributes 
 
line_matplotlib Artist 

DET Curve.  
ax_matplotlib Axes 

Axes with DET Curve.  
figure_matplotlib Figure 

Figure containing the curve.      See also  
det_curve


Compute error rates for different probability thresholds.  
plot_det_curve


Plot detection error tradeoff (DET) curve.    Examples >>> import matplotlib.pyplot as plt  
>>> import numpy as np
>>> from sklearn import metrics
>>> y = np.array([0, 0, 1, 1])
>>> pred = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, fnr, thresholds = metrics.det_curve(y, pred)
>>> display = metrics.DetCurveDisplay(
...     fpr=fpr, fnr=fnr, estimator_name='example estimator'
... )
>>> display.plot()  
>>> plt.show()      
 Methods  
plot([ax, name]) Plot visualization.    
plot(ax=None, *, name=None, **kwargs) [source]
 
Plot visualization.  Parameters 
 
axmatplotlib axes, default=None 

Axes object to plot on. If None, a new figure and axes is created.  
namestr, default=None 

Name of DET curve for labeling. If None, use the name of the estimator.    Returns 
 
displayDetCurveDisplay 

Object that stores computed values.
plot(ax=None, *, name=None, **kwargs) [source]
 
Plot visualization.  Parameters 
 
axmatplotlib axes, default=None 

Axes object to plot on. If None, a new figure and axes is created.  
namestr, default=None 

Name of DET curve for labeling. If None, use the name of the estimator.    Returns 
 
displayDetCurveDisplay 

Object that stores computed values.
sklearn.metrics.det_curve(y_true, y_score, pos_label=None, sample_weight=None) [source]
 
Compute error rates for different probability thresholds.  Note This metric is used for evaluation of ranking and error tradeoffs of a binary classification task.  Read more in the User Guide.  New in version 0.24.   Parameters 
 
y_truendarray of shape (n_samples,) 

True binary labels. If labels are not either {-1, 1} or {0, 1}, then pos_label should be explicitly given.  
y_scorendarray of shape of (n_samples,) 

Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by “decision_function” on some classifiers).  
pos_labelint or str, default=None 

The label of the positive class. When pos_label=None, if y_true is in {-1, 1} or {0, 1}, pos_label is set to 1, otherwise an error will be raised.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.    Returns 
 
fprndarray of shape (n_thresholds,) 

False positive rate (FPR) such that element i is the false positive rate of predictions with score >= thresholds[i]. This is occasionally referred to as false acceptance propability or fall-out.  
fnrndarray of shape (n_thresholds,) 

False negative rate (FNR) such that element i is the false negative rate of predictions with score >= thresholds[i]. This is occasionally referred to as false rejection or miss rate.  
thresholdsndarray of shape (n_thresholds,) 

Decreasing score values.      See also  
plot_det_curve


Plot detection error tradeoff (DET) curve.  
DetCurveDisplay


DET curve visualization.  
roc_curve


Compute Receiver operating characteristic (ROC) curve.  
precision_recall_curve


Compute precision-recall curve.    Examples >>> import numpy as np
>>> from sklearn.metrics import det_curve
>>> y_true = np.array([0, 0, 1, 1])
>>> y_scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, fnr, thresholds = det_curve(y_true, y_scores)
>>> fpr
array([0.5, 0.5, 0. ])
>>> fnr
array([0. , 0.5, 0.5])
>>> thresholds
array([0.35, 0.4 , 0.8 ])
sklearn.metrics.explained_variance_score(y_true, y_pred, *, sample_weight=None, multioutput='uniform_average') [source]
 
Explained variance regression score function. Best possible score is 1.0, lower values are worse. Read more in the User Guide.  Parameters 
 
y_truearray-like of shape (n_samples,) or (n_samples, n_outputs) 

Ground truth (correct) target values.  
y_predarray-like of shape (n_samples,) or (n_samples, n_outputs) 

Estimated target values.  
sample_weightarray-like of shape (n_samples,), default=None 

Sample weights.  
multioutput{‘raw_values’, ‘uniform_average’, ‘variance_weighted’} or array-like of shape (n_outputs,), default=’uniform_average’ 

Defines aggregating of multiple output scores. Array-like value defines weights used to average scores.  ‘raw_values’ :

Returns a full set of scores in case of multioutput input.  ‘uniform_average’ :

Scores of all outputs are averaged with uniform weight.  ‘variance_weighted’ :

Scores of all outputs are averaged, weighted by the variances of each individual output.      Returns 
 
scorefloat or ndarray of floats 

The explained variance or ndarray if ‘multioutput’ is ‘raw_values’.     Notes This is not a symmetric function. Examples >>> from sklearn.metrics import explained_variance_score
>>> y_true = [3, -0.5, 2, 7]
>>> y_pred = [2.5, 0.0, 2, 8]
>>> explained_variance_score(y_true, y_pred)
0.957...
>>> y_true = [[0.5, 1], [-1, 1], [7, -6]]
>>> y_pred = [[0, 2], [-1, 2], [8, -5]]
>>> explained_variance_score(y_true, y_pred, multioutput='uniform_average')
0.983...
sklearn.metrics.fowlkes_mallows_score(labels_true, labels_pred, *, sparse=False) [source]
 
Measure the similarity of two clusterings of a set of points.  New in version 0.18.  The Fowlkes-Mallows index (FMI) is defined as the geometric mean between of the precision and recall: FMI = TP / sqrt((TP + FP) * (TP + FN))
 Where TP is the number of True Positive (i.e. the number of pair of points that belongs in the same clusters in both labels_true and labels_pred), FP is the number of False Positive (i.e. the number of pair of points that belongs in the same clusters in labels_true and not in labels_pred) and FN is the number of False Negative (i.e the number of pair of points that belongs in the same clusters in labels_pred and not in labels_True). The score ranges from 0 to 1. A high value indicates a good similarity between two clusters. Read more in the User Guide.  Parameters 
 
labels_trueint array, shape = (n_samples,) 

A clustering of the data into disjoint subsets.  
labels_predarray, shape = (n_samples, ) 

A clustering of the data into disjoint subsets.  
sparsebool, default=False 

Compute contingency matrix internally with sparse matrix.    Returns 
 
scorefloat 

The resulting Fowlkes-Mallows score.     References  
1  
E. B. Fowkles and C. L. Mallows, 1983. “A method for comparing two hierarchical clusterings”. Journal of the American Statistical Association  
2  
Wikipedia entry for the Fowlkes-Mallows Index   Examples Perfect labelings are both homogeneous and complete, hence have score 1.0: >>> from sklearn.metrics.cluster import fowlkes_mallows_score
>>> fowlkes_mallows_score([0, 0, 1, 1], [0, 0, 1, 1])
1.0
>>> fowlkes_mallows_score([0, 0, 1, 1], [1, 1, 0, 0])
1.0
 If classes members are completely split across different clusters, the assignment is totally random, hence the FMI is null: >>> fowlkes_mallows_score([0, 0, 0, 0], [0, 1, 2, 3])
0.0
sklearn.metrics.get_scorer(scoring) [source]
 
Get a scorer from string. Read more in the User Guide.  Parameters 
 
scoringstr or callable 

Scoring method as string. If callable it is returned as is.    Returns 
 
scorercallable 

The scorer.