Issue 1:
Hello, everyone!

I am unable to replicate Weka's behavior on scikit learn using the Gaussian Naive Bayes classifier.

I am using StratifiedKFold (which was double-checked).

Nevertheless, Weka's and sklearn's results are inconsistent.

The procedure (in Weka) is: load CSV file, select NaiveBayes classifier, select 2-fold cross validation, run. In Python, I am running the source code that I pasted inline.

Since this problem also happens with SMO classifiers, I am guessing that there is something different in a pre-processing step, which Weka might be doing implicitly.

On Weka, I get this confusion matrix:

a b c <-- classified as

3 0 0 | a = A

3 0 0 | b = B

0 0 3 | c = C

On Sklearn, I get:

[[3 1 1]

[0 2 0]

[0 0 2]]

Thanks for everything!

My data points are (in a file named fiction.csv):

P0,P1,P2,LABEL

0.1,0.1,0.5,A

0.2,0.3,0.7,A

1.1,1.1,0.0,B

1.1,1.0,0.2,B

0.8,1.3,0.1,B

0.0,0.0,0.0,A

5,7,5,C

9,7,9,C

10,10,10,C

The source code is:

import csv
import numpy as np

from sklearn.preprocessing import StandardScaler, normalize, MinMaxScaler
from sklearn.cross_validation import StratifiedKFold
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import f1_score, confusion_matrix, classification_report

def elements(x):
p = {}
for k in x:
if k not in p:
p[k] = 1
else:
p[k] += 1
return p.keys(), p

def open_dataset(dataset_filename):

with open(dataset_filename, 'rb') as f:
data = list(csv.reader(f))

data = np.array(data)
x_ = data[1:, 0:-1].astype(float)
y_ = data[1:, -1].astype(str)
attr_names = data[0, :].astype(str)

labels, nel = elements(y_)
labels.sort()

folds = StratifiedKFold(y_, n_folds=2)
train_index, test_index = next(iter(folds))

X_train = x_[train_index,:]
y_train = y_[train_index]
X_test = x_[test_index,:]
y_test = y_[test_index]

return X_train, y_train, X_test, y_test, labels, nel

X_train, y_train, X_test, y_test, labels, nel = open_dataset("fiction.csv")

machine = GaussianNB()
machine.fit(X_train, y_train)
p1 = machine.predict(X_test)
machine.fit(X_test, y_test)
p2 = machine.predict(X_train)
f = f1_score(list(p1) + list(p2), list(y_test) + list(y_train), average='weighted')
print f
a = confusion_matrix(list(p1) + list(p2), list(y_test) + list(y_train))
print a

Issue 2:
I dig into #3945 and #2640. PR #3945 claims

Due to round off errors, the formula used to compute the weighted covariance matrix may lead to matrices with large negative eigenvalues.

Actually, the reason why the sample code given by @AlexisMignon in #2640 raises an exception is that the simplified formula C = (\sum_i w_i x_i x_i') - \mu \mu' does not apply when \mu is not the mean of X.

The gmm created in this line

gmm = mixt.GMM(2, params="wc", covariance_type="full", min_covar=1e-3)

does not estimate mean in M-step. The gmm.means_ is initialized by k-means, and does not change in EM. When the code in m_step before #3945 merged is executed, in one of component the mean is initialized to [[ 3.42118979 3.53824541]] by kmeans. However, the real mean is [0.02670257 0.02945094] before the exception is raised. Using C = (\sum_i w_i x_i x_i') - \mu \mu', of course, the estimated covariance is not positive definite. If

gmm = mixt.GMM(2, params="wmc", covariance_type="full", min_covar=1e-3)

We don't have the exception at all.

In #2640, without the snippet resulting the error, I also cannot figure out why the covariance is not positive definite. Anyway, I think the method log_multivariate_normal_density in gmm.py could be called by methods in other modules, so it is better to check the covariance first before Cholesky decomposition.

Besides, when using parameter params='wc' to initialize a gmm, we always have non-positive-definite covariance problems, with full and tied type covariance. You could try it with any random generated data. I think it is strange to build GMM without estimating means. It would be better to deprecate params initialization other than wmc.

Finally, the original code using equation C = (\sum_i w_i x_i x_i') - \mu \mu' before #3945 merged, I think, it is OK as long as we could guarantee \mu is the mean of X. But I found the code in Matlab use the way as #3945 does. I cannot see other reasons to use that way except preventing round-off errors, which I really doubt.

Ping @amueller