Repository: mlpack/mlpack

Issue #496: hmm_train failed to converge
Created: 2015-12-15T16:21:50Z
Hi,
I have trouble running on some of my files. I get this error message:

INFO ] EMFit::Estimate(): iteration 251, log-likelihood -163.419.
[INFO ] EMFit::Estimate(): iteration 252, log-likelihood -163.419.
[INFO ] GMM::Estimate(): log-likelihood of trained GMM is -163.419.
[INFO ] Cluster 6 is empty.
[DEBUG] Point 0 assigned to empty cluster 6.
[INFO ] Cluster 7 is empty.
[DEBUG] Point 1 assigned to empty cluster 7.
[INFO ] KMeans::Cluster(): iteration 1, residual inf.
[INFO ] Cluster 7 is empty.
[DEBUG] Point 32 assigned to empty cluster 7.
[INFO ] KMeans::Cluster(): iteration 2, residual inf.
[INFO ] KMeans::Cluster(): iteration 3, residual 0.269213.
[INFO ] KMeans::Cluster(): iteration 4, residual 0.
[INFO ] KMeans::Cluster(): converged after 4 iterations.
[INFO ] 2240 distance calculations.
[DEBUG] Covariance matrix is not positive definite.  Adding perturbation.
[DEBUG] Covariance matrix is not positive definite.  Adding perturbation.
error: chol(): failed to converge
terminate called after throwing an instance of 'std::runtime_error'
what():  chol(): failed to converge
Program received signal SIGABRT, Aborted.
0x00007ffff64e4cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff64e4cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff64e80d8 in __GI_abort () at abort.c:89
#2  0x00007ffff6def535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff6ded6d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff6ded703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff6ded922 in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff799e144 in arma::arma_bad<char [27]> (x=..., hurl=true) at /usr/include/armadillo_bits/debug.hpp:177
#7  0x00007ffff79a9387 in arma::op_chol::applyarma::Mat (out=..., X=...) at /usr/include/armadillo_bits/op_chol_meat.hpp:26
#8  0x00007ffff79a6809 in arma::Mat::Matarma::Mat<double, arma::op_chol> (this=0x7fffffffb300, X=...) at /usr/include/armadillo_bits/Mat_meat.hpp:3932
#9  0x00007ffff79a59ad in arma::Proxyarma::Op<arma::Mat<double, arma::op_chol> >::Proxy (this=0x7fffffffb300, A=...) at /usr/include/armadillo_bits/Proxy.hpp:309
#10 0x00007ffff79a2ba3 in arma::op_strans::apply_proxyarma::Op<arma::Mat<double, arma::op_chol> > (out=..., X=...) at /usr/include/armadillo_bits/op_strans_meat.hpp:220
#11 0x00007ffff799f88e in arma::op_htrans::applyarma::Op<arma::Mat<double, arma::op_chol> > (out=..., in=..., junk=0x0) at /usr/include/armadillo_bits/op_htrans_meat.hpp:265
#12 0x00007ffff799e9e2 in arma::Mat::operator=arma::Op<arma::Mat<double, arma::op_chol>, arma::op_htrans> (this=0x8fbc00, X=...) at /usr/include/armadillo_bits/Mat_meat.hpp:3948
#13 0x00007ffff799b7ab in mlpack::distribution::GaussianDistribution::FactorCovariance (this=0x8fbac0) at /org/share/home/adigueze/mlpack-master/src/mlpack/core/dists/gaussian_distribution.cpp:38
#14 0x00007ffff799b732 in mlpack::distribution::GaussianDistribution::Covariance(arma::Mat&&) (this=0x8fbac0, covariance=<unknown type in /org/share/home/adigueze/mlpack-master/bin/lib/libmlpack.so.1, CU 0x21ab0, DIE 0x52374>)
at /org/share/home/adigueze/mlpack-master/src/mlpack/core/dists/gaussian_distribution.cpp:30

#15 0x00000000005fe597 in mlpack::gmm::EMFit<mlpack::kmeans::KMeans<mlpack::metric::LMetric<2, true>, mlpack::kmeans::RandomPartition, mlpack::kmeans::MaxVarianceNewCluster, mlpack::kmeans::NaiveKMeans, arma::Mat >, mlpack::gmm::PositiveDefiniteConstraint>::InitialClustering (this=0x8fc280, observations=..., dists=..., weights=...) at /org/share/home/adigueze/mlpack-master/src/mlpack/../mlpack/methods/gmm/em_fit_impl.hpp:264
#16 0x00000000005f4472 in mlpack::gmm::EMFit<mlpack::kmeans::KMeans<mlpack::metric::LMetric<2, true>, mlpack::kmeans::RandomPartition, mlpack::kmeans::MaxVarianceNewCluster, mlpack::kmeans::NaiveKMeans, arma::Mat >, mlpack::gmm::PositiveDefiniteConstraint>::Estimate (this=0x8fc280, observations=..., dists=..., weights=..., useInitialModel=false) at /org/share/home/adigueze/mlpack-master/src/mlpack/../mlpack/methods/gmm/em_fit_impl.hpp:39
#17 0x00000000005e9027 in mlpack::gmm::GMM<mlpack::gmm::EMFit<mlpack::kmeans::KMeans<mlpack::metric::LMetric<2, true>, mlpack::kmeans::RandomPartition, mlpack::kmeans::MaxVarianceNewCluster, mlpack::kmeans::NaiveKMeans, arma::Mat >, mlpack::gmm::PositiveDefiniteConstraint> >::Estimate (this=0x8f68a0, observations=..., trials=1, useExistingModel=false) at /org/share/home/adigueze/mlpack-master/src/mlpack/../mlpack/methods/gmm/gmm_impl.hpp:190
#18 0x00000000005dd382 in mlpack::hmm::HMM<mlpack::gmm::GMM<mlpack::gmm::EMFit<mlpack::kmeans::KMeans<mlpack::metric::LMetric<2, true>, mlpack::kmeans::RandomPartition, mlpack::kmeans::MaxVarianceNewCluster, mlpack::kmeans::NaiveKMeans, arma::Mat >, mlpack::gmm::PositiveDefiniteConstraint> > >::Train (this=0x7fffffffd770, dataSeq=..., stateSeq=...) at /org/share/home/adigueze/mlpack-master/src/mlpack/methods/hmm/hmm_impl.hpp:276
#19 0x00000000005d1543 in Train::Apply<mlpack::hmm::HMM<mlpack::gmm::GMM<mlpack::gmm::EMFit<mlpack::kmeans::KMeans<mlpack::metric::LMetric<2, true>, mlpack::kmeans::RandomPartition, mlpack::kmeans::MaxVarianceNewCluster, mlpack::kmeans::NaiveKMeans, arma::Mat >, mlpack::gmm::PositiveDefiniteConstraint> > > > (hmm=..., trainSeqPtr=0x7fffffffd400) at /org/share/home/adigueze/mlpack-master/src/mlpack/methods/hmm/hmm_train_main.cpp:170
#20 0x00000000005c3415 in main (argc=14, argv=0x7fffffffe128) at /org/share/home/adigueze/mlpack-master/src/mlpack/methods/hmm/hmm_train_main.cpp:344

Here is the input file and the labels I use:
obsShoulderLeft.csv.txt
labels.csv.txt
I run hmm_train like this:
hmm_train  -v -i labels.csv -o runNormalizedTest.xml -t gmm -g 10 -n 13
I guess, the error means that the EM algorithm does not converge. I normalized my data, but this didn't change anything. Is there anything else I can do? Or is there anything wrong with my data? On other files everything works fine. Any help is very welcome.
Greetings
davud

Duplicate issue #538: HMM Training  fails to converge
Created: 2016-03-02T10:13:49Z
Hi,
I'm trying to use mlpack to train HMM for word models, but most of my HMMs failed to converge with the given error:

error: chol(): failed to converge

Here is the code (unfortunately, not standalone, it's inside another program):
    using GMM = mlpack::gmm::GMM;

    template<typename Distribution>
    using HMM = mlpack::hmm::HMM<Distribution>;

    using hmm_p = std::unique_ptr<HMM<GMM>>;

    constexpr const std::size_t n_gaussians = 1;

    const auto n_states_per_char = 1;
    const auto n_states = characters * n_states_per_char;
    const auto n_features = 9;

    auto hmm = std::make_unique<HMM<GMM>>(n_states, GMM(n_gaussians, n_features));

    std::vector<arma::mat> images(ref_a.size());
    std::vector<arma::Row<size_t>> labels(ref_a.size());

    //Fill images and labels...

     //Each image is initialized like this: 
     images[image] = arma::mat(n_features, width);
     labels[image] = arma::Row<size_t>(width);

    hmm->Train(images, labels);
I can give you the complete code, but it won't execute on its own.
Looking at the features and labels, it seems OK, but I'm not a specialist in HMM at all.
Is there some parameters I can play with to avoid this ? Is there anyway I can have some more verbose output that could help me investigate the issue ?

Closing comment (2016-03-03T02:14:55Z): What happens if you vary n_gaussians?  Does it fail when set to 1?
Also, is there any more output?  If you compile with debugging symbols that may help.
This may be related to #496, but that issue should have been solved with the mlpack 2.0.1 release.