@blackjid I ended up using an event to know when to destroy the carousel. 
Checking the source code for owl, I noticed that ng-repeat kicks before owl has time to reinitializate. So, when owl runs the remove it removes other elements that are not part of the owl carousel, thus making the owlCarousel() to fail.