Keep playing with coredata to fix the bug. Turns out that getting rid of the inverse relationship fixes the problem