Skip freeze/unfreeze on empty associations