Fix has_many :by => :ids associations when child objects are destroyed