Apply activerecord inverse_of for object sync