Handle case where user wants to ffdfappend a non-factor to a factor