If you are planning on moving back to England from France, you should be fine to do so if all of your family have British passports. However, following Brexit, family members with EU or other nationalities will need to apply for a family visa, before being granted residence in the UK.