text: The foster parents say "no discussions" took place between them and the council prior to the children's removal.
  The children - who are European migrants - were removed by social workers who accused the unnamed couple of belonging to a "racist party".
  Council leader Roger Stone said it was launching an immediate investigation.
  The BBC understands that the three children are all under 10 and one is a baby.
  The education secretary said the "wrong decision" was made "in the wrong way for the wrong reasons".
  By Alan SoadyPolitical correspondent, BBC News
  There's been an outbreak of agreement from the main parties on the principle that being a member of a political organisation such as UKIP should not prevent someone from being considered as a foster parent. Michael Gove called it "indefensible". Ed Miliband said it should "not be a bar", distancing himself from the decision made by Labour-run Rotherham Borough Council.
  The investigation announced by the government and the council inquiry will seek to establish the circumstances of this particular case.
  But, politically, it will be interesting to see whether it has any bearing on political campaigning in the run up to Thursday's Parliamentary by-election in Rotherham.
  He added that the Department for Education, under his leadership, would "ask the necessary questions" to determine what happened in this case.
  Officials from the department are understood to be trying to make contact with children's services counterparts at Rotherham Metropolitan Borough Council.
  Mr Gove told the BBC: "It is entirely wrong for this couple to have been treated in this way. That's why I believe we need a full explanation from the local authority as to why this decision was allowed to be taken.
  "If we say you cannot foster children because you're a member of a mainstream political party or because you have views on multiculturalism then that's utterly wrong.
  "This decision is arbitrary, ideological and indefensible."
  The education secretary added that the government was bringing forward new laws in the Children and Families' Bill to reverse the current position, which takes into account ethnic and cultural factors when placing children for adoption.
  "We want to make it clear that you should not allow ideological matters to preclude what's in the best interest of the child."
  Rotherham councillor Paul Lakin, cabinet member for children, young people and families Services has ordered an immediate investigation.
  He said: "Membership of a political party should not stop someone fostering children. The council takes its safeguarding responsibilities very seriously and always puts the needs of the children at the centre of that care."
  Mr Lakin added he wanted the report on his desk by Monday morning.
  Meanwhile, the Equality and Human Rights Commission (EHRC) said that the decision may have breached the parents' right to freedom of political opinion. It welcomed the council's decision to investigate what had happened.
  The council's strategic director of children and young people's services, Joyce Thacker, earlier defended the decision, saying it was influenced by UKIP's immigration policy.
  She said: "I have legal advice I have to follow for the placement of children and I was criticised before for not making sure their cultural and ethnic needs were met.
  "If the party mantra is, for example, ending the active promotion of multiculturalism I have to think about that... I have to think of their longer-term needs.
  Ms Thacker added she did not think UKIP was a racist party, but said it had "very clear" immigration policies which she could not ignore.
  SOURCE: UKIP website
  The unnamed couple in their late 50s, who have been approved foster parents for seven years, were eight weeks into the placement when they were approached by social workers about their membership of the party.
  The BBC's Robin Brant reports that UKIP sources say the couple have been members of the party for "over three years" and that Rotherham Metropolitan Borough Council was "tipped off" about their membership.
  And the couple first knew there were problems around their continued fostering on 14 November.
  The wife told the BBC she believed the decision had been taken "prior to the social worker visiting us in person to tell us that the children were being removed because we were UKIP members".
  She insisted they felt they were meeting the cultural needs of the three children.
  "We were actively encouraging these children to speak their own language... we enjoyed singing one of their folk songs in their native language.
  "Having been told of the religious denomination of these children - we took steps to ensure that a school of their denomination was found.
  "The children have now been placed with families who are white British, therefore how are these people going to meet the cultural needs of the children?" she added.
  UKIP's immigration policy states the party wants an "end [to] the active promotion of the doctrine of multiculturalism by local and national government", and urges Britain to leave the European Union (EU).
  The party's leader, Nigel Farage, said he was "very upset and very angry" about the decision and called for "heads to roll" at the council.
  He added: "This couple... have been fostering for many years and are very decent people. This was an awful shock to them, not to mention the upset for the children themselves."
  He also accused the Labour-controlled council of bigotry towards his party.
  Earlier, Labour leader Ed Miliband called for the council to urgently investigate the circumstances of the case.
  He insisted: "Being a member of UKIP should not be a bar to adopting or fostering children.
  "We need to find out the facts and the council urgently needs to get to the bottom of exactly what happened.
  "There are thousands of children who need to be looked after, who need fostering, we shouldn't have the situation where membership of a party like UKIP excludes you from doing that."
  Robert Tapsfield, chief executive of the Fostering Network, said the decision seemed "very strange" considering the council viewed the couple as providing a good standard of care.
  He said the placement of foster children should not be influenced by their political affiliations.
target: Michael Gove has branded a decision by Rotherham council to remove three children from a foster couple because they belong to UKIP as "indefensible".