A few years ago, the far-right Britain First claimed that 5,000 of its members had joined the Tory party.