The 5 June 1975 referendum on British EEC membership settled the question of Britain's position in Europe, and from that point onwards, the Labour party demanded its share of MEP nominations. O'Hagan thus lost his seat on 3 July 1975 after a joint decision by the Conservative and Labour parties to cease appointing Independents and Liberals to the European Parliament. He then joined the Conservative party and became a whip and a frontbench spokesman for the Conservatives in the House of Lords between 1977 and 1979. He was also involved in the Primrose League before its dissolution, serving as its Chancellor from April 1979 to April 1981.