Pirmajer was born in Trbovlje in present-day Slovenia. At the time of his birth, Trbovlje was renamed Trifail by the occupying Nazi German forces. He moved with his family to Serbia to Srbobran in 1947. His football skills were spotted while he was in elementary school and he joined the youth-team of Elan Srbobran. When he was 16, with a special medical permission, he was allowed to start playing in the main team of Elan Srbobran. After a handful of games he attracted attention from bigger clubs, and shortly after he joined RFK Novi Sad. He debuted in the season 1960–61 in which Novi Sad made their historical promotion to the Yugoslav First League after winning the 1960–61 Yugoslav Second League East. Pirmajer played the following two and a half seasons with Novi Sad in the First League, until the winter-break of the 1963–64 season when he was brought by FK Partizan. Pirmajer moved to the capital and played with Partizan a total of four and a half seasons during which he won one championship and was European vice-champion after losing the 1966 European Cup Final. During his spell in Partizan he got the record of having played 252 consecutive competitive matches.