There is no scientific proof to show that taking oral contraceptives has harmful long-term effects on fertility. If you are having issues falling pregnant, it is most likely due to other causes.
It is important to note that if you stop using birth control to prepare for pregnancy, then you should monitor your menstrual cycle closely. If your cycles do not become regular in 3-4 months after stopping your birth control, you should see your gynae to investigate this further as being on the pill can mask signs of conditions such as polycystic ovary syndrome (PCOS).