No, you can still conceive naturally, but it might be more difficult.
A low sperm count is known as oligozoospermia, which is where a man has less than 15 million sperm per millilitre of semen.
To put this into perspective, an average man will have around 15 - 200 million sperm per millilitre of semen.