Especially when there are unneutered female cats in the neighbourhood, entire stray male cats may be attracted to a particular area and can be very vocal, confrontational and aggressive towards other cats. Just having these cats around could be very stressful for your cat.
Entire male cats are also more likely to carry various diseases, such as FIV (Feline Immunodeficiency Virus), which could be transmitted to your cat through fighting. The best way to deal with this problem is to talk to your local rehoming charity for advice as they may be able to provide help (or direct you to other organisations that can help) by using a humane trap to capture the cat and either neuter him and return him to the same area or relocate him to a more suitable environment.