The dog does not credit the fence for its frustration. It associates its feeling to the people and dogs who pass by and stimulate it. You might have inadvertently added to this by cutting holes in the fence. Excessive visual access may be an over-stimulus that could increase the barking. Some nuisance barkers have actually been accidentally rewarded for the excessive barking.
Owners also reward the dog by giving attention or yelling or barking back at the dog.
This creates a dialogue that pleases the dog. First, notice and reward quiet behavior. We need to avoid falling into the practice of only noticing our dogs when they misbehave. We need to notice and acknowledge good, quiet behavior. Throughout the day, notice when your dog is quiet.