Take the recorder into a really quiet place, turn up the volume and play it back. Whatever noises you hear on that tape will be going, amplified, to the other end of your videoconferences as well. Our biggest ambient noise problem here is noisy air handlers. We will try to address this by wrapping the ducts in fiberglass batting. If you are having a room remodeled you might be able to have the ducting rerouted.
Larger ducts have less turbulence and so, are less noisy.
The design of the difuser also can affect noise levels. In a videoconference you should imagine you are on a stage participating in a panel discussion. Even though your remarks are directed at the person next to you, you should speak loudly enough to be heard throughout the hall. If someone asks a question in such a quiet voice that it obviously would not be heard at the other end, the question should be succinctly, and more loudly, repeated by the person answering so that the answer will be meaningful to the participants at the other locations.