Ensuring that the VoIP packets arrive at their destination in a timely manner is also important. VoIP traffic is very sensitive to transmission delays and dropped packets.
Network congestion is probably the biggest hurdle to clear VoIP communication.
Quality of Service, or QoS, is usually handled at the router or network edge level, although some IP-PBXs can also set it. Each edge device handles QoS differently, but for the most part it comes down to assigning a specific type of network traffic as a priority. By giving VoIP traffic more priority than say HTTP Web traffic, you can guarantee that if network bandwidth becomes scarce, VoIP traffic will still move through the network where HTTP traffic may not. This means that HTTP packets will be dropped in favor of VoIP packets. Setting QoS is one thing, being able to monitor it is another.