Since the strength of your broadband connection is so critical to VoIP, it's one of the largest drawbacks. VoIP phone service is not the same as the Internet. It doesn't stop there, either. The quality of the call can also be affected by a poor internet connection, which can cause issues like as jitter and latency.