The biggest drawbacks of SSL VPNs are the limited native application support beyond web apps, lack of end-to-end encryption, and reliance on web browsers as clients, which have potential vulnerabilities.