In fact, even a civil libertarian might prefer a world where wiretaps are possible. If we want to catch and convict the leaders of criminal organizations, there are usually only two good ways to do it. Or we can wiretap the leaders as they plan the crime. I once did a human rights report on the criminal justice system in El Salvador. I didn't expect the Salvadorans to teach me much about human rights.
Wiretaps have no such potential for coercive use.
The defendant is convicted or freed on the basis of his own, unarguable words. In addition, the world will be a safer place if criminals cannot take advantage of a ubiquitous, standardized encryption infrastructure that is immune from any conceivable law enforcement wiretap. Even if you're worried about illegal government taps, key escrow reinforces the existing requirement that every wiretap and every decryption must be lawfully authorized. The key escrow system means that proof of authority to tap must be certified and audited, so that illegal wiretapping by a rogue prosecutor or police officer is, as a practical matter, impossible. Unreadable encryption is the key to our future liberty.