In addition to being expensive, collecting usability metrics interferes with the goal of gathering qualitative insights to drive design decisions. As a compromise, you can measure users' ability to complete tasks.
Success rates are easy to understand and represent usability's bottom line.
Numbers are powerful. They offer a simple way to communicate usability findings to a general audience. In a previous Alertbox, I discussed ways of measuring and comparing usability metrics like time on task. If not, you are falling behind relative to both the competition and the needs of the new, less technically inclined users who are coming online. Unfortunately, there is a conflict between the need for numbers and the need for insight.