The Gilbert and Sullivan Operetta which gave us those words was written in 1885, and we are still today struggling with the prickly problem of fitting the punishment to the crime. And now, as then, locking people up is a very expensive enterprise.
Most people have an easy answer to that question.
Conservatives say build new jails and liberals say fund new anti-poverty programs. Both are wrong. The correct approach is to find practical, workable alternatives to incarceration for nonviolent offenders. West Virginia has spent more than 320 million dollars in the last dozen years building new prison beds. Besides all that tax money, it costs in addition, and on average, about 20 thousand dollars per year to keep one person incarcerated.