I know its a minor thing, but variable names should correspond to what they stand for. This is *not* verbosing but storing residuals/errors during the algorithm, which in fact costs computation. So verbose is clearly the wrong word for this. Dont touch other classes, just this one. Again, this is not your fault, its just a nice clean-up on the fly (the way to go (tm) in maintaining open-source :) )

The information that is printed is clearly not debug information, but algorithm iteration information, therefore it should be SG_INFO. You can even replace "add_debug_info" by something like "On iteration %d: error: %f, primal: %f", etc.
And there should be *no* if statement at all for the SG_INFO or SG_DEBUG. Moreover, never use SG_PRINT from within algorithms, that is the whole point of these different log-levels.

Summary:
 * SG_DEBUG: Things liker "entering", "leaving" (see QuadraticTimeMMD for examples). Or "doing this and that now"
 * SG_INFO: Things like iterations residuals while algorithms are running. Something that can be printed every iteration
 * SG_GCDEBUG gargabe collector debug, things like reference counting
 * SG_PRINT not to be used from within Shogun algorihtms, this is for users

All those macros are just called, no if statement needed.