Skip col_buffer allocation when not needed

In the line of #1118 this PR based on the discussion with @longjon and @shelhamer extends the cases when the col_buffer is not needed and a matrix multiplication is enough to compute the result.