Handle scalar symbolic factors better