I think the cake got its "velvet" name to describe the fine and velvety crumb of the cake. The word "velvet" primes people to expect a special texture. Many old-fashioned cakes that contained cocoa powder, back when cocoa powder was a new ingredient, were described as velvet because the texture was softer than that of a white or yellow cake.