Metal has no diffuse reflection.

If the material is metal, simply 0 the diffuse colour(or diffuse texture).