But it is not so. Having that thought alone or just a lustful look is already a violation, if we do it deliberately. The degree of violation is less when we do it unintentionally. But it is not that we don't commit any violation at all. It is also considered a violation but of a lesser degree.
Thought creates speech and speech creates action.
If we don't have any thoughts, we are unable to say anything. Without thoughts, our body would not be able to act. Therefore, there are three kinds of violations. The delicate ones are from our mind. The coarser ones are in our actions.