Fantasies are purely a product of one’s imagination, which is created by our thought process. They are imagined situations that might be away from reality but convey the creator’s desires or wishes. It is unlikely that a fantasy that is pleasant and leaves the person with a pleasurable feeling could generate negative energy. The only negative energy that can be produced could arise if one repents or feels guilty about it later on.
It is a completely subjective view. How one personally feels about such content and the feelings that it leaves you with would be important markers to label them as harmful or otherwise.
Just like alcohol, drugs and divorce etc. could be a sin for one but not for the other person, similar is the case with such sexual fantasies. If having such fantasies causes no adverse effect on ones personal, psychological, social and/or occupational functioning then there is little reason for it to be called as morally incorrect or sinful.
In reality, other women are not a part of your sexual world. You do not share any sexual interaction with them and the nature of your relationship with them leaves little opportunity for it. Thus, they are like forbidden fruits for you and it is human nature to derive thrill and excitement from the unknown!
As long as the individual is healthy and such fantasies do not impair his level of functioning or cause him or others around him any kind of stress, one can safely continue to enjoy ones sexual fantasies.