Petroleum products can dissolve latex, so if you use latex yes it is a concern.  Typically latex takes minutes to hours to weaken in petroleum.  I’m not familiar with nitrile gloves dissolving from Petroleum jelly (most offices use nitrile).