Not necessarily. Many lenders have expertise in originating mortgages but not servicing them. Other institutions are better equipped to service VA loans and offer customer service for the long haul.
The key thing is your loan terms cannot change. From a day-to-day perspective, this is mostly a matter of where you send your mortgage payment each month.