Outside of using the virtual machine itself, performance for the rest of your PC shouldn’t be very impacted by enabling virtualization.
However, there are mixed reports about some virtualization options taking up extra performance or causing minor issues when they’re enabled, so I wouldn’t recommend enabling virtualization when it isn’t in active use.