If you want to make the VTube Studio background transparent in OBS, you will need to Enable Transparency in OBS and then set a Chroma Key for the Scene that includes your VTube avatar.